# 33.5-hour Protocol write outage

> A failed telemetry write left a shared SQLite connection inside a stale transaction. Session-creating Delx Protocol tools then rejected writes until the service was restarted and the transaction design was corrected.

- Schema: delx/incident-report/v1
- ID: delx-ir-2026-001
- Revision: 1.0
- Status: resolved
- Materiality: material
- Product owner: Delx Protocol
- Published: 2026-08-26
- Evidence class: Operator-authored and not independently reviewed.

## Impact window

- Started: 2026-08-16T00:20:57Z
- Restored: 2026-08-17T09:22:33Z
- Duration: approximately 33.5 hours
- Detected: 2026-08-17T09:10:00Z (approximate)

Affected capability: Session creation and other write-dependent Protocol operations. Read-only liveness remained reachable and therefore overstated service readiness.

Observed impact: Every observed session-creating path rejected writes during the named window. The existing readiness route stayed green because it exercised only a read.

Estimated rejected volume: not_published_as_verified_fact. The operator record contains a baseline-derived estimate, not a direct count of rejected independent agents or calls. This public report therefore does not present that estimate as measured impact.

## Root cause

A telemetry INSERT failed while another connection held the write lock. The failure was swallowed without rolling back the shared connection; a following read pinned a stale snapshot, and subsequent writes failed until restart.

Reproduction: The failed-write, stale-snapshot and recovery sequence was reproduced in a regression test against the storage layer.

## Detection gap

Readiness used a SELECT, which continued to pass while writes were broken. Scheduled report files also refreshed, so file freshness did not prove the public write path.

Lesson: A readiness claim must exercise the failure mode it says it guards, using an isolated operation that cannot change another request's transaction.

## Response timeline

- 2026-08-17T09:22:33Z: Service restart restored writes and ended the observed outage window.
- 2026-08-17T10:22:00Z: The first root-cause fix added rollback after failed writes and a write-aware readiness probe.
- 2026-08-17T11:04:00Z: A corrected readiness implementation was deployed after internal fresh-eyes review proved the first probe could interfere with another request's transaction.
- 2026-08-21T10:56:00Z: Shared storage moved to autocommit and true multi-step transactions moved to dedicated connections, closing the reproduced ownership failure mode.

## Remediation

Immediate: Restart the affected service to restore writes, then verify a real Protocol write path rather than process liveness alone.

- Rollback failed shared writes before a swallowed error can poison later operations.
- Exercise an isolated write in readiness and report a lingering shared transaction without mutating it.
- Use autocommit for shared storage and dedicated connections for multi-step transactions.
- Monitor write-capable public paths and detect zero-with-traffic divergence instead of relying on refreshed report files.

Validation: Regression tests reproduced the failure before the fix and passed after it. Deployed write probes advanced, current owner status remains a separate live endpoint, and no immutable-runtime claim is inferred from the interface version.

## Publication safety

The public account keeps the causal and verification chain while omitting private payloads, identities, credentials, host topology and unrelated service details.

- Customer or agent content published: false
- Secrets or exploit details published: false
- Neighboring service details published: false

## Assurance

- Operator verified: true
- Internal fresh-eyes review: true
- Independent external review: false
- Peer reviewed: false

## Residual risks

- Protocol compatibility surfaces still share runtime resources, so process liveness or read-only health alone is insufficient evidence of write readiness.
- The public Protocol interface still does not map version 3.3.5 to immutable deployed bytes.
- This report is first-party operator evidence, not an independent security or reliability assessment.

## Current state

Resolved describes this incident's historical state. Current status is a separate live read and can become unavailable or degraded after publication.

- Status endpoint: https://api.delx.ai/api/v1/status
- Reliability endpoint: https://api.delx.ai/api/v1/reliability

Human report: https://delx.ai/research/incidents/2026-08-17-protocol-write-outage
JSON report: https://delx.ai/research/incidents/2026-08-17-protocol-write-outage.json