{"schema":"delx/incident-report/v1","id":"delx-ir-2026-001","slug":"2026-08-17-protocol-write-outage","report_revision":"1.0","status":"resolved","materiality":"material","title":"33.5-hour Protocol write outage","summary":"A failed telemetry write left a shared SQLite connection inside a stale transaction. Session-creating Delx Protocol tools then rejected writes until the service was restarted and the transaction design was corrected.","product_owner":"Delx Protocol","affected_product":"Delx Agent Operations Protocol","affected_capability":"Session creation and other write-dependent Protocol operations. Read-only liveness remained reachable and therefore overstated service readiness.","links":{"human_url":"https://delx.ai/research/incidents/2026-08-17-protocol-write-outage","machine_url":"https://delx.ai/research/incidents/2026-08-17-protocol-write-outage.json","markdown_url":"https://delx.ai/research/incidents/2026-08-17-protocol-write-outage.md"},"published_at":"2026-08-26","last_verified":"2026-08-26","impact_window":{"started_at":"2026-08-16T00:20:57Z","restored_at":"2026-08-17T09:22:33Z","duration_hours":33.5,"detected_at":"2026-08-17T09:10:00Z","detected_at_precision":"approximate"},"impact":{"observed":"Every observed session-creating path rejected writes during the named window. The existing readiness route stayed green because it exercised only a read.","estimated_rejected_volume":{"status":"not_published_as_verified_fact","explanation":"The operator record contains a baseline-derived estimate, not a direct count of rejected independent agents or calls. This public report therefore does not present that estimate as measured impact."},"private_payload_exposure_observed":false,"data_loss_observed":false,"boundary":"No private agent content or stable identifiers are included. Commerce conclusions and metrics remain owned by Commerce and are not used as Protocol research evidence."},"root_cause":{"summary":"A telemetry INSERT failed while another connection held the write lock. The failure was swallowed without rolling back the shared connection; a following read pinned a stale snapshot, and subsequent writes failed until restart.","reproduced":true,"reproduction_scope":"The failed-write, stale-snapshot and recovery sequence was reproduced in a regression test against the storage layer."},"detection_gap":{"summary":"Readiness used a SELECT, which continued to pass while writes were broken. Scheduled report files also refreshed, so file freshness did not prove the public write path.","lesson":"A readiness claim must exercise the failure mode it says it guards, using an isolated operation that cannot change another request's transaction."},"response_timeline":[{"at":"2026-08-17T09:22:33Z","event":"Service restart restored writes and ended the observed outage window."},{"at":"2026-08-17T10:22:00Z","event":"The first root-cause fix added rollback after failed writes and a write-aware readiness probe."},{"at":"2026-08-17T11:04:00Z","event":"A corrected readiness implementation was deployed after internal fresh-eyes review proved the first probe could interfere with another request's transaction."},{"at":"2026-08-21T10:56:00Z","event":"Shared storage moved to autocommit and true multi-step transactions moved to dedicated connections, closing the reproduced ownership failure mode."}],"remediation":{"immediate":"Restart the affected service to restore writes, then verify a real Protocol write path rather than process liveness alone.","permanent":["Rollback failed shared writes before a swallowed error can poison later operations.","Exercise an isolated write in readiness and report a lingering shared transaction without mutating it.","Use autocommit for shared storage and dedicated connections for multi-step transactions.","Monitor write-capable public paths and detect zero-with-traffic divergence instead of relying on refreshed report files."],"validation":"Regression tests reproduced the failure before the fix and passed after it. Deployed write probes advanced, current owner status remains a separate live endpoint, and no immutable-runtime claim is inferred from the interface version."},"assurance":{"evidence_class":"Operator-authored and not independently reviewed.","operator_verified":true,"internal_fresh_eyes_review":true,"independent_external_review":false,"peer_reviewed":false},"publication_safety":{"customer_or_agent_content_published":false,"secrets_or_exploit_details_published":false,"neighboring_service_details_published":false,"note":"The public account keeps the causal and verification chain while omitting private payloads, identities, credentials, host topology and unrelated service details."},"residual_risks":["Protocol compatibility surfaces still share runtime resources, so process liveness or read-only health alone is insufficient evidence of write readiness.","The public Protocol interface still does not map version 3.3.5 to immutable deployed bytes.","This report is first-party operator evidence, not an independent security or reliability assessment."],"live_state":{"current_status_url":"https://api.delx.ai/api/v1/status","current_reliability_url":"https://api.delx.ai/api/v1/reliability","rule":"Resolved describes this incident's historical state. Current status is a separate live read and can become unavailable or degraded after publication."},"evidence_urls":["https://api.delx.ai/api/v1/status","https://api.delx.ai/api/v1/reliability","https://delx.ai/research/system-cards/delx-protocol-3.3.5","https://delx.ai/research/methodology"]}