Example SEV-1 postmortem: orders-api outage

# Orders API outage after v41 deploy - **Date:** Aug 12 · **Severity:** SEV-1 · **Duration:** 17 min (02:14–02:31 UTC) - **Services:** orders-api, checkout · **Author:** Platform team ## Executive summary Deploy v41 removed an explicit HTTP client timeout from orders-api. During overnight peak, upstream calls piled up and exhausted the connection pool, returning 502s for ~12% of checkout attempts. Automatic rollback restored service in 17 minutes. Root cause identified with high confidence; prevention shipped same week. ## Impact ~12% of checkout requests failed during the window · ~$8k GMV exposed · zero support escalations · no data corruption ## Timeline | Time (UTC) | Event | Source | | --- | --- | --- | | 02:14 | v41 deploy begins (routine change) | CI/CD | | 02:19 | Checkout error-rate alert fires | Grafana | | 02:21 | On-call confirms 502 pattern from orders-api | PagerDuty | | 02:24 | Rollback decision — deploy diff shows removed timeout | Incident channel | | 02:31 | Rollback complete; error rate at baseline | Deploy record | ## Root cause (5 Whys) 1. Pool exhaustion → sockets never expired. 2. v41 replaced configured timeouts with library defaults (infinite). 3. Change passed review: diff looked like dead config removal. 4. No CI check validates required client timeout keys. 5. Guardrail absent because deploys had never structurally broken a dependency contract before. **Systemic finding:** config schema must enforce timeout presence. ## Detection & recovery Automated detection won (MTTD ~5 min, alert fired on error-rate, not saturation). Rolling back beat debugging under pressure — correct call, worth encoding in a runbook. ## What went well / poorly Well: alert coverage, decisive rollback, calm channel discipline. Poorly: no soak environment exercised pool behavior; deploy checklist didn't flag dependency-contract changes; pool-saturation metric existed but had no alert attached. ## Where we got lucky Peak load happened during low checkouts (nightly); one retry storm avoided because checkout clients capped retries already. ## Action items | Action | Type | Owner | Priority | | --- | --- | --- | --- | | Schema validation rejecting timeout-less clients | Prevent | Infra | P0 | | Integration test exercising soak behavior | Prevent | QA | P1 | | Alert on pool saturation ≥80% | Detect | SRE | P0 | | "Dependency-contract changes" review tag | Prevent | Eng leads | P1 |