Fictional composite example for format reference.
# Elevated payment declines during PSP failover
- **Date:** Jul 30 · **Severity:** SEV-2 · **Duration:** 42 min (11:02–11:44 UTC)
- **Services:** payments-gateway, PSP primary provider · **Author:** Payments team
## Executive summary
Payment provider (PSP) announced regional degradation; automatic failover to backup provider triggered correctly, but our outbound rate limiter treated both providers' limits as a shared bucket, throttling legitimate traffic to 60% of capacity. Declines rose 9× baseline until limiter reconfiguration. Failover design verified sound; limiter semantics now provider-scoped.
## Impact
Decline rate 14% vs 1.5% baseline for 42 minutes · no transactions lost (correct declines retried later) · increased support contacts (+35)
## Timeline
| Time (UTC) | Event | Source |
| --- | --- | --- |
| 11:00 | PSP posts degradation notice | PSP status feed |
| 11:02 | Auto-failover triggers to backup | payments-gateway |
| 11:06 | Decline-rate anomaly alert fires | Monitoring |
| 11:18 | Limiter shared-bucket hypothesis confirmed via dashboards | On-call |
| 11:44 | Backup-provider-scoped limits applied; declines normalize | Config release |
## Root cause (5 Whys)
1. Throttling despite spare backup capacity → shared limit bucket.
2. Limiter designed pre-backup-provider, keyed per-environment not per-endpoint.
3. No test simulated dual-provider configuration under load.
4. Failover drill checked availability, not throughput semantics.
5. **Systemic:** resilience testing must include rate-limit topology, not just health checks.
## Detection & recovery
Alert fired 4 min into impact (good). Diagnosis took 12 min — declined to guess under uncertainty, verified via dashboard slicing (right choice; wrong guess would have made it worse).
## What went well / poorly
Well: automated failover, disciplined verification-before-action, PSP comms.
Poorly: drills never exercised limiter behavior, backup limits under-provisioned vs peak, no synthetic transaction probe during failover windows.
## Where we got lucky
Traffic below Black-Friday peaks; partial declines auto-retry prevented lost sales.
## Action items
| Action | Type | Owner | Priority |
| --- | --- | --- | --- |
| Per-endpoint scoping in limiter config | Prevent | Payments | P0 |
| Failover drill including throughput assertions | Detect | SRE | P1 |
| Synthetic payment probe on provider switch | Detect | QA | P1 |
| Provision backup limits to 100% peak | Mitigate | Ops | P2 |