The overnight payments incident
The client and every number in this exercise are fictional, made up for practice.
Background
A credit union's online banking platform, which we run, has a service level objective of 99.9% of payment requests succeeding over a 30-day month. Payments average 2,000 requests a minute. Before tonight, 31,000 payment requests had failed this month. At 05:58 UTC a configuration change was deployed to the payments service. The timeline below comes from the incident record.
| Time | Event |
|---|---|
| 05:58 | Configuration change deployed to the payments service. |
| 06:04 | Alert: payment latency above threshold. About 30% of payment requests start failing. |
| 06:09 | Tier 1 acknowledges the alert. |
| 06:15 | Tier 1 confirms the failures and escalates to Tier 2. |
| 06:21 | Tier 2 finds the database connection pool exhausted since the 05:58 change. |
| 06:28 | Decision to roll back the change. |
| 06:38 | Rollback complete. |
| 06:46 | Failure rate back to normal. Incident resolved. |
Your task
- How many payment requests failed during the incident, and how much of the month's error budget has now been used?
- Break the 48 minutes from the change to recovery into stages. Where did the time go, and which stage would you shorten first?
- List three changes that would make this incident less likely or shorter next time.
- Write the first update to the credit union's head of operations, as you would have sent it at 06:20.
One way to approach it
1. The numbers. A 30-day month has 43,200 minutes, so about 86.4 million payment requests (2,000 × 43,200). The objective allows 0.1% to fail: 86,400 requests. The incident ran 42 minutes at 30% failure: 42 × 2,000 × 0.30 = 25,200 failed requests. With the 31,000 earlier, the month has used 56,200, or about 65% of the budget, leaving about 30,200.
2. Where the time went. Detection 6 minutes (05:58 to 06:04), acknowledgement 5 (to 06:09), confirming and escalating 6 (to 06:15), diagnosis 6 (to 06:21), deciding 7 (to 06:28), rolling back 10 (to 06:38), recovery 8 (to 06:46). The cheapest stage to shorten is the decision: the change was the obvious suspect from 06:04, and a rule that a change in the last hour is rolled back first, investigated second, would have saved most of the 24 minutes between the alert and the decision.
3. Changes. A deployment check that watches the payment failure rate and rolls back automatically if it rises within minutes of a change. Load testing for connection pool settings before release. A runbook step for Tier 1: if a change went out in the last hour, page the service owner at once rather than after confirmation. Strong answers also ask why the alert was on latency rather than on failed payments, which is what members feel.
4. The update. Something like: “Since 06:04 UTC about a third of online payments have been failing. Members may see errors when sending payments; other online banking is working. We have found a likely cause linked to a change made at 05:58 and are preparing to reverse it. Next update by 06:45, sooner if service is restored.” It says what is affected, what is being done and when the next update comes, and guesses at nothing.