INC-2219: anatomy of a production incident, minute by minute
A feature flag reached 100% of a large SaaS platform at 13:40 UTC. Forty-eight minutes later, 12% of epic board views were failing. What happened, why four layers of defense lined up, and what a project manager should do in the room and after it.
On Tuesday 3 August 2027, a change to the epic boards query reached every group on GitLab.com. It had done everything right on paper. It sat behind a feature flag. It had rolled out to 1% of groups on Monday and to 10% on Tuesday morning with no errors. At 13:40 UTC the flag went to 100%.
By 14:12 an alert fired. By 14:26 the incident was severity 2 and the public status page said degraded performance. By 14:40 the service was mitigated. By 15:20 it was resolved. Then the real work started: understanding why four separate defences had failed at once, and fixing every one of them.
INC-2219 is a case from Projects, For Real, and it's one of the scenarios our students run live in week 14. It's worth studying closely because almost every software incident looks like it: a reasonable change, a reasonable rollout, and a combination of small gaps that nobody had lined up before.
The setup
The change belonged to the Run the Engine team, a product group that keeps a large product area healthy while also building a new feature (Beacon) with the same people. The work, labeled E2, moved epic boards onto a new work-items API. Like most changes on a continuously deployed service, it shipped dark behind a feature flag and was switched on in steps.
Summary. Epic boards returned errors or timed out for groups with large numbers of epics and issues. About 12% of epic board page views failed during the incident. Cause: a new query path, enabled by feature flag at 13:40, performed a sequential scan on large namespaces because a supporting index was missing. Mitigated by disabling the flag.
The timeline
| Time | Event |
|---|---|
| 13:40 | Flag epic_boards_work_items_query enabled for 100% of groups (rollout issue #45070) |
| 13:52 | Error rate on the epic boards endpoint begins rising as US morning traffic grows |
| 14:12 | Alert: web apdex below SLO for the boards feature category (sustained 5 min) |
| 14:18 | Engineer on call declares an incident, severity 3; incident channel and call open |
| 14:21 | Support reports 4 tickets from large customers: epic boards unusable |
| 14:26 | Severity raised to 2; communications lead engaged; status page: degraded performance, epic boards |
| 14:31 | Fatima (pulled in by group escalation) correlates the start of errors with the flag change in the feature-flag log |
| 14:33 | State → identified. Mitigation agreed: disable the flag |
| 14:40 | Flag disabled via ChatOps. State → monitoring |
| 14:46 | Error rate and apdex back to baseline |
| 15:20 | State → resolved. Status page: operational. Review required (severity 2) |
102 minutes from first errors to resolution; 48 minutes from first errors to mitigation, 20 of them before the alert fired. No data loss. Four support tickets; two customers asked for a root-cause analysis through their account teams.
What went right, and why it matters
A good review starts with what worked, because those are the things you must not lose when you fix the rest.
- The change was behind a flag. Mitigation was one ChatOps command, not a revert and a redeploy. That single design choice probably saved an hour.
- The feature-flag log existed. Fatima correlated the error curve with the flag change in under ten minutes of joining the call.
- Severity moved when the facts moved. It was declared at 3, then raised to 2 as soon as customer impact was clear, which pulled in the communications lead and the status page.
- The escalation path worked. The owning development team was paged in and on the call within 10 minutes.
Roles: who does what when the clock is running
The classic incident failure is that everyone investigates and nobody coordinates or communicates. GitLab's handbook describes three roles that prevent it. The incident lead owns the incident and drives it to resolution; every incident has one, set deliberately. Responders do the technical investigation and mitigation. A communications lead handles stakeholders, customers and the status page, required for severity 1 and 2. These are filled from on-call rotations: the engineer on call for first response and an incident manager on call for larger incidents.
Severity runs from 1 (critical) to 4 (low, an inconvenience with a workaround), and all severity 1 and 2 incidents need a review. The incident's state moves through investigating, active, identified, monitoring and resolved. Anything lasting more than five minutes is communicated publicly.
The blameless review
Incident reviews at GitLab are blameless: they focus on why and how the incident happened, never on who was at fault, so people share what they actually saw. Each review has a review initiator (usually the incident lead), a DRI from the team that owns the affected service (here Beacon's group, with the engineering manager Sam Adeyemi accountable), and a bar raiser: a peer from a panel of engineering managers, product managers and staff engineers who asks the hard questions and must approve the review before it closes. The handbook expects the review within five working days of resolution.
The bar raiser's questions are the useful part. Is the root cause clearly identified? Do the corrective actions match the type of root cause? Did we have the right observability? Did the change go through the right control flow? Did we miss it in testing? Could we have detected and mitigated faster? Where else could this happen? And one rule that cuts through a lot of theatre: if a code change caused the incident and none of the corrective actions make a similar defect less likely to escape again, the review isn't finished.
- Why did epic boards time out? The new query did a sequential scan on the work-items hierarchy table for namespaces with more than about 50,000 items.
- Why did it do a sequential scan? A composite index the new query needed didn't exist; the old query used a different access path.
- Why wasn't the missing index caught in review? The database review checked the query plan against a mid-sized group, where the planner chose an acceptable plan. Large namespaces weren't tested.
- Why didn't the 1% and 10% steps reveal it? The flag rolled out by percentage of groups, sampled at random. Fewer than 0.5% of groups are large, and none of the very largest landed in the 10% sample. Then 10% → 100% happened the same day.
- Why did that pattern seem safe? The rollout checklist was written for features whose load is spread evenly. Nobody had considered risk concentrated in a few very large namespaces.
Contributing factors: alerting on the aggregate boards category hid the epic board endpoint's degradation for 20 minutes; the 10% → 100% step left no time for the US morning peak to test the change.
Notice that every answer is about the system, not a person. Not “the reviewer missed it” but “the review checked a mid-sized group.” That isn't politeness. It's accuracy: the reviewer did what the process asked.
Four slices of Swiss cheese
The five whys implies one neat chain of causes. Real incidents usually have several. The Swiss cheese model explains INC-2219 better: four layers of defense existed, each had a hole, and on 3 August the holes lined up.
Corrective actions that actually prevent the next one
GitLab labels corrective actions, links them to the incident and gives them severity and priority labels matching the incident, which puts them under the same resolution targets as bugs of that severity. The handbook asks for SMART actions: not “investigate monitoring for this scenario” but “add alerting for all cases where this service returns more than 1% errors.”
| # | Corrective action | Addresses | Owner | Status |
|---|---|---|---|---|
| CA-1 | Add composite index for work-items hierarchy lookups by namespace | Proximate cause | Fatima | ✓ Merged 9 Aug |
| CA-2 | Rollout template for flags on board and list finders: enable for the 20 largest namespaces first; monitor 24 h | Rollout sampling | Alex | ✓ 18 Aug |
| CA-3 | Database review guideline: include a plan from the largest namespace | Review gap | Alex + DB maintainers | ✓ 27 Aug |
| CA-4 | Endpoint-level apdex alert for epic and issue boards, with runbook | Detection time | Sam + Observability | ✓ 13 Sep |
| CA-5 | At least 24 h between rollout steps above 10%; one step must span a US and an EU peak | Same-day jump | Sam | ✓ 16 Aug |
| CA-6 | Check other finders migrated under E2 for the same pattern | Beyond this incident | Fatima | ✓ 1 found, fixed |
The set has one property worth copying: every slice of cheese gets its own action. CA-1 fixes this incident. CA-2, CA-3 and CA-5 fix the holes that let a whole class of defects through. CA-4 fixes detection. CA-6 goes looking for the same hole elsewhere, and finds one. That's what the bar raiser looked for before approving the review on 9 August.
The flag came back through a production change request (classified C3) on 10 to 13 August: the 20 largest namespaces first, then 10%, 50% and 100% over three days, with the metrics to watch and the rollback command written down. It reached 100% with p95 latency 40% lower than before the original change, thanks to the index.
What it cost the project
This is the part project managers miss. For the team, INC-2219 wasn't 102 minutes of degraded service. The review and corrective actions took one engineer most of a week and another several days, on top of a July security fix. Together, unplanned work took about 30% of the group's capacity for three weeks. The month's error budget was 96% spent. And the Beacon feature, due for general availability in release 20.4, had 34 weight of remaining scope against about 22 available.
The team held the date and cut scope from the bottom of the story map, through a formal commitment change request that took two days and no meetings. Three customers who had asked for the deferred capability heard about it from their account managers in time to plan.
The same discipline outside software
On 14 October, during the hypercare of a new expense system (Ledger), the first reimbursement file to payroll was rejected for employees in three countries. About 60 people would miss the payroll run unless it was fixed within two days. The PM treated it as a severity 2 hypercare incident: an incident issue, an owner, a workaround (accounts payable processed the payments manually by the deadline) and a root-cause analysis within five business days.
| Category | Possible cause raised | Verified? |
|---|---|---|
| Data | Amounts in zero-decimal currencies (JPY, KRW, CLP) sent with two decimals | ✓ Root cause |
| Process | Integration test cases used only USD, EUR, GBP, CAD | ✓ Contributing |
| Technology | Export uses one global currency format | ✓ Contributing |
| Requirements | BRD said “amounts in local currency” without precision rules | ✓ Contributing |
| People | New payroll analyst unfamiliar with the file | ✗ Not a factor |
| Suppliers | Vendor release changed the export | ✗ No changes |
The first explanation offered in the room was the new payroll analyst. The fishbone's categories forced the group to check everything else, and the real causes were a data rule, a testing gap and an ambiguous requirement: three holes lining up again. Two of the three corrective actions changed templates that future projects will use. The incident became an improvement to the organization, not just a fix.
A checklist for project managers
- Know your project's severity definitions and who declares an incident.
- During an incident, don't investigate. Make sure there's an incident lead, help with communication, and protect the responders from interruptions.
- Afterwards, insist on a blameless review with a corrective action for each layer of defense that failed.
- Put the corrective actions into the plan with owners, dates and capacity.
- Ask: where else could this happen?
How students practice this
In week 14 every Gantt Guild student runs a 30-minute live incident simulation as incident lead, then writes the blameless review: timeline, five whys, Swiss cheese and SMART corrective actions. They also rebuild the incident's timing metrics from raw event logs in SQL.
Adapted from Projects, For Real, chapters 38 and 39. The projects, people, vendors and numbers are the book's fictional case studies, set against GitLab's publicly documented practices. Gantt Guild is not affiliated with GitLab Inc.
