Failure Museum.Made with ❤️ by Aniruddh Jangid
Submit a case

EXHIBIT 029 · IT & SOFTWARE

GitLab: backups were not a recovery plan

A production database incident led to data loss and a prolonged restoration effort.

OrganizationGitLab
Period2017
LocationGlobal
StatusRecovered
ImpactHigh
EvidenceTechnical postmortem
Compare

Illustrative draft · Source-backed synopsis · Museum analysis marked separately · Updated 5 October 2026

GitLab interface in December 2015; contextual product screenshot, not a record of the January 2017 database incident.
GitLab interface in December 2015; contextual product screenshot, not a record of the January 2017 database incident.YorickPeterse Interface screenshot · Context photograph; not the incident itself. · MIT · Resized for display.
01 / EXECUTIVE SUMMARY

The outcome is only the beginning.

A production database incident led to data loss and a prolonged restoration effort. [1]

THE PRINCIPLE TO CARRY FORWARD

Measure successful restore drills, not the existence of backup jobs; build tooling that makes destructive operations difficult to confuse.

02 / THE PROMISE

What the system was trying to do.

Keep developers’ repositories and collaboration data available.

Museum interpretation

The assumption beneath the promise

Museum hypothesis to examine: A completed backup job implies a timely and usable restoration. This is an analytical proposition to test against the record, not an attributed statement by the organization.

Success should be assessed against the intended service and its safety, reliability, or integrity constraints. A headline outcome cannot tell us whether those constraints were visible, tested, or owned before the event.

03 / WHAT HAPPENED

A sequence, not a single moment.

The intended system

Keep developers’ repositories and collaboration data available.

Source-backed synopsis

The event or review

A production database incident led to data loss and a prolonged restoration effort. [1]

Source-backed synopsis

The documented aftermath

GitLab published an open postmortem and improved recovery practices. [1]

Source-backed synopsis
Timeline of documented milestones. Gaps are not evidence that nothing happened. Sources: [1]
04 / WARNING SIGNS

What could be seen at the time?

Documented in cited record

The postmortem described backup and recovery mechanisms that did not all work as expected. [1]

Who could see it?

See the cited investigation for named teams and the information they held. The synopsis does not infer awareness by every stakeholder.

Why was it not enough?

Signals need interpretation, authority, and a route to action. This is an analytical question, not proof that a warning was deliberately ignored.

05 / KEY DECISIONS

Choices made within constraints.

Museum analysis: Restoration needed validation under the same permissions, dependencies, and time pressure as an incident.

Available alternatives

Pause, test, narrow the operating envelope, seek independent review, or build a recovery option. These are proposed analytical alternatives; feasibility at the time is not established.

Information available

The postmortem described backup and recovery mechanisms that did not all work as expected. [1]

Incentives & constraints

Delivery pressure, cost, authority, and incomplete knowledge may shape a decision. Their specific weight is not established by this synopsis.

Risk accepted

Ask whether the relevant risk was recognized, who had authority to accept it, and whether affected people understood its implications.

06 / SYSTEM MAP

How conditions connected.

Select a node to inspect its evidence. Links show a proposed analytical relationship, not a measured causal effect.

05 / External context04 / Warning signal03 / Decisions02 / Technical conditions01 / Trigger06 / Consequences
Triggering event

A production database incident led to data loss and a prolonged restoration effort. [1]

Legend: numbered nodes = analytical categories; dashed arrows = proposed influence or propagation. Museum interpretation informed by the sources below. Feedback and omitted influences require further investigation.
07 / CONSEQUENCES

Beyond the immediate event.

Users lost access, and some recent database data could not be recovered. [1]

Technical impactOperational impactCustomer impact

Categories identify documented or relevant consequences. Financial, human, and environmental totals are not estimated here when the source base does not support them.

08 / WHAT THE MODELS EXPLAIN

More than one lens.

Analytical lens

Fault Tree Analysis

Work backward from a defined loss using logical combinations of conditions.

In this case, use the lens to examine untested recovery. This application is museum interpretation, not a finding of the original investigation.

Limit: Results depend on the chosen top event and completeness of branches.

Analytical lens

FMEA

Identify failure modes, their effects, and controls before or during system development.

In this case, use the lens to examine untested recovery. This application is museum interpretation, not a finding of the original investigation.

Limit: Lists can miss interactions and unanticipated operating conditions; scores are not exact probabilities.

Analytical lens

Systems thinking

Examine relationships, boundaries, feedback, and incentives across a whole system.

In this case, use the lens to examine untested recovery. This application is museum interpretation, not a finding of the original investigation.

Limit: A broad lens needs explicit boundaries and evidence to avoid explaining everything after the fact.

09 / COUNTERFACTUALS

What could have changed the outcome?

Counterfactuals are hypotheses. They identify possible intervention points without claiming that a different choice would certainly have prevented the event.

INTERVENTION 2 / During early testing

Test the operating envelope

Exercise realistic boundary conditions and shared dependencies rather than validating components only in isolation.

Cost & feasibility

Requires time and independent review before commitment.

Likely effectiveness

May reduce exposure or consequences. Not quantified; feasibility and effect must be assessed against evidence available at the time.

10 / LESSONS

From hindsight to a usable practice.

FOR FOUNDERS

Measure successful restore drills, not the existence of backup jobs; build tooling that makes destructive operations difficult to confuse.

Before accelerating hiring or spending, name the assumptions that remain untested and set evidence thresholds for the next commitment.

11 / AFTERWARD

Recovery is another investigation.

GitLab published an open postmortem and improved recovery practices. [1]

A corrective action is evidence of a response; it is not, by itself, evidence that the wider pattern has disappeared.

12 / SOURCES & EVIDENCE

Follow the record.

Documented claims are linked to sources. Analytical applications, lessons, and intervention proposals are museum interpretation. This draft does not establish motives or a single complete causal explanation.

  1. 1
    about.gitlab.com — GitLab case record

    Technical postmortem · Accessed 4 October 2026

Editorial history & limits

Version 1 · 4 October 2026 — source-backed illustrative synopsis created. Independent editorial review is pending. No claim of exhaustive investigation. New sources may alter the analysis.