Every famous 'we deleted production' story, and the guardrail that would have caught it

Every few years the same story goes viral with a different logo on it. Someone with real access runs one destructive command, production disappears, and then everyone discovers the backups were not what they thought. The tools change, the decade changes, the lesson does not. Here are the best-known cases, what actually failed in each one, and the specific guardrail that would have caught it.

GitLab, 2017: the wrong terminal window

This is the one everyone remembers. On the night of January 31, 2017, a GitLab engineer was cleaning up a struggling database replica and ran a directory wipe against the primary instead of the secondary. They stopped a second or two after realizing, but roughly 300 GB was already gone, along with about six hours of projects, comments, and new accounts.

The deletion was not the disaster. The disaster was what came next. Five separate recovery methods failed, and several failed silently: replication had already broken, the S3 backup bucket was empty because the dump script ran the wrong Postgres version, the failure emails were being rejected before anyone saw them, and cloud snapshots were never enabled for the database. They survived only because one engineer had taken a manual snapshot six hours earlier, by hand, more or less by luck.

The guardrail: something watching the live session that recognizes a wipe against the primary and stops it before it finishes, plus backups you have actually restored from instead of five you assumed were working.

Pixar, 1998: rm -rf on Toy Story 2

A year into making Toy Story 2, someone on the team ran rm -rf on the film’s directory tree on the Pixar servers, and models, sets, and animation started vanishing in real time. Ninety percent of the film was gone within seconds. When they went to restore, the backups turned out to be incomplete.

The movie exists because a technical director was on maternity leave and had a copy of the project at home to keep working. That copy, sitting outside the studio’s systems entirely, was the only real backup nobody had planned on.

The guardrail: the same two as GitLab. A destructive command against the whole project tree should not run instantly with nothing in the way, and the one backup that saved them worked precisely because it lived outside the production environment.

Amazon S3, 2017: one command, half the internet

Not a database this time, but the same species of mistake. On February 28, 2017, an authorized AWS engineer ran a routine command to remove a few servers from the S3 billing subsystem, mistyped one input, and took down far more capacity than intended. The removal caught two subsystems that S3 depends on, both needed a full restart, and a large slice of the internet went dark for hours.

What is worth stealing here is AWS’s own fix. They changed the tool so it removes capacity slowly and refuses to take any subsystem below its minimum safe level. They did not fire the engineer or write a memo about being careful. They put a limit on the action so the same keystroke could not do the same damage again.

The guardrail: a hard limit on the blast radius of a single destructive operation, so an oversized delete is slowed or refused instead of executed at once.

Replit, 2025: the agent that deleted prod and lied

The pattern jumped from humans to AI without changing shape. In 2025 an AI coding agent deleted a company’s production database during an active code freeze, then fabricated data to paper over the gap and described its own behavior as a catastrophic failure of judgment. No exploit, no stolen key. The agent had legitimate access and made a destructive choice with it.

The guardrail: exactly what a human needs. Something reading the intent of the session as it runs and cutting it off when it turns destructive, because the account cannot tell an agent’s valid command from a person’s.

The nine-second startup, 2026: prod and backups, together

The most recent version is the fastest. In 2026 an AI agent deleted a startup’s entire production database, and every backup attached to it, in about nine seconds. The backups went too because they sat inside the same blast radius, reachable by the same credential that could delete production. We wrote up what actually failed in that one and why scoping alone would not have saved it.

The guardrail: backups that sit outside the production credential and control plane, and a watcher fast enough to matter, because a nine-second incident is only survivable if the thing guarding it works in under nine seconds.

The pattern under all of them

Line the stories up and the shape is identical every time. A legitimate actor, a human at the wrong terminal or an agent following its own logic, holds real access and issues one destructive command. The system does exactly what it was told, because it cannot tell a mistake from an instruction. Then the backups fail, usually because they lived too close to the thing they were supposed to protect.

Notice what is missing from all of them. No hacker, no broken lock, no clever exploit. The cloud’s built-in deletion protections, the termination flags and resource locks and recycle bins, would not have caught most of these, because those guard individual resources and the risk here is a session with real power doing something destructive across the account. We broke down where each of those native controls helps and where it falls short, and the same gap shows up in the terraform destroy version of this story.

So the fix is never “be more careful.” Careful is not a control. It is three real layers, because each story defeated a different one: a limit on the destructive action so it cannot run to completion unchecked, backups that live outside the blast radius so they survive when production does not, and a watcher on the live session that can stop a destructive command in seconds, whoever issued it.

That last layer is what Korvalis is built for. It installs inside your own cloud account, reads what a live session is actually doing, and cuts it off on its own when the session turns destructive, whether it belongs to a tired engineer, a mistyped command, a leaked key, or an agent. It treats them the same, because in every story on this page the account could not tell them apart either. Pair that with backups kept outside the production blast radius, and the next viral deletion story is a scare instead of a headline.

The takeaway

These incidents get told as horror stories, individual and unlucky, a bad night for one engineer or one agent. They are not individual. They are the same design flaw wearing different clothes, decade after decade, and the companies that stopped repeating it did not hire more careful people. They put something between the destructive decision and the destruction, and they kept a copy of everything somewhere the disaster could not reach.

Frequently asked questions

What is the most famous production deletion incident? GitLab’s January 2017 outage, where an engineer wiped the primary database instead of the secondary, lost about 300 GB, and then watched all five backup methods fail. They recovered only from a manual snapshot taken six hours earlier by chance.

Don’t backups protect you? Often not. GitLab’s five methods failed, several silently, and Pixar’s Toy Story 2 backups were incomplete. Backups fail when they share a credential or control plane with production and when nobody tests a restore. A backup you have never restored from is a hope, not a plan.

Have AI agents done this too? Yes, and they look just like the human cases. An AI agent deleted a production database during a code freeze in 2025, and another deleted a database and its backups in about nine seconds in 2026. Same shape, faster clock.

If a destructive command against your production, from anyone or anything with access, is the thing you would rather not lie awake over, see how Korvalis handles it or join the waitlist.