One database change took down half the internet
On November 18, 2025, a big chunk of the internet stopped working. X went down. ChatGPT went down. Shopping carts, dashboards, login screens, all throwing errors at the same time. For a few hours it felt like something huge had been attacked.
It had not. There was no hacker. According to Cloudflare’s own writeup, the outage started with a routine change to database permissions. That change made an internal config file quietly double in size. A piece of code downstream expected that file to be small, could not handle the bigger version, and crashed the process it was running in. Because that process sat in the critical path for traffic, the crash spread across the network.
No malice. No breach. One internal change, doing its job in a way nobody predicted, and a service that a huge slice of the web depends on fell over.
Why this should bother you more, not less
It is tempting to file this under “big company problem” and move on. Cloudflare runs infrastructure at a scale most of us will never touch, and the specifics here are about their internal systems, not yours. Korvalis would not have prevented the Cloudflare outage, and I am not going to pretend otherwise.
But strip away the scale and look at the shape of it. Someone made a normal change. The change was allowed. The blast radius was enormous. Nobody set out to break anything.
That shape is the most common way things break in the cloud, and it gets worse the smaller you are. Cloudflare had the tooling to detect the problem, roll back a config file, and write a detailed post-mortem the same day. On a ten-person team, the person who runs the change and the person who notices the damage are usually the same tired human, and there is no second version of the file to roll back to.
Most cloud damage is self-inflicted
The scary story is always the hacker. The boring truth is that most outages and most data loss come from inside: a wrong command, a permission that was broader than anyone remembered, a script pointed at production when it meant staging.
The most upvoted programming thread of all time is a new engineer writing “accidentally destroyed production database on first day.” The verdict from thousands of replies was not “fire the kid.” It was “if a first-day hire can delete production, the system failed, not the person.” Even GitLab has publicly walked through the time an engineer wiped the wrong database, backups and all.
These are careful people at serious companies. The problem was never carelessness. It was that the system trusted a single action to be safe, and it wasn’t.
What actually helps
You cannot prevent every mistake. People will always run the wrong command eventually. What you can change is what happens next: whether one bad action quietly succeeds, or whether something is watching that can catch it and stop it before the damage spreads.
That is the entire idea behind Korvalis. It sits inside your own cloud, watches what your sessions are actually doing, and when a session turns destructive it shuts it down on its own, in seconds, whoever or whatever is behind it. Not by matching commands against a blocklist, but by reading what the session is trying to do. You can read more about how that detection works, or the specific case of preventing accidental deletion across AWS, Azure, and GCP.
The Cloudflare outage will get a hundred think-pieces about resilience and rollout discipline. Fair enough. But the quieter lesson is the one that applies to every team, not just the ones running the backbone of the internet: the change that takes you down usually comes from the inside, on an ordinary day, from someone who meant well.
If that is a risk you would rather have something watching, join the waitlist and we will be in touch.