Three hours of Cloudflare 5xx: how one config change took down half the web

On November 18, 2025, starting around 11:20 UTC, visitors to a large number of websites saw 5xx error pages. They had one thing in common: Cloudflare. The outage lasted about three hours, and recovery was gradual rather than a single flip.

Cloudflare published a postmortem afterwards stating clearly that it was not a cyberattack and not a compromise of configuration.

The root cause was not an attack

According to the postmortem, the trigger was an internal configuration change: a configuration file used by a Bot Management feature is generated automatically, and after a permissions change it was written with content far larger than expected. The core proxy hit a latent bug while parsing it, the process failed, and traffic through that proxy started erroring.

Three ingredients had to line up: an auto-generated config, an unanticipated size, and a parsing path that had never been exercised at real scale. Any one of them alone looks harmless.

Why the blast radius was so large

Cloudflare is a single point by design: global traffic passes through it. That is not a mistake — reverse proxying, WAF and CDN all require traffic to flow through the same layer. The cost is that any failure in that layer is amplified into “half the internet is down”.

Two propagation details made it worse:

• Configuration is distributed globally. A bad config does not hit one region; it spreads as edge nodes sync;
• Rollback is slower than rollout. Rollout is automated; rollback requires a human to decide, confirm, and walk it back layer by layer. The gap is the outage duration.

Three engineering lessons

1. Canary your configuration. Any global config rollout should hit a small subset of nodes first, with metrics checked before scaling up. Starting at 1% would have cut the cost by two orders of magnitude;
2. Put hard limits on config. Auto-generated configuration needs a size ceiling and validation; refuse to distribute beyond it. Plenty of incidents are not “the value is wrong” but “the value is enormous”;
3. Decide the degradation path in advance. Fail-open versus fail-closed is a product decision, but it must be made ahead of time and verified in drills.

The judgement

There was no attacker and no zero-day in this incident — just an automatically generated config file and a code path nobody had stepped on. The fragile point of modern infrastructure is often not the attack surface but the blast radius of “normal operations”. The lesson for self-hosted services is direct: treat any globally applied configuration change as a deployment.

Cover image: Cloudflare postmortem artwork (blog.cloudflare.com).