One regular expression takes down Cloudflare
A firewall rule deployed globally in seconds exhausts CPU on every edge server.
The setup at Cloudflare
Cloudflare's web application firewall (WAF) runs managed rules on every HTTP request at the edge. Rule changes are pushed through a distributed key-value store that reaches the whole network in seconds, because new threats need fast responses. New rules can run in a simulate mode that logs matches without blocking.
How it was configured
WAF: Lua, using PCRE (a backtracking regex engine)
rule deploys: global, in seconds, after CI tests pass
simulate mode: rule runs on real traffic, nothing blockedWhat you know
- The team handled 476 rule change requests in the 60 days before the incident.
- Simulate mode still executes the rule on every matching request.
What happened next
Make the call
2 questions · first-try answers earn the most XP
What actually happened
The new rule contained a regular expression with a pattern like .*(?:.*=.*) that backtracked enormously on some inputs. Because Cloudflare's WAF used a backtracking engine with no runtime limit, it consumed every CPU core serving HTTP and HTTPS.
Engineers identified the WAF from CPU profiles and logs, and at 14:02 UTC proposed a global kill of the WAF managed rules. Traffic recovered, and the outage lasted 27 minutes. A protection that would have limited regex CPU use had been removed by mistake in a refactor weeks earlier.
Cloudflare's write-up lists the gaps: the engine had no complexity guarantees, the test suite could not detect excessive CPU use, and the process allowed a non-emergency rule change to go global without a staged rollout. Fixes included restoring the CPU protection, inspecting all 3,868 rules, profiling rules in tests, moving to RE2 or the Rust regex engine, and staged rollouts for rule changes.
Sources
Lessons to take away
- Anything that can change behavior globally in seconds needs a staged rollout path, even if it also has an emergency fast path.
- Kill switches for each risky feature turn a long outage into a short one.
- Tests should measure resource use, not only correctness: CPU per rule would have flagged this.
- Protections removed in a refactor are easy to miss; keep tests that prove they still exist.
Variations to think through
Change one fact and decide again. There is no single right answer here.
- An active attack needs a new rule worldwide within five minutes. How do you keep that fast path safe?
- The rule caused a 20% CPU increase instead of 100%. How would you notice before it became an outage?
- Your own API gateway lets customers upload regex-based rules. How do you protect it?