Reddit's Pi Day Kubernetes upgrade
An in-place upgrade of Reddit's oldest cluster quietly breaks pod networking.
The setup at Reddit
Reddit ran its oldest and largest Kubernetes cluster, which served the legacy core of the site, on Kubernetes 1.23. Pod networking used Calico with BGP route reflectors, a setup configured years earlier and unique to that cluster. Upgrades were done in place.
How it was configured
cluster: oldest and largest, Kubernetes 1.23, upgraded in place
CNI: Calico, BGP through route reflectors
route reflectors chosen by a node label selectorWhat you know
- Kubernetes does not support downgrading a cluster in place.
- The cluster's networking setup had no twin anywhere else to test against.
What happened next
Make the call
2 questions · first-try answers earn the most XP
What actually happened
The 1.24 upgrade removed the old node-role.kubernetes.io/master label from control plane nodes (kubeadm's upgrade does this, as the 1.24 release notes describe). The cluster's Calico route reflectors were selected by that old label, so after the upgrade no nodes matched, BGP routes stopped being distributed, and pod networking failed.
Reddit's engineers wrote that it took hours to decide that a rollback, a high-risk action on its own, was the right course. They restored the cluster from a backup, and found the subtle cause only afterwards. Reports put the outage at 314 minutes, fittingly on Pi Day.
In the post-incident write-up, Reddit called Kubernetes version and component upgrades a big footgun for them and committed to making cluster upgrades safe.
Sources
Lessons to take away
- One-of-a-kind clusters are risky: standardize so an upgrade is tested on an identical cluster first.
- Rehearse restores, including certificates and add-on configuration, before you need one under pressure.
- Prefer blue-green cluster upgrades (build a new cluster, move traffic) for critical clusters, so rollback is a traffic switch.
- Deciding to roll back is itself hard; agree on criteria and time limits in advance.
Variations to think through
Change one fact and decide again. There is no single right answer here.
- You can afford a second cluster for a few weeks. How would you upgrade by moving traffic instead of in place?
- The upgrade breaks only 10% of nodes. Do you stop, continue, or roll back?
- How would you detect, within a minute, that pod-to-pod networking is broken?