SeniorKubernetes
Every deploy drops a few hundred requests
Rolling updates look clean, yet clients see connection errors during each one.
The setup today
A service behind a Kubernetes Service and an ingress deploys several times a day with a rolling update. When a pod is asked to stop, the app exits as soon as it receives SIGTERM.
Current configuration
strategy: RollingUpdate (maxUnavailable 0, maxSurge 25%)
terminationGracePeriodSeconds: 30 # default
app: on SIGTERM -> exit immediately
no preStop hook, no PodDisruptionBudgetWhat you know
- Each deploy causes 100 to 400 failed requests (connection refused or reset).
- Node upgrades sometimes take down two of the three pods at once.
The new requirement
Deploys and node maintenance must cause zero failed requests.
Make the call
2 questions · first-try answers earn the most XP
What a strong answer also covers
- Keep connections short or send Connection: close during shutdown, so keep-alive clients reconnect elsewhere.
- Make sure preStop delay plus drain time fits inside terminationGracePeriodSeconds.
- Clients and proxies should retry idempotent requests once, as a safety net.
- Measure failed requests per deploy as a metric, so regressions are caught automatically.
Variations to think through
Change one fact and decide again. There is no single right answer here.
- Some requests are file uploads that take two minutes. How long should the grace period be?
- The service holds WebSocket connections for hours. How do you deploy without dropping them all at once?
- A PodDisruptionBudget blocks a node upgrade forever. What went wrong, and how do you fix it?