Kubernetes rollout 503s: coordinate endpoint withdrawal and shutdown

In 2025, I investigated 503 Service Unavailable responses during Kubernetes rollouts. My first article framed the fix as a choice between handling SIGTERM in the application and adding Kubernetes lifecycle hooks. That split is misleading. A pod rollout is a handoff between traffic routing and process shutdown, and those steps do not happen in one place or at one instant.

The application must stop taking new work and finish work it already accepted. Kubernetes and the service routing layer must stop choosing the terminating pod. A preStop delay or a readiness probe can help with part of that handoff, but neither proves the whole route has drained.

What happens when a pod terminates

When a pod is deleted, Kubernetes starts its graceful termination period. The kubelet runs a configured preStop hook before sending the container’s stop signal. The grace period countdown includes the time spent in the hook. After the hook completes, the runtime sends the stop signal, normally SIGTERM; any process still running when the grace period expires is killed. The pod lifecycle documentation and container hook documentation describe this sequence.

On Kubernetes 1.26 and later, the control plane also updates the pod’s EndpointSlice state. A terminating endpoint is marked terminating and is normally not used for regular service traffic. There is a special case: service proxies may still use an endpoint that is both serving and terminating when every available endpoint is terminating. The EndpointSlice documentation describes those conditions.

That means “the pod is terminating” and “no request can reach it” are not the same event. There may also be a separate ingress or load balancer outside the Kubernetes Service. Its connection draining behaviour has to be checked on its own.

What each mechanism can and cannot do

The application owns its open work

On SIGTERM, an application can enter a draining state and make its readiness endpoint fail. Kubernetes observes that failure on a later probe. Pod deletion also updates the EndpointSlice independently. The application can then stop accepting new work, wait for in flight requests, and close its dependencies. Node’s server.close() stops a server from accepting new connections and waits for connections that are still handling a request or response to finish. The exact shutdown steps depend on the server framework and the resources the process owns.

The process needs a bounded shutdown path. If the database pool, queue consumer or an open request never drains, the kubelet eventually reaches the end of the grace period and kills the container. Set the grace period from measured drain time, including any preStop work, request duration and resource cleanup. The default is 30 seconds, but a default is not evidence that 30 seconds fits the application.

Readiness reports whether a pod should receive service traffic

A failed readiness probe marks the pod unready, so matching Services stop selecting it as a ready endpoint. Readiness is useful when an application is starting, overloaded or entering its own drain state. It does not by itself drain a connection already in progress, and a probe interval is not a precise promise about when an external load balancer stops routing.

Kubernetes also changes the EndpointSlice state when a pod is deleted. That is why a readiness probe is not a substitute for graceful application shutdown. The probe documentation explains the role of readiness and the endpoint updates on pod deletion.

preStop delays the signal, within the same budget

A preStop hook can keep the process alive for a measured interval before Kubernetes sends the stop signal. A short delay can give service routing or an external load balancer time to withdraw the endpoint while the application is still able to answer requests. In this pattern, the hook provides that delay before SIGTERM; do not add the same delay again in the signal handler unless measurements show a separate interval is needed.

The delay must come from observed propagation time. A fixed sleep copied from an example can be too short for the real route or waste most of the grace period. The hook also runs before the application receives SIGTERM, so the application cannot begin its own signal handling until the hook finishes.

A safer shutdown sequence

For a service that sees rollout errors, start with one drain delay and measure each boundary. If the delay is in preStop, the order is:

1. Kubernetes runs the preStop hook while the application can still respond.
2. After the measured delay, Kubernetes sends SIGTERM.
3. The application records the signal, enters its draining state, stops accepting new connections and waits for in flight requests to finish.
4. The application closes database pools, queue consumers and other resources.
5. The process exits before the pod’s termination grace period expires.

If there is no preStop delay, the application can instead start a measured routing wait after SIGTERM, then stop accepting connections and drain requests. Do not use both waits by default. If the application uses a readiness endpoint, make it fail as soon as draining starts and account for the probe interval. Keep the routing delay and request drain budget separate in your measurements. A long request that started before endpoint withdrawal still needs time to finish.

Do not use liveness as a shutdown switch. Liveness answers whether Kubernetes should restart a stuck container. Readiness answers whether the container should receive Service traffic. Mixing them can restart healthy pods during load or slow shutdown.

Find which part produced the 503

Reproduce the rollout while watching the request and pod timelines together. Log when the process receives SIGTERM, when it marks itself unready, when it stops accepting connections, how many requests remain, and when it exits. Watch the pod’s EndpointSlice conditions during the same interval. If traffic passes through an ingress or cloud load balancer, inspect that layer’s target health and drain settings too.

The response source matters. A 503 from the application means something different from a 503 generated by an ingress with no healthy upstreams. A connection reset during shutdown is different again. Without those signals, changing the probe period or adding a sleep can hide one symptom while leaving the actual handoff unmeasured.

The rule I use now is simple: let the application own its work, let readiness describe whether it can take new work, and treat preStop as a measured delay for the routing layer. Then verify the full path through the same Service or load balancer clients use.

The original Medium article is the shorter version I wrote in 2025.