nginx connect() failed (111: Connection refused) During Deployment Rollouts
A connect() failed (111: Connection refused) error in nginx ingress means nginx opened a TCP connection to an upstream pod IP and the pod actively refused it: nothing was listening on that port. During Kubernetes deployment rollouts, this is the signature of traffic being routed to pods that are restarting or not yet ready. Users see 502s. A short burst that self-resolves is expected turbulence. A burst on every deploy means your rollouts are not actually zero-downtime.
maxUnavailable: 0. Persistent 111s outside a rollout are a different failure class: a selector mismatch or a process that never bound its port.
- errno 111 (ECONNREFUSED)
- Kernel-level TCP refusal. The host is reachable but no process is listening on the target port.
- Readiness probe
- The gate that decides whether a pod receives Service traffic. No probe means traffic arrives the moment the container starts, ready or not.
- Endpoints / EndpointSlice
- The live list of pod IPs behind a Service. nginx ingress watches this list to build its upstream set.
- maxUnavailable / maxSurge
- Rolling update knobs controlling how many replicas can be down or extra during a rollout.
- preStop hook
- A command run before SIGTERM, commonly
sleep 5, giving the endpoint list time to propagate before the pod stops listening.
What does this incident look like?
The smoking gun is in the nginx ingress controller logs:
2026/07/16 18:25:06 [error] 1378#1378: *3938790 connect() failed (111: Connection refused)
while connecting to upstream, client: 192.168.2.1, server: app.internal.example,
request: "GET /workspaces/ws-example HTTP/1.1",
upstream: "http://10.1.254.73:4000/workspaces/ws-example"Real example from our own development cluster:
- Trigger: rolling restart of four interdependent deployments (api, mcp, mobius, ui) in rapid succession at 18:24 UTC
- Pattern: 36
connect() failed (111)errors in a roughly 90-second window, all from the ingress controller, all targeting upstreams mid-restart - Secondary effect: 2 MCP tool call failures in a dependent service whose upstream was restarting at the same moment
- Resolution: self-resolved as pods passed readiness; error rate returned to baseline within 2 minutes
This is the textbook shape. The errors cluster tightly around the deploy event, hit multiple upstreams, and clear on their own. The tell that separates this from a real outage is the correlation with the rollout, not the error text itself.
connect() failed (110: Connection timed out) is a network path or firewall problem, not a refusal. no live upstreams while connecting to upstream means nginx has zero endpoints at all, versus a refusal from a listed endpoint. 502 Bad Gateway is the client-visible symptom this error produces, but 502 has many other causes. 503 Service Temporarily Unavailable means nginx knows there is no backend, versus nginx trying one and failing.
Why does this happen?
In roughly descending order of frequency:
1. Traffic routed to pods before they are ready
Missing readiness probes, or probes that only check process-up rather than serving-ready. Kubernetes adds the pod IP to the endpoint list, nginx proxies to it, and the app has not bound its port yet.
2. Endpoint propagation lag on shutdown
The mirror image. A pod gets SIGTERM and closes its listener immediately, but nginx has not yet removed it from the upstream set. For a few hundred milliseconds to a few seconds, nginx proxies to a dead port. Fix: a preStop sleep 5 plus graceful shutdown in the app.
3. Rollout strategy too aggressive
maxUnavailable allows too many replicas down at once, or the deployment runs a single replica, guaranteeing a window with zero ready pods on every deploy.
4. Interdependent services bounced simultaneously
Restarting api, mcp, mobius, and ui at the same moment multiplies the not-ready window and produces secondary failures (the MCP tool call failures above). Stagger restarts of services that call each other.
5. Persistent case: nothing will ever listen
The Service selector does not match pod labels, targetPort does not match the container port, or the process crashes after container start. If 111s continue outside any rollout, stop looking at rollout mechanics.
How severe is it?
| Pattern | Severity | Why |
|---|---|---|
| Burst during rollout, clears < 2 min | Minor | Transient, but users saw 502s during the window |
| Burst on every rollout | Minor → Major | Deploys are not zero-downtime; impact scales with deploy frequency |
| Single-replica critical service | Major | Guaranteed downtime window on every deploy |
| Persistent outside rollouts | Major | Selector mismatch, port mismatch, or crash loop: nothing is listening |
How do I debug and remediate?
1. Correlate the error window with deploy events
kubectl rollout history deployment/<name> and cluster events. If the burst brackets a rollout and cleared, you have the transient class.
2. Confirm current pod health
kubectl get pods -n <namespace>All pods Running with full READY counts, and nginx error ratio back under your baseline (< 2% is a reasonable default).
3. Verify readiness probes gate on serving readiness
A probe that returns 200 before the app can serve real traffic is worse than useless: it converts a visible failure into an intermittent one. Probe the actual serving path.
4. Add a preStop delay and graceful shutdown
lifecycle:
preStop:
exec:
command: ["sleep", "5"]Plus SIGTERM handling in the app: stop accepting new connections, drain in-flight requests, then exit.
5. Tighten the rollout strategy
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0A new pod must be ready before an old one is removed. For single-replica deployments this is the only path to zero-downtime deploys short of adding a replica.
6. Stagger interdependent restarts
If service A calls service B, restarting both simultaneously guarantees A’s calls fail even after A is healthy. Order the rollout, or accept the turbulence window and annotate it explicitly.
7. Alert on the ratio, not the event
A single 111 is noise. Alert on ingress error ratio above baseline sustained over minutes, and suppress or annotate during known deploy windows.
8. If persistent: check the wiring
kubectl get endpoints <service>. Empty endpoints means selector mismatch. Populated endpoints with continuing 111s means the container port and targetPort disagree, or the process is not binding.
How does Dstl8 detect this?
Dstl8 groups the burst into one incident, names the cause, and connects the secondary failures. Here is the actual detection from our own development cluster:
Three things to notice:
- One incident, not 36 alerts. The burst is grouped and named as a rollout event, so on-call reads a story instead of a page storm.
- The causal link is made automatically. The restart of four specific deployments at 18:24 is named as the cause of the ingress errors. Nobody grepped timestamps across two log sources.
- The cascade is attached, not orphaned. The 2 MCP tool call failures in a dependent service land inside the same incident with the same root cause, instead of opening a second mystery.
Related patterns
References
- Kubernetes readiness probes: kubernetes.io/docs: liveness, readiness and startup probes
- Kubernetes rolling update strategy: kubernetes.io/docs: Deployment rolling updates
- Pod termination lifecycle: kubernetes.io/docs: Pod termination
- ingress-nginx documentation: kubernetes.github.io/ingress-nginx
Catch rollout turbulence before your users refresh into a 502.
Dstl8 groups the error burst, ties it to the deploy that caused it, and attaches the downstream cascade to the same root cause. One incident, not 36 alerts.