🔥 Partial Endpoint Failure

ingress-nginx `upstream timed out (110)` Returning 504 Errors

upstream timed out (110: Operation timed out) means ingress-nginx waited the full proxy-read-timeout for response headers from a backend pod and gave up, returning 504 to the client. The controller is working correctly. The question worth answering is whether every endpoint is slow or one endpoint is bad, and the access log answers it directly.

Platforms: ingress-nginx · Kubernetes · NGINX Severity: Minor to Major (depends on endpoint spread) Status: Upstream latency or endpoint health failure Updated: August 24, 2026

By Bob Quillin

Co-Founder, ControlTheory · Last reviewed August 24, 2026

TL;DRA 504 from ingress-nginx is a timeout waiting on a backend, defaulting to 60 seconds. The diagnostic value is in the access log fields, not the error log: upstream_addr names the pod that timed out and upstream_response_time shows how long it took. When a 504 to one pod IP is followed seconds later by a 200 to a different pod IP, you have a single bad endpoint rather than a saturated service, and restarting the deployment will appear to fix it while guaranteeing a recurrence.

What does an ingress-nginx upstream timeout look like?

Two log lines, and the second one carries more information than the first.

The error log states the failure:

[error] 29#29: *75267541 upstream timed out (110: Operation timed out) while
  reading response header from upstream, client: 203.0.113.10,
  server: api.example.com, request: "POST /v1/chat/completions HTTP/1.1",
  upstream: "http://10.20.40.10:4000/v1/chat/completions"

The access log gives you the diagnosis:

"POST /v1/chat/completions HTTP/1.1" 504 160 "-" 131384 60.005
  [app-backend-4000] [] 10.20.40.10:4000 0 60.001 504
  "POST /v1/chat/completions HTTP/1.1" 200 11487 "-" 40360 38.792
  [app-backend-4000] [] 10.20.55.30:4000 11487 38.791 200

Three things fall out of that pair immediately. The 60.001 upstream response time is exactly the default proxy-read-timeout, so this is the timeout firing rather than a connection error. The two requests went to different pod IPs and only one failed. And the successful one took 38.791 seconds, which is already inside two thirds of the timeout budget.

So this is not a service that is down. It is a service whose normal latency runs close enough to the ceiling that one degraded endpoint pushes requests over it.

Not to be confused with A 502 with connect() failed (111: Connection refused) is a different failure. That means no process accepted the connection, typically because a pod is starting or terminating during a rollout. A 504 means the connection succeeded and the response never arrived. Refused points at pod lifecycle, timed out points at pod performance.

Why does one endpoint time out while others succeed?

ingress-nginx load balances across the endpoints of a service. Every endpoint that is Ready receives traffic, and readiness is defined by a probe that often measures whether the process answers rather than whether it can do useful work.

What makes one endpoint slow while its peers are fine:

Uneven request cost. One pod picked up several expensive requests and its queue is deep. Long-running requests, model inference, and large uploads all produce this.
Resource throttling. CPU limits throttling one pod, or memory pressure driving GC pauses that the other replicas are not experiencing.
Connection pool exhaustion inside the pod, so new requests queue behind saturated capacity while the health endpoint still answers instantly.
A noisy neighbor on the node, contending for CPU or IO.
A readiness probe that does not reflect capacity. A liveness check on /healthz returns instantly regardless of how deep the work queue is, so an overloaded pod stays in rotation.

Cause five is what turns any of the first four into user-visible 504s. Kubernetes will keep routing to a pod that cannot serve, because nothing has told it otherwise.

Why does it appear to resolve and then come back?

This is the part worth understanding before declaring an incident closed.

Traffic distribution across endpoints is not even, and a degraded pod does not fail every request. A quiet period, a completed batch of expensive work, or simple routing luck produces a clean stretch of 200s, and the incident looks resolved. Then the pod picks up another expensive request and the 504s return.

In one real occurrence, errors ran from 20:31, appeared fully resolved at 01:18 with all subsequent requests returning 200, and then new 504s appeared at 01:47. The resolution was an artifact of the sampling window, not a change in the system.

Two practical consequences. Do not close on a clean five-minute window when baseline latency sits near the timeout; the window has to exceed the interval between expensive requests. And restarting the deployment will always appear to work, because it replaces the degraded pod, while leaving the condition that degraded it untouched.

How do I diagnose and fix ingress-nginx 504s?

1. Group the 504s by upstream_addr

kubectl -n ingress-nginx logs -l app.kubernetes.io/name=ingress-nginx --tail=5000 \
    | grep ' 504 ' | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+:[0-9]+' \
    | sort | uniq -c | sort -rn

Concentrated on one address means one bad endpoint. Spread evenly means the service is saturated. These are different problems and the fixes do not overlap.

2. Compare upstream_response_time on successes

If successful requests are already consuming most of the timeout budget, the service is operating with no headroom and any degradation becomes a 504. That is a capacity finding, not a timeout finding, and raising the timeout only defers it.

3. Inspect the specific pod

kubectl get endpoints <service> -o wide
  kubectl top pod -l app=<label>
  kubectl describe pod <pod> | grep -A 5 -iE 'throttl|limits|restart'

CPU throttling on one replica and not others confirms cause two.

4. Match the timeout to real request duration

For genuinely long-running endpoints, the default 60 seconds may simply be wrong:

nginx.ingress.kubernetes.io/proxy-read-timeout: "180"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "180"

Set it per-Ingress rather than globally. A global increase means every slow request everywhere now holds a worker for three minutes.

5. Make readiness reflect capacity

A readiness probe that fails when the work queue is deep removes an overloaded pod from rotation until it recovers. This is the fix that actually prevents recurrence, and it is the one most often skipped because the probe currently passes.

6. Verify over a window longer than the failure interval

Given how these incidents fake their own resolution, confirm over a period that spans several expensive requests, not five quiet minutes.

How does Dstl8 detect this?

A 6.5% error rate on one ingress path does not cross a threshold, and the incident hides its own recurrence behind clean windows. Dstl8 tracks the pattern rather than the rate, and continues evaluating after an apparent recovery, so a return is attached to the original incident instead of arriving as a new unrelated one.

Real Detection

Incident Resolved PLATFORM

ingress-nginx upstream timeouts persisting with 504 errors

Controller upstream timeouts to backend pods, 6.5% error rate across 92 logs. Problem appeared to resolve with all subsequent requests returning 200 OK, then new errors emerged 29 minutes later showing 60-second timeouts against a single upstream address, indicating the problem did not actually resolve. Correlated across 2 deployments, 2 services, 3 pods, 3 hosts. No alert rule configured.

Started

20:45

Aug 6

Span

5h26m

Severity

Minor

Events

Note what the narrative does: it revises its own earlier conclusion, stating that the apparent resolution was contradicted by later evidence. That is the behavior you want from an analysis layer and the opposite of an alert that fires, clears, and fires again as three unrelated pages.

Frequently asked questions

What does upstream timed out (110: Operation timed out) mean in ingress-nginx?

ingress-nginx waited the full proxy-read-timeout, 60 seconds by default, for response headers from a backend pod and gave up, returning 504 to the client. The connection succeeded but the response never arrived, which points at backend performance rather than pod lifecycle.

How do I tell if one pod is causing my 504 errors?

Group the 504 responses by upstream_addr in the ingress-nginx access log. If the failures concentrate on a single pod IP while other IPs return 200 for the same path, one endpoint is degraded. If they spread evenly across all endpoints, the service is saturated.

Why do ingress-nginx 504 errors seem to resolve and then return?

A degraded pod does not fail every request. Quiet periods or routing luck produce clean windows that look like resolution, then the pod picks up another expensive request and the 504s return. Restarting the deployment appears to fix it because it replaces the pod, while leaving the underlying condition intact.

Related patterns

References

ingress-nginx docs: Annotations reference
ingress-nginx docs: Log format
NGINX docs: proxy_read_timeout
Kubernetes docs: Configure liveness, readiness and startup probes

An incident that revises its own conclusion.

Dstl8 keeps evaluating after an apparent recovery, so a returning failure attaches to the original incident instead of arriving as a new one. No rules to write, no thresholds to tune.

Start Free 14-Day Trial →

ingress-nginx upstream timed out (110) Returning 504 Errors