kube-apiserver error dialing backend: use of closed network connection

error dialing backend means the Kubernetes API server could not open or reuse a connection to a kubelet on port 10250. The API server proxies kubectl logs, exec, attach, port-forward, and metrics requests through that connection, so when it fails those requests return 500 to the client. Ordinary pod traffic is unaffected, which makes the failure look like a tooling problem rather than a cluster one.

What does the error dialing backend failure look like?

The API server logs it as an error it could not convert into a proper status object, which is why the wrapper text is unusually verbose:

E0727 18:14:17.456046 1 status.go:71] apiserver received an error that is not
  an metav1.Status: &errors.errorString{s:"error dialing backend: write tcp
  10.20.30.2:32980->10.20.30.5:10250: use of closed network connection"}:
  error dialing backend: write tcp 10.20.30.2:32980->10.20.30.5:10250:
  use of closed network connection
  

Read the socket pair. The left address is the API server, the right is the kubelet on the target node at port 10250. That tells you which node to look at without correlating anything else.

The three tails and what each means:

The user-facing symptom is a 500 from kubectl logs or kubectl exec, often on one node and not others, and often intermittently.

Why does the connection get closed?

use of closed network connection is a Go runtime error meaning a write was attempted on a socket already closed locally. The API server maintains pooled connections to kubelets for streaming and proxy requests. When something invalidates one without the pool noticing, the next request through it fails immediately.

Realistic causes:

1. kubelet restarted. A config reload, a version upgrade, or a crash. Every pooled connection to that node becomes invalid at once.
2. A middlebox idle timeout. A NAT gateway, load balancer, or firewall between the control plane and worker nodes silently dropping idle TCP connections. This is the most common cause of the intermittent version, because the failure only appears after a connection has been idle past the timeout.
3. konnectivity or egress-selector proxy failure. On clusters using an egress selector, the tunnel agent restarting produces exactly this.
4. Node network interruption. A CNI restart, a node reboot, or an overlay blip.
5. kubelet serving certificate rotation without a graceful reload.

Cause two is worth checking first when the failure is sporadic and correlates with quiet periods rather than load.

What actually breaks?

A narrow set of operations, all of which happen to be the ones engineers reach for when debugging something else:

• kubectl logs, exec, attach, port-forward return 500 for pods on the affected node
• metrics-server scrapes fail, leaving gaps in kubectl top and starving HPA of data, which can freeze autoscaling without any autoscaler error
• Webhook and aggregated API calls routed through the same proxy path fail
• Running workloads are completely unaffected. Pod-to-pod traffic never touches this path.

The combination is unusually frustrating: the cluster is fine, but the tools you would use to confirm that it is fine are the ones failing. At a low error rate it reads as flakiness in kubectl rather than a cluster condition.

How do I diagnose and fix it?

1. Identify the target node from the socket pair

The right-hand address in the error is the kubelet. Map it to a node:

kubectl get nodes -o wide | grep 10.20.30.5
  

2. Check kubelet health on that node

kubectl describe node <node> | head -30
  systemctl status kubelet          # on the node
  journalctl -u kubelet --since '30 min ago' | tail -50
  

A restart in the window explains a burst of closed-connection errors that then stops.

3. Test reachability of port 10250 from the control plane

curl -k -m 5 https://10.20.30.5:10250/healthz
  

connection refused means kubelet is not listening. A hang means the path is filtered. A 401 is a healthy result here: the port answered and rejected an unauthenticated request, which is the expected behavior.

4. Rule out an idle timeout in the path

If the failure is intermittent and does not correlate with kubelet restarts, look for a NAT gateway, cloud load balancer, or firewall between the control plane and nodes, and compare its idle timeout against the API server’s connection reuse. A timeout shorter than the reuse window produces this error indefinitely and at random.

5. Check the tunnel, if you use one

kubectl -n kube-system get pods -l k8s-app=konnectivity-agent
  kubectl -n kube-system logs -l k8s-app=konnectivity-agent --tail=50
  

Agent restarts invalidate every proxied connection through them.

6. Confirm recovery against the failing operation

kubectl logs --tail=1 <pod-on-that-node>
  kubectl top node <node>
  

Health endpoints will pass even while proxying is broken, so verify with the operation that was actually failing.

How does Dstl8 detect this?

At a 5% error rate this is one error in twenty requests, well under any threshold anyone would set, and the failing operations are ones humans invoke rather than services. Dstl8 baselines each service against its own history, so an error pattern appearing in kube-apiserver logs where none existed becomes an incident regardless of how small the rate is.

The incident correlates the API server pod with the control plane host and preserves the exact socket pair from the log line, which is what identifies the target node. A 5% error rate on a low-volume endpoint is invisible to rate-based alerting and obvious to a baseline.

Frequently asked questions

Related patterns

References

• Kubernetes docs: Master-node communication
• Kubernetes docs: kubelet authentication and authorization
• Kubernetes docs: Set up konnectivity service
• Kubernetes docs: Ports and protocols