leader election lost Across Every Kubernetes Controller

leader election lost means a controller failed to renew its lease with the Kubernetes API server within the renew deadline, so it exits deliberately to let another replica take over. A single controller reporting it is routine. A dozen unrelated controllers reporting it within the same few seconds is not a controller problem at all: it means the API server became briefly unreachable and every leased component reacted at once.

What does the cluster-wide signature look like?

Three different components, three different log formats, one shared cause. Same six-second window:

{"level":"error","ts":"2026-08-20T10:10:33.001Z","logger":"setup",
   "msg":"problem running manager","error":"leader election lost"}
  

[error]: #0 Exception 'HTTP status code , Timed out connecting to server'
  encountered fetching pod metadata from Kubernetes API v1 endpoint
  https://10.20.0.1:443/api
  

[ERROR][1] client.go 290: Error getting cluster information
  config ClusterInformation="default" error=Get "https://10.20.0.1:443/apis/
  crd.projectcalico.org/v1/clusterinformations/default": context deadline exceeded
  

The tell is not any one line. It is that the GitOps controllers, the CNI controller, and the log shipper all failed against the same API server endpoint at the same instant, in three different vocabularies. No single service’s logs would reveal this.

In a real occurrence, at least twelve components across four nodes reported failures in the same window, and every one of them recovered without intervention.

Why does an API server blip take down every controller at once?

Leader election in controller-runtime uses a Lease object in the API. The leader renews it on an interval, and if renewal fails past the renew deadline, the manager shuts down rather than risk two active leaders writing conflicting state. That is correct and deliberate behavior.

The defaults are tight. Lease duration is measured in seconds, not minutes, so a control plane interruption of ten to fifteen seconds is enough to expire leases cluster-wide. Every component holding one reacts in the same window.

What produces the interruption:

1. API server restart or rollout. A config change, a certificate rotation, or a control plane upgrade.
2. etcd latency spike. Disk contention on the etcd volume is the most common cause. Every write, including lease renewals, queues behind it.
3. Control plane node pressure. CPU or memory contention slowing request handling.
4. Load balancer or network interruption in front of the API server endpoint.
5. Request flood. A misbehaving client exhausting priority-and-fairness capacity so that lease renewals queue.

Cause two is worth checking first, because etcd latency degrades gradually and produces repeated brief disruptions rather than one clean outage.

What breaks during the window?

Less than the log volume suggests, and that is worth knowing before anyone escalates.

• Reconciliation pauses. GitOps controllers stop applying, so a deploy landing in that window is delayed rather than lost.
• Controller pods restart. Restart counts increment across many deployments at once, which looks alarming on a dashboard the next morning.
• Policy and network state stop updating, though existing programmed state stays in place.
• Telemetry gaps. Log shippers that enrich records with pod metadata drop the enrichment or the records, leaving a hole in exactly the window you will later want to investigate.
• Running workloads are unaffected. Pods keep serving. No user-facing impact.

The lasting cost is usually the telemetry gap. The event that caused the disruption is least likely to be captured, because the shippers were failing at the time.

How do I investigate a cluster-wide leader election loss?

1. Confirm simultaneity before anything else

kubectl get events -A --sort-by=.lastTimestamp | grep -i leader
  kubectl get pods -A --sort-by=.status.startTime | tail -30
  

A cluster of restarts within the same few seconds confirms a shared cause and rules out per-controller investigation.

2. Check the API server for that window

kubectl -n kube-system logs kube-apiserver-<node> --since=2h | \
    grep -iE 'etcd|timeout|shutting down|Stopped'
  kubectl -n kube-system get pods -l component=kube-apiserver \
    -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount
  

3. Check etcd latency

kubectl -n kube-system logs etcd-<node> --since=2h | \
    grep -iE 'took too long|slow|apply request'
  

apply request took too long is the single most useful line in the whole investigation. It appears before the disruption rather than during it.

4. Confirm recovery rather than assuming it

kubectl get lease -A | grep -vE 'AGE|kube-node-lease'
  flux get all -A
  

Leases should show recent renewals and controllers should report successful reconciliations. Recovery is usually automatic and complete within a minute.

5. Decide whether tuning is warranted

Extending leaseDurationSeconds and renewDeadlineSeconds makes controllers tolerate longer blips, at the cost of slower failover when a leader genuinely dies. Only worth doing if disruptions are frequent and short. If they are frequent, the real fix is etcd disk performance, not lease tuning.

6. Close the telemetry gap

If log shippers dropped records during the window, the next occurrence will be equally hard to investigate. Buffering to disk on the shipper is a more durable fix than anything done to the controllers.

How does Dstl8 detect this?

Every component logs this in its own format, and a rule written for one would not match another. Nobody writes twelve of them. Dstl8 baselines each service independently, so simultaneous anomalies across unrelated services correlate into one incident rather than twelve, which is what turns a pile of controller errors into a control plane event.

The conclusion in that summary, that this was cluster-wide rather than isolated, is only reachable by comparing services against each other. It is the difference between twelve separate controller alerts and one sentence naming the actual cause.

Frequently asked questions

Related patterns

References

• Kubernetes docs: Coordinated leader election
• Kubernetes docs: Operating etcd clusters
• Kubernetes docs: API priority and fairness
• controller-runtime: Manager leader election options