failed to acquire lease kube-system/snapshot-controller-leader

failed to acquire lease kube-system/snapshot-controller-leader is logged by any snapshot-controller replica that is not the leader. At a low, steady rate this is healthy standby behavior and should be ignored. When the rate spikes, it means the lease is changing hands repeatedly, and VolumeSnapshot create and delete operations stall while no replica holds leadership long enough to complete them.

What does the log line look like, and when does it matter?

The line itself is unremarkable, and note the level: I for informational, not E.

I0728 22:18:56.502652 1 leaderelection.go:255] failed to acquire lease
  kube-system/snapshot-controller-leader
  I0728 22:18:50.744936 1 leaderelection.go:255] failed to acquire lease
  kube-system/snapshot-controller-leader
  I0728 22:18:41.769713 1 leaderelection.go:255] failed to acquire lease
  kube-system/snapshot-controller-leader
  

Interpretation depends entirely on rate:

This makes the pattern effectively invisible to conventional monitoring. Severity filtering drops it because it is informational. Text matching cannot distinguish the healthy case from the failing one, because the text is identical. Only the rate carries the signal, and the rate has no natural threshold, because what counts as abnormal depends on the replica count and the polling interval of that particular deployment.

Why does lease acquisition start thrashing?

The Lease object in kube-system is a normal API resource. The holder renews it on an interval, and every other replica polls to see whether it has expired. Thrashing means the holder is failing to renew reliably while still running.

Causes, in the order worth checking:

1. API server latency. Renewals are API writes. If they queue, they miss the deadline. Usually accompanied by similar behavior in other leased controllers.
2. The holder pod is degraded but alive. CPU throttled or memory pressured enough to miss renewal windows, but not enough to fail a probe or get killed. This is the most common single-controller cause.
3. Node-level network interruption on the holder’s node, intermittent rather than total.
4. Clock skew between nodes, making lease expiry calculations disagree.
5. etcd latency, which surfaces as API latency but has a different fix.

Cause two is worth checking first when only this controller is affected. If several controllers are thrashing together, it is cause one or five.

What stalls while the lease is contested?

A narrow, quiet set of operations:

• VolumeSnapshot creation stalls. The resource is created in the API and simply never becomes ready.
• Snapshot deletion stalls, so VolumeSnapshotContent objects accumulate and the underlying storage snapshots are never released, which has a billing consequence on managed storage.
• Backup jobs hang or time out waiting on snapshot readiness. Depending on the tool, they may report success having captured nothing.
• Restores are unavailable for the window, which nobody discovers until they need one.
• Every pod stays Ready. The controller is running and logging. No restarts, no probe failures.

The asymmetry matters. Nothing breaks visibly today, and the cost lands entirely on a future restore attempt.

How do I investigate lease thrashing?

1. Look at the lease itself

kubectl -n kube-system get lease snapshot-controller-leader -o yaml
  

holderIdentity names the current holder and renewTime shows the last successful renewal. Run it twice a few seconds apart. A changing holderIdentity confirms thrashing directly, without inferring anything from logs.

2. Check the holder’s health

kubectl -n kube-system get pods -l app=snapshot-controller -o wide
  kubectl top pod -n kube-system -l app=snapshot-controller
  kubectl -n kube-system describe pod <holder> | grep -A 5 -iE 'throttl|limits|restart'
  

Throttling on the holder while the standby is idle explains the pattern completely.

3. Rule out a cluster-wide cause

kubectl get lease -A --sort-by=.spec.renewTime | tail -20
  

Stale renew times across many unrelated leases point at API server or etcd latency, which is a different investigation and a different fix.

4. Check what stalled

kubectl get volumesnapshot -A
  kubectl get volumesnapshotcontent | wc -l
  

Snapshots stuck with readyToUse false, or content objects accumulating well beyond your retention policy, tell you the operational cost of the window.

5. Force a clean handover if the holder is degraded

kubectl -n kube-system delete pod <holder>
  

The standby acquires within a lease duration. This is a workaround, not a fix, and it will recur if the underlying resource pressure is unaddressed.

6. Address the cause

For a throttled holder, raise CPU limits. For API latency, look at etcd disk performance. Lease duration tuning is available but rarely the right answer here, because a controller that cannot renew a lease also cannot reliably complete snapshot operations.

How does Dstl8 detect this?

This is the clearest case for baselining over rules in the whole library. The log line is identical whether the system is healthy or failing, it is logged at informational level, and the threshold that separates the two states differs per deployment. No static rule can express it. A per-service baseline can, because it knows what this controller’s normal rate has been.

The detection is entirely rate-based against this controller’s own history, not against a configured number. It also closes itself on observing a successful renewal, rather than waiting for someone to acknowledge a page for a log line that is normal most of the time.

Frequently asked questions

Related patterns

References

• Kubernetes docs: Volume snapshots
• Kubernetes CSI docs: Snapshot controller
• Kubernetes docs: Lease API
• Kubernetes docs: Coordinated leader election