calico-kube-controllers Cannot Watch hostendpoints: RBAC Forbidden

calico-kube-controllers hostendpoints forbidden is an RBAC failure where the controller’s service account lacks the watch verb on hostendpoints.crd.projectcalico.org. The watcher cannot be created, so host endpoint state stops syncing and policy for those endpoints is no longer enforced or updated. The controller retries indefinitely, logging warnings rather than crashing, so the pod stays Ready and no Kubernetes signal reports a problem.

What does the Calico RBAC hostendpoints error look like?

The warning repeats on a retry loop, roughly twice a second, from the controller’s watcher cache:

[WARNING][14] kube-controllers/watchercache.go 355: Failed to create watcher;
  will retry. ListRoot=".../v3/pc.org/hostendpoints"
  error=connection is unauthorized: hostendpoints.crd.projectcalico.org is
  forbidden: User "system:serviceaccount:kube-system:calico-kube-controllers"
  cannot watch resource "hostendpoints" in API group "crd.projectcalico.org"
  at the cluster scope errorsWithoutProgress=5 performFullResync=false
  

Three fields carry the diagnosis. The service account is named explicitly, so there is no ambiguity about which identity was denied. The resource and API group are named, so you know exactly which rule is missing. And errorsWithoutProgress increments, which distinguishes a persistent RBAC denial from a transient API server blip that recovers.

The log level is WARNING, not ERROR. Anything filtering on error severity misses this entirely.

Why does the RBAC rule go missing?

calico-kube-controllers maintains watchers over Calico’s CRDs to keep datastore state in sync with policy. hostendpoints represent node network interfaces, and policy attached to them governs traffic to and from the hosts themselves rather than pods.

The permission usually disappears through drift rather than a deliberate change:

1. Partial upgrade. CRDs and the controller image update while the RBAC manifest does not. New Calico versions add resources to the required ClusterRole, and a manifest pinned in Git keeps the old rule set.
2. Manifest divergence. A locally modified copy of the Calico manifests, applied by a GitOps controller, silently overwrites what the operator installed.
3. Operator to manifest migration, or the reverse, where the two install methods manage the ClusterRole differently.
4. Restrictive policy tooling that prunes verbs deemed unnecessary, with watch an easy target because list and get appear to cover the same ground.

Because Calico continues to function for pod networking throughout, nothing about the cluster’s behavior suggests a permission is missing.

What breaks while the watcher cannot start?

Pod-to-pod networking is unaffected, which is why this survives so long. What stops is the host endpoint layer:

• Host endpoint policy stops updating. Rules already programmed stay in place. New or changed HostEndpoint resources are never applied.
• New nodes may not be protected. A node joining the cluster does not get its host endpoint policy programmed, so host-level rules intended to cover it are absent.
• Policy changes appear to succeed. kubectl apply accepts the resource and it exists in the API. It simply never reaches the dataplane.
• The controller never restarts. It warns and retries. No CrashLoopBackOff, no restart count, no probe failure, no event.

The last two combine into the real risk. Someone applies a host endpoint policy to close a gap, sees no error, and believes the gap is closed.

How do I fix the Calico hostendpoints RBAC denial?

1. Confirm the denial directly

kubectl auth can-i watch hostendpoints.crd.projectcalico.org \
    --as=system:serviceaccount:kube-system:calico-kube-controllers
  

A no confirms it without relying on log interpretation. Repeat with list and get to see how much of the rule survives.

2. Inspect the current ClusterRole

kubectl get clusterrole calico-kube-controllers -o yaml | \
    grep -A 12 'crd.projectcalico.org'
  

Look for a rule covering hostendpoints and check which verbs it lists. The common failure is a rule that exists with get and list but not watch.

3. Add the missing verbs

- apiGroups: ["crd.projectcalico.org"]
    resources: ["hostendpoints"]
    verbs: ["get", "list", "watch", "create", "update", "delete"]
  

Apply it, then confirm:

kubectl auth can-i watch hostendpoints.crd.projectcalico.org \
    --as=system:serviceaccount:kube-system:calico-kube-controllers
  

4. Restart the controller to clear the retry loop

kubectl -n kube-system rollout restart deployment calico-kube-controllers
  kubectl -n kube-system logs -l k8s-app=calico-kube-controllers --tail=50
  

The watcher cache backs off, so the corrected permission may not take effect promptly without a restart. A clean start logs successful watcher creation with no errorsWithoutProgress.

5. Verify host endpoint policy actually applied

calicoctl get hostendpoints -o wide
  

Compare against the HostEndpoint resources you expect. Anything created during the outage window needs re-checking, since it was accepted by the API but never programmed.

6. Stop the drift from recurring

Take the ClusterRole from the Calico release matching your deployed version rather than maintaining a local copy, and diff it on every upgrade. If a GitOps controller manages Calico, confirm it is not reconciling an older manifest over the operator’s.

How does Dstl8 detect this?

A repeating WARNING from one controller is invisible to every conventional signal. There is no restart to count, no error rate that crosses a threshold, and severity filtering discards it before it reaches a dashboard. Dstl8 baselines each service against its own history, so a warning pattern that was not present before becomes an incident, and the services it correlates with reveal how far the effect reaches.

The correlation is the part no single log line gives you. One missing RBAC verb, and the incident links the controller to the CNI daemonset, the load balancer, the database statefulset, the GitOps controllers, and the monitoring stack across three nodes. Reading the controller’s logs alone tells you a watcher failed. Reading the correlation tells you what it costs.

Frequently asked questions

Related patterns

References

• Calico docs: Host endpoints
• Calico docs: kube-controllers configuration
• Kubernetes docs: Using RBAC authorization
• Kubernetes docs: kubectl auth can-i