MetalLB speaker Failing Gratuitous ARP: no such device or address

MetalLB gratuitous ARP failure occurs when the speaker daemonset cannot send an ARP announcement for a LoadBalancer IP because the network interface it is sending on does not exist or is down. The error is sendto: no such device or address. Without the announcement, upstream ARP caches keep pointing at whichever node last claimed the address, so traffic is delivered to a node that no longer owns it and disappears.

What does the MetalLB ARP announcement failure look like?

The speaker logs structured JSON, one entry per address per attempt:

{"caller":"announcer.go:210",
   "error":"writing \"OperationRequest\" gratuitous packet for \"10.30.40.201\":
    write packet ee:ee:ee:ee:ee:ee: sendto: no such device or address",
   "ip":"10.30.40.201","level":"error",
   "msg":"failed to make gratuitous ARP announcement",
   "op":"gratuitousAnnounce","ts":"2026-08-15T20:14:50Z"}
  

Two details are worth reading carefully.

sendto: no such device or address is the syscall failing, not the network rejecting the packet. This is ENXIO: the socket is bound to an interface that is not there. Nothing left the node.

The ip field varies across entries while everything else stays constant. A real occurrence produced forty errors in one minute across five distinct addresses, all from a single speaker pod on a single node. That shape identifies the failure as node-local rather than address-specific, which narrows the investigation immediately.

Why does the interface disappear?

MetalLB in layer 2 mode elects one node per service IP. That node’s speaker sends gratuitous ARP so upstream switches and routers learn which MAC owns the address, and it re-announces periodically and on failover.

The announcement requires a raw socket on a specific interface. It fails when:

1. The interface named in L2Advertisement does not exist on that node. A node built from a different image, a different NIC layout, or a cloud instance type with different device naming. This is the leading cause on mixed-hardware clusters.
2. Predictable interface naming changed after a kernel or firmware update, so ens192 becomes ens161 and the selector no longer matches.
3. A bond or VLAN interface went down, taking its children with it.
4. A CNI restart removed and recreated interfaces, leaving the speaker’s socket bound to a device that no longer exists.
5. The interface exists but is administratively down, so the kernel refuses the send.

Cause four explains the intermittent, self-resolving variant. The interface returns and announcements resume, so the incident closes without anyone acting, and then recurs the next time the CNI restarts.

Why is this dangerous rather than merely noisy?

Because every signal Kubernetes produces says the system is healthy.

• The Service has an external IP. Allocation worked.
• Endpoints are Ready. The backing pods are fine.
• The speaker pod is Running. It logs the error and keeps going. No restart, no probe failure, no event.
• Other nodes’ speakers are unaffected, so a cluster-wide view looks normal.

Meanwhile, upstream ARP caches hold whatever they learned last. If the previous owner node is still up, traffic goes there and is dropped because it no longer holds the address. If the entry ages out, the address becomes unreachable entirely. Either way, from the client side the symptom is a connection timeout to a service that Kubernetes reports as perfectly healthy, on a node that reports as perfectly healthy.

This is the failure mode most likely to consume an afternoon of application debugging before anyone looks at ARP.

How do I diagnose and fix it?

1. Identify the node from the speaker pod

kubectl -n metallb-system get pods -l app=metallb,component=speaker -o wide
  kubectl -n metallb-system logs <speaker-pod> --tail=100 | grep gratuitous
  

Errors from one pod and not its peers confirm a node-local problem.

2. Compare the configured interface against reality

kubectl -n metallb-system get l2advertisement -o yaml | grep -A 5 interfaces
  

Then on the affected node:

ip -br link show
  

A name in spec.interfaces with no match in ip link is the whole answer. Compare against a healthy node; different naming between nodes is the common finding.

3. Check interface state, not just existence

ip -br link show | grep -vE 'UP'
  

An interface that exists but is DOWN produces the same error as one that is absent.

4. Confirm the reachability impact

From a host on the same layer 2 segment:

arping -I <iface> 10.30.40.201
  ip neigh show 10.30.40.201
  

No reply, or a MAC belonging to a node that should not own the address, confirms the announcement never landed. This is the check that connects a log error to real impact.

5. Fix the selector or the interface

If node interface names differ, either omit spec.interfaces so MetalLB selects automatically, or split into per-node-selector L2Advertisement resources. Omitting is usually correct; the selector exists for multi-homed nodes, not as a default.

6. Force re-announcement

kubectl -n metallb-system rollout restart daemonset speaker
  

Announcements are periodic, so without a restart the corrected configuration may take a full interval to take effect and will look like it did not work.

7. Treat recurrence as the real signal

If this has happened before on the same node and self-resolved, the underlying cause is still present. The same pattern appearing twice eleven days apart on the same node is a configuration problem waiting for its next trigger, not two unrelated blips.

How does Dstl8 detect this?

A burst of forty errors lasting three seconds, from one daemonset pod, at a level most pipelines discard, ending before any evaluation window closes. Nothing restarts and no metric moves. Dstl8 baselines each service against its own history, so a pattern that was not there before registers regardless of how brief it is, and repeat occurrences are visible as a series rather than as isolated noise.

A three-second failure is shorter than most evaluation intervals, which is why nothing conventional catches it. The detail that matters most is not the burst itself but that it was the second occurrence of the same pattern on the same node, which turns an ignorable blip into a standing configuration problem.

Frequently asked questions

Related patterns

References

• MetalLB docs: Layer 2 configuration
• MetalLB docs: Layer 2 mode concepts
• MetalLB docs: Troubleshooting
• RFC 5227: IPv4 Address Conflict Detection (gratuitous ARP semantics)