MySQL InnoDB Cluster Instance Ejected, Stuck in Read-Only Mode

A MySQL InnoDB Cluster ejected instance is a member that Group Replication has expelled from the group. The instance stays up and serving reads, but sets super_read_only=ON and rejects every write. Automatic rejoin fails when the instance’s local metadata no longer matches the cluster’s, producing RuntimeError: The instance does not belong to the cluster. The pod stays Ready throughout, so Kubernetes reports the workload as healthy.

What does an ejected InnoDB Cluster instance look like?

The recovery sidecar or operator loop reports the rejoin attempt and its failure together:

ERROR: RuntimeError: The instance 'mysql-innodb-cluster-1:3306' does not belong
  to the cluster: 'prodCluster'.
  Write operations on the InnoDB cluster will not be allowed.
  + mysqlsh --defaults-extra-file=/tmp/my.cnf -- cluster rejoin-instance \
    mysql-innodb-cluster-1:3306
  

Two distinct facts are stacked in that output and they are easy to conflate. The instance is not in the group, and the attempt to put it back was rejected. The second is the harder problem: a rejoin that fails on metadata will keep failing on retry, so an automatic recovery loop can run for hours without progress.

From the application side the symptom is different and arrives first:

ERROR 1290 (HY000): The MySQL server is running with the --super-read-only
  option so it cannot execute this statement
  

Why does an instance get ejected and refuse to rejoin?

Group Replication runs a failure detector across members. A member that stops responding within the suspicion timeout, or that cannot apply the group’s transaction stream, is expelled to preserve consistency. On expulsion the instance sets super_read_only=ON. That part is the system working correctly.

Ejection triggers, in rough order of frequency on Kubernetes:

1. Network partition or interruption. A CNI restart, a node drain, or an overlay network blip longer than the suspicion timeout.
2. Node pressure. Memory or IO pressure stalls the applier thread until the member falls too far behind.
3. Pod restart. A restarted member rejoins from its last known state, which may already be too stale.
4. Transaction gap beyond binlog retention. The member needs transactions the donors have already purged, so incremental recovery cannot complete.

The rejoin then fails for a separate reason. rejoin-instance requires the instance to still be registered in the cluster’s metadata schema. If the metadata was removed, or the instance’s local copy diverged, MySQL Shell treats it as a standalone server and refuses to rejoin something that is not, as far as the metadata is concerned, a member at all.

What breaks while an instance sits read-only?

Whether anything breaks depends entirely on how traffic reaches the instance, which is why this failure is inconsistent and hard to attribute.

• Writes routed to the ejected member fail with error 1290. If your application uses MySQL Router or a read-write split, they are routed away and nothing fails. If it connects through a plain Kubernetes Service that still lists the pod, some fraction of writes fail while the rest succeed.
• Intermittent, unattributable errors. A partial write failure that depends on which backend a connection landed on is one of the harder application symptoms to trace, and it rarely points at the database.
• Reduced redundancy. A three-member group running two members has no tolerance left. The next expulsion loses quorum and takes the whole cluster read-only.
• The pod stays Ready. mysqld is running and answering, so liveness and readiness probes pass. Kubernetes has no opinion about group membership.

The last point is why this runs long. Every Kubernetes-native signal says the workload is healthy.

How do I recover an ejected InnoDB Cluster instance?

1. Establish the actual cluster state first

mysqlsh --uri root@mysql-innodb-cluster-0:3306 -- cluster status
  

Read status and each member’s memberState. OK_NO_TOLERANCE with one member MISSING confirms a single ejection. NO_QUORUM is a different incident and a different procedure.

2. Try the ordinary rejoin

mysqlsh --uri root@mysql-innodb-cluster-0:3306 -- \
    cluster rejoin-instance mysql-innodb-cluster-1:3306
  

If this succeeds, the metadata was intact and the instance was simply out of the group. If it returns does not belong to the cluster, continue.

3. Rescan to reconcile metadata

mysqlsh --uri root@mysql-innodb-cluster-0:3306 -- cluster rescan --addUnmanaged
  

rescan detects instances present in the group but absent from metadata, and the reverse. This resolves the majority of metadata-mismatch cases without touching data.

4. Re-add the instance with clone recovery

If rescan does not resolve it, the instance’s transaction history has diverged past incremental recovery. Remove and re-add it, forcing a full clone:

mysqlsh --uri root@mysql-innodb-cluster-0:3306 -- \
    cluster remove-instance mysql-innodb-cluster-1:3306 --force
  
  mysqlsh --uri root@mysql-innodb-cluster-0:3306 -- \
    cluster add-instance mysql-innodb-cluster-1:3306 --recoveryMethod=clone
  

Clone recovery drops the target’s data directory and copies from a donor. Confirm you are pointing at the ejected member and not the primary before running it.

5. Confirm the read-only flag actually cleared

SELECT @@super_read_only, @@read_only;
  SELECT MEMBER_HOST, MEMBER_STATE, MEMBER_ROLE
  FROM performance_schema.replication_group_members;
  

A rejoined secondary is expected to remain read-only. What you are checking is that it reports ONLINE in the group, not that it accepts writes.

6. Address why it was expelled

Rejoining fixes the symptom. If the cause was a network blip, review group_replication_member_expel_timeout, which defaults low enough that ordinary CNI restarts can trigger expulsion. If the cause was a transaction gap, extend binlog retention so incremental recovery remains possible. If it was node pressure, the fix is resource limits, not database configuration.

How does Dstl8 detect this?

There is no alert for this because there is no metric that moves. The pod is Ready, the process is running, and the cluster as a whole is still serving. The only evidence is a RuntimeError string in a recovery sidecar’s output, which no rule anticipates. Dstl8 baselines each service against its own normal, so an error pattern appearing in a statefulset that normally emits none is surfaced on its own.

The incident names the failing instance, captures both the ejection and the failed rejoin as one event rather than two unrelated error bursts, and cites the exact RuntimeError the conclusion was drawn from. Nothing about the workload’s Kubernetes state would have surfaced it.

Frequently asked questions

Related patterns

References

• MySQL docs: InnoDB Cluster troubleshooting
• MySQL docs: Group Replication failure detection
• MySQL docs: Rejoining a cluster
• MySQL docs: group_replication_member_expel_timeout