Connections are not disconnected when promoting replica MySQL

[Masayuki Ishii]

Describe the bug
When a replica MySQL is promoted to primary by switch over or failover, the connections via replica svc are not disconnected.

Environments

• MOCO v0.17.0
• kind 0.19.0 (k8s 1.27.1)

To Reproduce

1. Create a MySQLCluster consisting of three MySQLs.

apiVersion: moco.cybozu.com/v1beta2
kind: MySQLCluster
metadata:
  name: test-cluster
spec:
  replicas: 3
  podTemplate:
    spec:
      containers:
        - name: mysqld
          image: ghcr.io/cybozu-go/moco/mysql:8.0.28
  volumeClaimTemplates:
    - metadata:
        name: mysql-data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 500Mi

2. Create a user in the MySQLCluster

$ kubectl-moco mysql test-cluster -u moco-admin -it
mysql> CREATE USER 'hoge'@'%' IDENTIFIED BY 'piyo';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'hoge'@'%';
mysql> exit

3. Create a client pod

apiVersion: v1
kind: Pod
metadata:
  name: test-client
spec:
  containers:
  - command:
    - /usr/local/bin/pause
    image: ghcr.io/cybozu/ubuntu-debug:22.04
    imagePullPolicy: Always
    name: ubuntu
  securityContext:
    runAsGroup: 10000
    runAsUser: 10000

5. Connect to a replica MySQL from the client pod via replica service

$ kubectl exec -it test-client -- bash
test-client:/$ mysql -u hoge -ppiyo -h moco-test-cluster-replica.default.svc # via replica svc

mysql> show variables like 'hostname';
+---------------+---------------------+
| Variable_name | Value               |
+---------------+---------------------+
| hostname      | moco-test-cluster-1 | ★ Try to connect to the smaller index one of the two replicas
+---------------+---------------------+
1 row in set (0.00 sec)

mysql> show variables like 'read_only';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only     | ON    |
+---------------+-------+
1 row in set (0.00 sec)

mysql> show variables like 'super_read_only';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| super_read_only | ON    |
+-----------------+-------+
1 row in set (0.00 sec)

★ Do not exit! Execute the same queries later on this console.

6. Do the switchover on different console

$ kubectl get mysqlcluster
NAME           AVAILABLE   HEALTHY   PRIMARY   SYNCED REPLICAS   ERRANT REPLICAS   LAST BACKUP
test-cluster   True        True      0         3                                   <no value>

$ kubectl-moco switchover test-cluster

$ kubectl get mysqlcluster
NAME           AVAILABLE   HEALTHY   PRIMARY   SYNCED REPLICAS   ERRANT REPLICAS   LAST BACKUP
test-cluster   True        True      1         3                                   <no value>
★ ↑ moco-test-cluster-1 become primary!

7. Execute the queries on the client pod

mysql> show variables like 'hostname';
+---------------+---------------------+
| Variable_name | Value               |
+---------------+---------------------+
| hostname      | moco-test-cluster-1 |
+---------------+---------------------+
1 row in set (0.00 sec)

mysql> show variables like 'read_only';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only     | OFF   | ★ It become writable even though it is connected via replica svc.
+---------------+-------+
1 row in set (0.00 sec)

mysql> show variables like 'super_read_only';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| super_read_only | OFF   |
+-----------------+-------+
1 row in set (0.00 sec)

Expected behavior

I would like existing connections to be disconnected if the MySQL role is changed.

Additional context

Similar problems occur when using Cillium LB, as well as intra-cluster communication.

cilium/cilium#27730
cilium/cilium#27726

[kouki kamada]

resolved at #587