Connections are not disconnected when promoting replica MySQL
masa213f opened this issue · comments
Describe the bug
When a replica MySQL is promoted to primary by switch over or failover, the connections via replica svc are not disconnected.
Environments
- MOCO v0.17.0
- kind 0.19.0 (k8s 1.27.1)
To Reproduce
- Create a MySQLCluster consisting of three MySQLs.
apiVersion: moco.cybozu.com/v1beta2
kind: MySQLCluster
metadata:
name: test-cluster
spec:
replicas: 3
podTemplate:
spec:
containers:
- name: mysqld
image: ghcr.io/cybozu-go/moco/mysql:8.0.28
volumeClaimTemplates:
- metadata:
name: mysql-data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 500Mi
- Create a user in the MySQLCluster
$ kubectl-moco mysql test-cluster -u moco-admin -it
mysql> CREATE USER 'hoge'@'%' IDENTIFIED BY 'piyo';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'hoge'@'%';
mysql> exit
- Create a client pod
apiVersion: v1
kind: Pod
metadata:
name: test-client
spec:
containers:
- command:
- /usr/local/bin/pause
image: ghcr.io/cybozu/ubuntu-debug:22.04
imagePullPolicy: Always
name: ubuntu
securityContext:
runAsGroup: 10000
runAsUser: 10000
- Connect to a replica MySQL from the client pod via replica service
$ kubectl exec -it test-client -- bash
test-client:/$ mysql -u hoge -ppiyo -h moco-test-cluster-replica.default.svc # via replica svc
mysql> show variables like 'hostname';
+---------------+---------------------+
| Variable_name | Value |
+---------------+---------------------+
| hostname | moco-test-cluster-1 | ★ Try to connect to the smaller index one of the two replicas
+---------------+---------------------+
1 row in set (0.00 sec)
mysql> show variables like 'read_only';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only | ON |
+---------------+-------+
1 row in set (0.00 sec)
mysql> show variables like 'super_read_only';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| super_read_only | ON |
+-----------------+-------+
1 row in set (0.00 sec)
★ Do not exit! Execute the same queries later on this console.
- Do the switchover on different console
$ kubectl get mysqlcluster
NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS LAST BACKUP
test-cluster True True 0 3 <no value>
$ kubectl-moco switchover test-cluster
$ kubectl get mysqlcluster
NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS LAST BACKUP
test-cluster True True 1 3 <no value>
★ ↑ moco-test-cluster-1 become primary!
- Execute the queries on the client pod
mysql> show variables like 'hostname';
+---------------+---------------------+
| Variable_name | Value |
+---------------+---------------------+
| hostname | moco-test-cluster-1 |
+---------------+---------------------+
1 row in set (0.00 sec)
mysql> show variables like 'read_only';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only | OFF | ★ It become writable even though it is connected via replica svc.
+---------------+-------+
1 row in set (0.00 sec)
mysql> show variables like 'super_read_only';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| super_read_only | OFF |
+-----------------+-------+
1 row in set (0.00 sec)
Expected behavior
I would like existing connections to be disconnected if the MySQL role is changed.
Additional context
Similar problems occur when using Cillium LB, as well as intra-cluster communication.
resolved at #587