Downgrade to `3.4.31` Fails in Single-Node Etcd Cluster Using `--next-cluster-version-compatible=true`; Retains Version `3.5`
seshachalam-yv opened this issue · comments
Bug report criteria
- This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
- This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
- You have read the etcd bug reporting guidelines.
- Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.
What happened?
I attempted to downgrade a single-node etcd cluster from version 3.5.11 to 3.4.31 using the --next-cluster-version-compatible=true flag. Despite the downgrade procedure, the curl http://localhost:2379/version endpoint and the etcd logs continued to show the etcd server running version 3.5.11, indicating that the downgrade did not take effect. In contrast, downgrading in a multi-node cluster resulted in the curl http://localhost:2379/version endpoint reflecting the correct downgraded version ({"etcdserver":"3.4.31","etcdcluster":"3.4.0"}).
What did you expect to happen?
After the downgrade process, I expected the /version endpoint to reflect the new version as {"etcdserver":"3.4.31","etcdcluster":"3.4.0"} in the single-node cluster, similar to the results observed in multi-node clusters.
How can we reproduce it (as minimally and precisely as possible)?
-
Setup a single-node etcd cluster running version 3.5.11.
-
Attempt to downgrade to etcd version 3.4.31 using the command with the flag
--next-cluster-version-compatible=true.~/go/src/etcd-io/etcd/bin/etcd --next-cluster-version-compatible=true -
After the downgrade attempt, query the version with:
curl http://localhost:2379/version. -
Even logs will show as
024-04-18 12:26:11.388528 N | etcdserver/membership: set the initial cluster version to 3.5
2024-04-18 12:26:11.388558 I | etcdserver/api: enabled capabilities for version 3.5
Anything else we need to know?
- This issue appears to be specific to single-node cluster. Multi-node clusters do not exhibit this problem and display the correct version information post-downgrade.
- Logs during the downgrade process show that the downgrade commands execute without error but still reflect version 3.5 capabilities being enabled.
Etcd version (please run commands below)
$ etcd --version
etcd Version: 3.4.31
Git SHA: 950cd5fbe
Go Version: go1.21.6
Go OS/Arch: darwin/amd64
$ etcdctl version
etcdctl version: 3.4.31
API version: 3.4Etcd configuration (command line flags or environment variables)
paste your configuration here
Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
$ etcdctl member list -w table
+------------------+---------+---------+-----------------------+-----------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+---------+-----------------------+-----------------------+------------+
| 8e9e05c52164694d | started | default | http://localhost:2380 | http://localhost:2379 | false |
+------------------+---------+---------+-----------------------+-----------------------+------------+
$ etcdctl endpoint status -w table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 8e9e05c52164694d | 3.4.31 | 98 kB | true | false | 4 | 9 | 9 | |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+Relevant log output
~/go/src/etcd-io/etcd/bin/etcd --next-cluster-version-compatible=true
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
A future release of golang/protobuf will delete this package,
which has long been excluded from the compatibility promise.
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2024-04-18 12:26:11.293145 W | embed: Running http and grpc server on single port. This is not recommended for production.
2024-04-18 12:26:11.293277 I | etcdmain: etcd Version: 3.4.31
2024-04-18 12:26:11.293281 I | etcdmain: Git SHA: 950cd5fbe
2024-04-18 12:26:11.293283 I | etcdmain: Go Version: go1.21.6
2024-04-18 12:26:11.293290 I | etcdmain: Go OS/Arch: darwin/amd64
2024-04-18 12:26:11.293294 I | etcdmain: setting maximum number of CPUs to 10, total number of available CPUs is 10
2024-04-18 12:26:11.293504 N | etcdmain: failed to detect default host (default host not supported on darwin_amd64)
2024-04-18 12:26:11.293510 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2024-04-18 12:26:11.293740 N | etcdmain: the server is already initialized as member before, starting as etcd member...
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2024-04-18 12:26:11.293772 W | embed: Running http and grpc server on single port. This is not recommended for production.
2024-04-18 12:26:11.297174 I | embed: name = default
2024-04-18 12:26:11.297181 I | embed: data dir = default.etcd
2024-04-18 12:26:11.297184 I | embed: member dir = default.etcd/member
2024-04-18 12:26:11.297187 I | embed: heartbeat = 100ms
2024-04-18 12:26:11.297189 I | embed: election = 1000ms
2024-04-18 12:26:11.297191 I | embed: snapshot count = 100000
2024-04-18 12:26:11.297271 I | embed: advertise client URLs = http://localhost:2379
2024-04-18 12:26:11.297277 I | embed: initial advertise peer URLs = http://localhost:2380
2024-04-18 12:26:11.297341 I | embed: initial cluster =
2024-04-18 12:26:11.322667 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 5
raft2024/04/18 12:26:11 INFO: 8e9e05c52164694d switched to configuration voters=()
raft2024/04/18 12:26:11 INFO: 8e9e05c52164694d became follower at term 2
raft2024/04/18 12:26:11 INFO: newRaft 8e9e05c52164694d [peers: [], term: 2, commit: 5, applied: 0, lastindex: 5, lastterm: 2]
2024-04-18 12:26:11.338791 W | auth: simple token is not cryptographically signed
2024-04-18 12:26:11.361152 I | mvcc: downgrade meta bucket: remove keys [confState, term]
2024-04-18 12:26:11.379416 I | etcdserver: starting server... [version: 3.4.31, cluster version: to_be_decided]
2024-04-18 12:26:11.379764 E | etcdserver: cannot monitor file descriptor usage (cannot get FDUsage on darwin)
2024-04-18 12:26:11.383287 I | pkg/fileutil: started to purge file, dir: default.etcd/member/snap, suffix: snap.db, max: 5, interval: 30s
2024-04-18 12:26:11.383307 I | pkg/fileutil: started to purge file, dir: default.etcd/member/snap, suffix: snap, max: 5, interval: 30s
2024-04-18 12:26:11.383315 I | pkg/fileutil: started to purge file, dir: default.etcd/member/wal, suffix: wal, max: 5, interval: 30s
raft2024/04/18 12:26:11 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
2024-04-18 12:26:11.385005 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2024-04-18 12:26:11.385589 I | embed: listening for peers on 127.0.0.1:2380
2024-04-18 12:26:11.388528 N | etcdserver/membership: set the initial cluster version to 3.5
2024-04-18 12:26:11.388558 I | etcdserver/api: enabled capabilities for version 3.5
raft2024/04/18 12:26:12 INFO: 8e9e05c52164694d is starting a new election at term 2
raft2024/04/18 12:26:12 INFO: 8e9e05c52164694d became candidate at term 3
raft2024/04/18 12:26:12 INFO: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3
raft2024/04/18 12:26:12 INFO: 8e9e05c52164694d became leader at term 3
raft2024/04/18 12:26:12 INFO: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3
2024-04-18 12:26:12.571032 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2024-04-18 12:26:12.571172 I | embed: ready to serve client requests
2024-04-18 12:26:12.576038 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
Thank you for @seshachalam-yv raising the issue, found the reason in #17821
Link to #17853
Please update changelog for 3.4.32 before we close this issue.