Embedded locksmith does not work with etcd 3.5 or 3.4

[Matasx]

Description

When using Flatcar LTS (or possibly other channels, as the etcd go client in locksmith repository was not updated for years), the locksmith is unable to connect to etcd instance that runs version 3.4 or 3.5 (only these are currently supported from security perspective: https://endoflife.date/etcd ). I was able to make it work with version 3.0, but I don't like running obsolete etcd in production environment.

Impact

Unable to run secure etcd server in production as locksmith etcd client does not support current version of etcd (3.4 or 3.5).

Environment and steps to reproduce

Have Flatcar machine with private IP 10.0.0.3 provisioned with Flacar LTS-2024.
Run local etcd instance and use this instance for locksmith reboot strategy, e.g. using Butane config (transpile with ct):

etcd:
  version:                     3.5.13
  # version:                     3.4.32
  name:                        s1
  advertise_client_urls:       http://10.0.0.3:2379
  initial_advertise_peer_urls: http://10.0.0.3:2380
  listen_client_urls:          http://10.0.0.3:2379
  listen_peer_urls:            http://10.0.0.3:2380
  initial_cluster:             s1=http://10.0.0.3:2380
 
update:
  group: lts-2024
  server: https://public.update.flatcar-linux.net/v1/update/

locksmith:
  reboot_strategy: etcd-lock
  window_start:    Sun 4:00
  window_length:   2h
  etcd_endpoints:  http://10.0.0.3:2379
  group:           common

Expected behavior

locksmithd.service should run without any errors. locksmithctl should be able to display status.

Additional information

Actual behavior:

> systemctl status locksmithd.service
...
May 06 13:32:55 vultr.guest locksmithd[1156]: Unlocking old locks failed: error setting up lock: Error initializing etcd client: creating etcd lock client: client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint. Retrying in 20s.

> locksmithctl --endpoint=http://10.0.0.3:2379 status
Error initializing etcd client: creating etcd lock client: client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint

[Kai Lüke]

The etcd2 protocol is needed, you need to configure it with --enable-v2.

For v3 support we have an existing issue: #510 (comment)

[Matasx]

Thank you, I was looking for this option but couldn't find it.
I was able to make it work with the compatibility flag.
If somebody else needs this, here is the working setup for etcd:

etcd:
  version:                     3.5.13
  enable_v2:                   true
  name:                        s1
  advertise_client_urls:       http://10.0.0.3:2379
  initial_advertise_peer_urls: http://10.0.0.3:2380
  listen_client_urls:          http://10.0.0.3:2379
  listen_peer_urls:            http://10.0.0.3:2380
  initial_cluster:             s1=http://10.0.0.3:2380

Also note that this option is already deprecated in 3.5 and will be dropped in etcd 3.6.