Problem: Current Develop Docker Build breaking SSH Hostkey Checking

[Fablos]

Issue

Hello,

the current Develop Image seems to break SSH Connections when we keep Hostkey-Checking enabled.

In the ansible.cfg:

ssh_args = -F ssh.d/config -o ControlMaster=auto -o ControlPersist=1800s

and in the ssh.d/config:

Host *
        StrictHostKeyChecking accept-new
        UserKnownHostsFile ssh.d/known_hosts

This is working fine with the :latest Docker Image.
Entirely disabling Hostkey Checks could be argued to lessen the Security.
No Error occurs when leaving the default ansible.cfg alone in /tmp/semaphore/ansible.cfg.

This Error occurs on all Hosts when sticking to above Config:
Maybe I just miss some change lately?
Please let me know if you need anything else to know.

Task 2 added to queue
Started: 2
Run TaskRunner with template: Ping
Preparing: 2
No collection/requirements.yml file found. Skip galaxy install process.
No collection/requirements.yml file found. Skip galaxy install process.
No role/requirements.yml file found. Skip galaxy install process.
No role/requirements.yml file found. Skip galaxy install process.
ansible-playbook [core 2.16.7]
  config file = /tmp/semaphore/ansible.cfg
  configured module search path = ['/tmp/semaphore/library']
  ansible python module location = /opt/semaphore/venv/lib/python3.11/site-packages/ansible
  ansible collection location = /tmp/semaphore/collections
  executable location = /opt/semaphore/venv/bin/ansible-playbook
  python version = 3.11.9 (main, Apr 14 2024, 13:40:00) [GCC 13.2.1 20231014] (/opt/semaphore/venv/bin/python3)
  jinja version = 3.1.4
  libyaml = True
Using /tmp/semaphore/ansible.cfg as config file
setting up inventory plugins
Loading collection ansible.builtin from 
Parsed /projects/ansible/inventory/ping.yml inventory source with yaml plugin
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Loading collection community.general from /opt/semaphore/venv/lib/python3.11/site-packages/ansible_collections/community/general
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Loading callback plugin community.general.yaml of type stdout, v2.0 from /opt/semaphore/venv/lib/python3.11/site-packages/ansible_collections/community/general/plugins/callback/yaml.py
Loading callback plugin ara_default of type awesome, v2.0 from /projects/ansible/env/lib64/python3.11/site-packages/ara/plugins/callback/ara_default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: ping.yml *************************************************************
Positional arguments: playbooks/ping.yml
verbosity: 4
remote_user: ########
connection: ssh
become: True
become_method: sudo
tags: ('all',)
inventory: ('/projects/ansible/inventory/ping.yml',)
extra_vars: ('{"semaphore_vars":{"task_details":{"id":2,"url":null,"username":"########"}}}',)
forks: 25
1 plays in playbooks/ping.yml

PLAY [all] *********************************************************************
 Attempting python interpreter discovery
 ESTABLISH SSH CONNECTION FOR USER: ########
 SSH: EXEC ssh -vvv '-o BatchMode=yes' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="########"' -o ConnectTimeout=10 ############### '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'python3.12'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.11'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.10'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
 (255, b'', b'OpenSSH_9.6p1, OpenSSL 3.1.5 30 Jan 2024\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 22: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/home/semaphore/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/home/semaphore/.ssh/known_hosts2\'\r\ndebug2: resolving "###############" port 22\r\ndebug3: resolve_host: lookup ###############:22\r\ndebug3: channel_clear_timeouts: clearing\r\ndebug3: ssh_connect_direct: entering\r\ndebug1: Connecting to ############### [10.0.230.169] port 22.\r\ndebug3: set_sock_tos: set socket 3 IP_TOS 0x48\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 10000 ms remain after connect\r\ndebug1: identity file /home/semaphore/.ssh/id_rsa type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_rsa-cert type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_ecdsa type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_ecdsa-cert type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_ecdsa_sk type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_ecdsa_sk-cert type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_ed25519 type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_ed25519-cert type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_ed25519_sk type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_ed25519_sk-cert type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_xmss type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_xmss-cert type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_dsa type -1\r\ndebug1: identity file /home/semaphore/.ssh/id_dsa-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_9.6\r\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_8.7\r\ndebug1: compat_banner: match: OpenSSH_8.7 pat OpenSSH* compat 0x04000000\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to ###############:22 as \'########\'\r\ndebug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts: No such file or directory\r\ndebug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or directory\r\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory\r\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory\r\ndebug3: order_hostkeyalgs: no algorithms matched; accept original\r\ndebug3: send packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug3: receive packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: local client KEXINIT proposal\r\ndebug2: KEX algorithms: [e-mail],curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,[e-mail]\r\ndebug2: host key algorithms: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[e-mail],[e-mail],rsa-sha2-512,rsa-sha2-256\r\ndebug2: ciphers ctos: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail]\r\ndebug2: ciphers stoc: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail]\r\ndebug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\ndebug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\ndebug2: compression ctos: none,[e-mail],zlib\r\ndebug2: compression stoc: none,[e-mail],zlib\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2: first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug2: peer server KEXINIT proposal\r\ndebug2: KEX algorithms: [e-mail],curve25519-sha256,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256,diffie-hellman-group18-sha512,diffie-hellman-group16-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,[e-mail]\r\ndebug2: host key algorithms: ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256\r\ndebug2: ciphers ctos: [e-mail],aes256-ctr,[e-mail],aes128-ctr\r\ndebug2: ciphers stoc: [e-mail],aes256-ctr,[e-mail],aes128-ctr\r\ndebug2: MACs ctos: [e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512\r\ndebug2: MACs stoc: [e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512\r\ndebug2: compression ctos: none,[e-mail]\r\ndebug2: compression stoc: none,[e-mail]\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2: first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug3: kex_choose_conf: will use strict KEX ordering\r\ndebug1: kex: algorithm: curve25519-sha256\r\ndebug1: kex: host key algorithm: ssh-ed25519\r\ndebug1: kex: server->client cipher: aes128-ctr MAC: [e-mail] compression: none\r\ndebug1: kex: client->server cipher: aes128-ctr MAC: [e-mail] compression: none\r\ndebug3: send packet: type 30\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug3: receive packet: type 31\r\ndebug1: SSH2_MSG_KEX_ECDH_REPLY received\r\ndebug1: Server host key: ssh-ed25519 SHA256:LgYVd6HE0T3Wb4qp3+Huq4qCH0l3zl95gOts0vytpro\r\ndebug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts: No such file or directory\r\ndebug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or directory\r\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory\r\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory\r\ndebug3: hostkeys_find_by_key_hostfile: trying user hostfile "/home/semaphore/.ssh/known_hosts"\r\ndebug1: hostkeys_find_by_key_hostfile: hostkeys file /home/semaphore/.ssh/known_hosts does not exist\r\ndebug3: hostkeys_find_by_key_hostfile: trying user hostfile "/home/semaphore/.ssh/known_hosts2"\r\ndebug1: hostkeys_find_by_key_hostfile: hostkeys file /home/semaphore/.ssh/known_hosts2 does not exist\r\ndebug3: hostkeys_find_by_key_hostfile: trying system hostfile "/etc/ssh/ssh_known_hosts"\r\ndebug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts does not exist\r\ndebug3: hostkeys_find_by_key_hostfile: trying system hostfile "/etc/ssh/ssh_known_hosts2"\r\ndebug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts2 does not exist\r\nHost key verification failed.\r\n')
[WARNING]: Unhandled error in Python interpreter discovery for host
###############: Failed to connect to the host via ssh: OpenSSH_9.6p1,
OpenSSL 3.1.5 30 Jan 2024  debug1: Reading configuration data
/etc/ssh/ssh_config  debug1: /etc/ssh/ssh_config line 22: include
/etc/ssh/ssh_config.d/*.conf matched no files  debug3: expanded
UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/semaphore/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' ->
'/home/semaphore/.ssh/known_hosts2'  debug2: resolving "###############"
port 22  debug3: resolve_host: lookup ###############:22  debug3:
channel_clear_timeouts: clearing  debug3: ssh_connect_direct: entering  debug1:
Connecting to ############### [10.0.230.169] port 22.  debug3:
set_sock_tos: set socket 3 IP_TOS 0x48  debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK  debug1: Connection established.  debug3:
timeout: 10000 ms remain after connect  debug1: identity file
/home/semaphore/.ssh/id_rsa type -1  debug1: identity file
/home/semaphore/.ssh/id_rsa-cert type -1  debug1: identity file
/home/semaphore/.ssh/id_ecdsa type -1  debug1: identity file
/home/semaphore/.ssh/id_ecdsa-cert type -1  debug1: identity file
/home/semaphore/.ssh/id_ecdsa_sk type -1  debug1: identity file
/home/semaphore/.ssh/id_ecdsa_sk-cert type -1  debug1: identity file
/home/semaphore/.ssh/id_ed25519 type -1  debug1: identity file
/home/semaphore/.ssh/id_ed25519-cert type -1  debug1: identity file
/home/semaphore/.ssh/id_ed25519_sk type -1  debug1: identity file
/home/semaphore/.ssh/id_ed25519_sk-cert type -1  debug1: identity file
/home/semaphore/.ssh/id_xmss type -1  debug1: identity file
/home/semaphore/.ssh/id_xmss-cert type -1  debug1: identity file
/home/semaphore/.ssh/id_dsa type -1  debug1: identity file
/home/semaphore/.ssh/id_dsa-cert type -1  debug1: Local version string
SSH-2.0-OpenSSH_9.6  debug1: Remote protocol version 2.0, remote software
version OpenSSH_8.7  debug1: compat_banner: match: OpenSSH_8.7 pat OpenSSH*
compat 0x04000000  debug2: fd 3 setting O_NONBLOCK  debug1: Authenticating to
###############:22 as '########'  debug1: load_hostkeys: fopen
/home/semaphore/.ssh/known_hosts: No such file or directory  debug1:
load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or
directory  debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file
or directory  debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such
file or directory  debug3: order_hostkeyalgs: no algorithms matched; accept
original  debug3: send packet: type 20  debug1: SSH2_MSG_KEXINIT sent  debug3:
receive packet: type 20  debug1: SSH2_MSG_KEXINIT received  debug2: local
client KEXINIT proposal  debug2: KEX algorithms: sntrup761x25519-
[e-mail],curve25519-sha256,[e-mail],ecdh-
sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-
exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-
group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-
[e-mail]  debug2: host key algorithms: ssh-
[e-mail],[e-mail],ecdsa-
[e-mail],[e-mail],sk-
[e-mail],sk-ecdsa-
[e-mail],[e-mail],rsa-
[e-mail],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-
sha2-nistp384,ecdsa-sha2-nistp521,[e-mail],sk-ecdsa-
[e-mail],rsa-sha2-512,rsa-sha2-256  debug2: ciphers ctos: chac
[e-mail],aes128-ctr,aes192-ctr,aes256-ctr,aes128-
[e-mail],[e-mail]  debug2: ciphers stoc: chacha20-
[e-mail],aes128-ctr,aes192-ctr,aes256-ctr,aes128-
[e-mail],[e-mail]  debug2: MACs ctos:
[e-mail],[e-mail],hmac-
[e-mail],[e-mail],hmac-
[e-mail],[e-mail],[e-mail],hmac-
sha2-256,hmac-sha2-512,hmac-sha1  debug2: MACs stoc:
[e-mail],[e-mail],hmac-
[e-mail],[e-mail],hmac-
[e-mail],[e-mail],[e-mail],hmac-
sha2-256,hmac-sha2-512,hmac-sha1  debug2: compression ctos:
none,[e-mail],zlib  debug2: compression stoc:
none,[e-mail],zlib  debug2: languages ctos:   debug2: languages stoc:
debug2: first_kex_follows 0   debug2: reserved 0   debug2: peer server KEXINIT
proposal  debug2: KEX algorithms:
[e-mail],curve25519-sha256,ecdh-sha2-nistp521,ecdh-
sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256,diffie-
hellman-group18-sha512,diffie-hellman-group16-sha512,diffie-hellman-
group14-sha256,diffie-hellman-group14-sha1,[e-mail]
debug2: host key algorithms: ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-
rsa,ecdsa-sha2-nistp256  debug2: ciphers ctos:
[e-mail],aes256-ctr,[e-mail],aes128-ctr  debug2:
ciphers stoc:
[e-mail],aes256-ctr,[e-mail],aes128-ctr  debug2:
MACs ctos: [e-mail],hmac-sha2-256,hmac-
sha1,[e-mail],hmac-sha2-512  debug2: MACs stoc:
[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-
sha2-512  debug2: compression ctos: none,[e-mail]  debug2: compression
stoc: none,[e-mail]  debug2: languages ctos:   debug2: languages stoc:
debug2: first_kex_follows 0   debug2: reserved 0   debug3: kex_choose_conf:
will use strict KEX ordering  debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519  debug1: kex: server->client
cipher: aes128-ctr MAC: [e-mail] compression: none  debug1:
kex: client->server cipher: aes128-ctr MAC: [e-mail]
compression: none  debug3: send packet: type 30  debug1: expecting
SSH2_MSG_KEX_ECDH_REPLY  debug3: receive packet: type 31  debug1:
SSH2_MSG_KEX_ECDH_REPLY received  debug1: Server host key: ssh-ed25519
SHA256:LgYVd6HE0T3Wb4qp3+Huq4qCH0l3zl95gOts0vytpro  debug1: load_hostkeys:
fopen /home/semaphore/.ssh/known_hosts: No such file or directory  debug1:
load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or
directory  debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file
or directory  debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such
file or directory  debug3: hostkeys_find_by_key_hostfile: trying user hostfile
"/home/semaphore/.ssh/known_hosts"  debug1: hostkeys_find_by_key_hostfile:
hostkeys file /home/semaphore/.ssh/known_hosts does not exist  debug3:
hostkeys_find_by_key_hostfile: trying user hostfile
"/home/semaphore/.ssh/known_hosts2"  debug1: hostkeys_find_by_key_hostfile:
hostkeys file /home/semaphore/.ssh/known_hosts2 does not exist  debug3:
hostkeys_find_by_key_hostfile: trying system hostfile
"/etc/ssh/ssh_known_hosts"  debug1: hostkeys_find_by_key_hostfile: hostkeys
file /etc/ssh/ssh_known_hosts does not exist  debug3:
hostkeys_find_by_key_hostfile: trying system hostfile
"/etc/ssh/ssh_known_hosts2"  debug1: hostkeys_find_by_key_hostfile: hostkeys
file /etc/ssh/ssh_known_hosts2 does not exist  Host key verification failed.
Using module file /opt/semaphore/venv/lib/python3.11/site-packages/ansible/modules/setup.py
Pipelining is enabled.
 ESTABLISH SSH CONNECTION FOR USER: ########
 SSH: EXEC ssh -vvv '-o BatchMode=yes' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="########"' -o ConnectTimeout=10 ############### '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-lozndgzsjkcdjbgggmyjalvmttldwskd ; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''

TASK [Gathering Facts] *********************************************************
task path: /projects/ansible/playbooks/ping.yml:1
fatal: [###############]: UNREACHABLE! => changed=false 
  msg: |-
    Data could not be sent to remote host "###############". Make sure this host can be reached over ssh: OpenSSH_9.6p1, OpenSSL 3.1.5 30 Jan 2024
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: /etc/ssh/ssh_config line 22: include /etc/ssh/ssh_config.d/*.conf matched no files
    debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/semaphore/.ssh/known_hosts'
    debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/semaphore/.ssh/known_hosts2'
    debug2: resolving "###############" port 22
    debug3: resolve_host: lookup ###############:22
    debug3: channel_clear_timeouts: clearing
    debug3: ssh_connect_direct: entering
    debug1: Connecting to ############### [10.0.230.169] port 22.
    debug3: set_sock_tos: set socket 3 IP_TOS 0x48
    debug2: fd 3 setting O_NONBLOCK
    debug1: fd 3 clearing O_NONBLOCK
    debug1: Connection established.
    debug3: timeout: 10000 ms remain after connect
    debug1: identity file /home/semaphore/.ssh/id_rsa type -1
    debug1: identity file /home/semaphore/.ssh/id_rsa-cert type -1
    debug1: identity file /home/semaphore/.ssh/id_ecdsa type -1
    debug1: identity file /home/semaphore/.ssh/id_ecdsa-cert type -1
    debug1: identity file /home/semaphore/.ssh/id_ecdsa_sk type -1
    debug1: identity file /home/semaphore/.ssh/id_ecdsa_sk-cert type -1
    debug1: identity file /home/semaphore/.ssh/id_ed25519 type -1
    debug1: identity file /home/semaphore/.ssh/id_ed25519-cert type -1
    debug1: identity file /home/semaphore/.ssh/id_ed25519_sk type -1
    debug1: identity file /home/semaphore/.ssh/id_ed25519_sk-cert type -1
    debug1: identity file /home/semaphore/.ssh/id_xmss type -1
    debug1: identity file /home/semaphore/.ssh/id_xmss-cert type -1
    debug1: identity file /home/semaphore/.ssh/id_dsa type -1
    debug1: identity file /home/semaphore/.ssh/id_dsa-cert type -1
    debug1: Local version string SSH-2.0-OpenSSH_9.6
    debug1: Remote protocol version 2.0, remote software version OpenSSH_8.7
    debug1: compat_banner: match: OpenSSH_8.7 pat OpenSSH* compat 0x04000000
    debug2: fd 3 setting O_NONBLOCK
    debug1: Authenticating to ###############:22 as '########'
    debug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts: No such file or directory
    debug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or directory
    debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
    debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
    debug3: order_hostkeyalgs: no algorithms matched; accept original
    debug3: send packet: type 20
    debug1: SSH2_MSG_KEXINIT sent
    debug3: receive packet: type 20
    debug1: SSH2_MSG_KEXINIT received
    debug2: local client KEXINIT proposal
    debug2: KEX algorithms: [e-mail],curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,[e-mail]
    debug2: host key algorithms: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[e-mail],[e-mail],rsa-sha2-512,rsa-sha2-256
    debug2: ciphers ctos: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail]
    debug2: ciphers stoc: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail]
    debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: compression ctos: none,[e-mail],zlib
    debug2: compression stoc: none,[e-mail],zlib
    debug2: languages ctos:
    debug2: languages stoc:
    debug2: first_kex_follows 0
    debug2: reserved 0
    debug2: peer server KEXINIT proposal
    debug2: KEX algorithms: [e-mail],curve25519-sha256,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256,diffie-hellman-group18-sha512,diffie-hellman-group16-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,[e-mail]
    debug2: host key algorithms: ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256
    debug2: ciphers ctos: [e-mail],aes256-ctr,[e-mail],aes128-ctr
    debug2: ciphers stoc: [e-mail],aes256-ctr,[e-mail],aes128-ctr
    debug2: MACs ctos: [e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
    debug2: MACs stoc: [e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
    debug2: compression ctos: none,[e-mail]
    debug2: compression stoc: none,[e-mail]
    debug2: languages ctos:
    debug2: languages stoc:
    debug2: first_kex_follows 0
    debug2: reserved 0
    debug3: kex_choose_conf: will use strict KEX ordering
    debug1: kex: algorithm: curve25519-sha256
    debug1: kex: host key algorithm: ssh-ed25519
    debug1: kex: server->client cipher: aes128-ctr MAC: [e-mail] compression: none
    debug1: kex: client->server cipher: aes128-ctr MAC: [e-mail] compression: none
    debug3: send packet: type 30
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug3: receive packet: type 31
    debug1: SSH2_MSG_KEX_ECDH_REPLY received
    debug1: Server host key: ssh-ed25519 SHA256:LgYVd6HE0T3Wb4qp3+Huq4qCH0l3zl95gOts0vytpro
    debug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts: No such file or directory
    debug1: load_hostkeys: fopen /home/semaphore/.ssh/known_hosts2: No such file or directory
    debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
    debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
    debug3: hostkeys_find_by_key_hostfile: trying user hostfile "/home/semaphore/.ssh/known_hosts"
    debug1: hostkeys_find_by_key_hostfile: hostkeys file /home/semaphore/.ssh/known_hosts does not exist
    debug3: hostkeys_find_by_key_hostfile: trying user hostfile "/home/semaphore/.ssh/known_hosts2"
    debug1: hostkeys_find_by_key_hostfile: hostkeys file /home/semaphore/.ssh/known_hosts2 does not exist
    debug3: hostkeys_find_by_key_hostfile: trying system hostfile "/etc/ssh/ssh_known_hosts"
    debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts does not exist
    debug3: hostkeys_find_by_key_hostfile: trying system hostfile "/etc/ssh/ssh_known_hosts2"
    debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts2 does not exist
    Host key verification failed.
  unreachable: true
PLAY RECAP *********************************************************************
###############      : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   
Running playbook failed: exit status 4

Impact

Ansible (task execution)

Installation method

Docker

Database

Postgres

Browser

Microsoft Edge

Semaphore Version

develop-f144075-1717871677

Ansible Version

ansible [core 2.16.7]
  config file = None
  configured module search path = ['/home/semaphore/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/semaphore/venv/lib/python3.11/site-packages/ansible
  ansible collection location = /home/semaphore/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/semaphore/venv/bin/ansible
  python version = 3.11.9 (main, Apr 14 2024, 13:40:00) [GCC 13.2.1 20231014] (/opt/semaphore/venv/bin/python3)
  jinja version = 3.1.4
  libyaml = True

Logs & errors

No response

Manual installation - system information

No response

Configuration

No response

Additional information

No response

[Denis Gukov]

Hi @Fabl0s
Did you try to add environment variable ANSIBLE_HOST_KEY_CHECKING=True?

[Fablos]

Hi @Fabl0s Did you try to add environment variable ANSIBLE_HOST_KEY_CHECKING=True?

Not via Variable, I added it to my ansible.cfg as a workarround for now and it does work that way.
I can also check via EnvVar but I'd expect the same result.
But I would much preferr to auto-accept new keys and deny changed keys as a default over no checking at all.
At least as an opt-in if you absolutely want to keep it off by default.

My point about this beeing not Ideal still stands regarding Security:

Ansible enables host key checking by default. Checking host keys guards against server spoofing and man-in-the-middle attacks, but it does require some maintenance.

https://docs.ansible.com/ansible/latest/inventory_guide/connection_details.html#managing-host-key-checking

A more secure default should be kept if its already there in ansible imo.
It also can cause issues with enterprises security compliance aswell.

[Denis Gukov]

Hi @Fabl0s

If a new host is not in ‘known_hosts’ your control node may prompt for confirmation of the key, which results in an interactive experience if using Ansible, from say, cron.

It is why this scenario doesn't work. I tried disable interactive but it breaks authentication by login/password.

"Why Semaphore hangs" - most frequently asked question.

[Fablos]

Hi @Fabl0s

If a new host is not in ‘known_hosts’ your control node may prompt for confirmation of the key, which results in an interactive experience if using Ansible, from say, cron.

It is why this scenario doesn't work. I tried disable interactive but it breaks authentication by login/password.

"Why Semaphore hangs" - most frequently asked question.

I think I never had any interactive parts when using ssh flag "accept-new" - Ansible would just fail that single node in a run when we replaced a node. Maybe that can be an option?

However, could Key-Checking still be some sort of opt-in for those who want it and dont use Password Logins anyway?