Rocky Linux 9 - unable to complete Kitchen commands due to failure in connection

Question

Rocky Linux 9 - unable to complete Kitchen commands due to failure in connection

gabytsilver opened this issue 2 years ago · comments

When using the Rocky Linux 9 Vagrant box (https://app.vagrantup.com/bento/boxes/rockylinux-9), Kitchen commands do not complete successfully, with the error:

Waiting for SSH service on 127.0.0.1:2200, retrying in 3 seconds

The OS declaration in .kitchen.yml:

- name: linux9
  driver_config:
        box: 'bento/rockylinux-9' 
        synced_folders:
                - ['../installers', '/installers']
        customize:
                memory:  6200
  provisioner:
      product_name: chef
      download_url: 'http://packages.cinc.sh/files/stable/cinc/17.10.0/el/7/cinc-17.10.0-1.el7.x86_64.rpm'
      chef_omnibus_root: /opt/cinc
      chef_client_path: /opt/cinc/bin/cinc-client

Note that we use the same structure when using the bento/centos-8.2 Vagrant box.

Describe the problem

Using the defined OS in the .kitchen.yml results in a connection error hang:

Waiting for SSH service on 127.0.0.1:2200, retrying in 3 seconds

When I execute vagrant ssh-config in the .vagrant directory:

$ vagrant ssh-config
Host default
  HostName 127.0.0.1
  User vagrant
  Port 2200
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile <cookbook path>/.kitchen/kitchen-vagrant/<suit-linux9>/.vagrant/machines/default/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

I am able to login into the VM using vagrant ssh command from .vagrant directory.

Software Version

Using both ChefDK 3.9 and ChefDK 4.13, Vagrant 2.2.19
Windows 10
VirtualBox Version: 6.0

Replication Case

Add to .kitchen.yml the declaration for the Rocky Linux 9 (attached above)
Execute $ kitchen create <suit_name>-linux9

Stacktrace

$ kitchen converge pg-foundation-rhel9try
-----> Starting Kitchen (v1.24.0)
WARN: Unresolved specs during Gem::Specification.reset:
      bundler (>= 1.10)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
-----> Creating <suit-linux9> ...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Importing base box 'bento/rockylinux-9'...
==> default: Matching MAC address for NAT networking...
       ==> default: Checking if box 'bento/rockylinux-9' version '202207.20.0' is up to date...
       ==> default: Setting the name of the VM: <suit-linux9>
       ==> default: Fixed port collision for 22 => 2222. Now on port 2200.
       ==> default: Clearing any previously set network interfaces...
       ==> default: Preparing network interfaces based on configuration...
           default: Adapter 1: nat
       ==> default: Forwarding ports...
           default: 22 (guest) => 2200 (host) (adapter 1)
       ==> default: Running 'pre-boot' VM customizations...
       ==> default: Booting VM...
       ==> default: Waiting for machine to boot. This may take a few minutes...
           default: SSH address: 127.0.0.1:2200
           default: SSH username: vagrant
           default: SSH auth method: private key
           default:
           default: Vagrant insecure key detected. Vagrant will automatically replace
           default: this with a newly generated keypair for better security.
           default:
           default: Inserting generated public key within guest...
           default: Removing insecure key from the guest if it's present...
           default: Key inserted! Disconnecting and reconnecting using new SSH key...
       ==> default: Machine booted and ready!
       ==> default: Checking for guest additions in VM...
           default: The guest additions on this VM do not match the installed version of
           default: VirtualBox! In most cases this is fine, but in rare cases it can
           default: prevent things such as shared folders from working properly. If you see
           default: shared folder errors, please make sure the guest additions within the
           default: virtual machine match the version of VirtualBox you have installed on
           default: your host and reload your VM.
           default:
           default: Guest Additions Version: 6.1.34
           default: VirtualBox Version: 6.0
       ==> default: Setting hostname...
       ==> default: Mounting shared folders...
           default: /installers => <local path>/installers
       ==> default: Machine not provisioned because `--no-provision` is specified.
       Waiting for SSH service on 127.0.0.1:2200, retrying in 3 seconds
       Waiting for SSH service on 127.0.0.1:2200, retrying in 3 seconds
       Waiting for SSH service on 127.0.0.1:2200, retrying in 3 seconds
       Waiting for SSH service on 127.0.0.1:2200, retrying in 3 seconds
       Waiting for SSH service on 127.0.0.1:2200, retrying in 3 seconds
       Waiting for SSH service on 127.0.0.1:2200, retrying in 3 seconds
       Waiting for SSH service on 127.0.0.1:2200, retrying in 3 seconds
       Waiting for SSH service on 127.0.0.1:2200, retrying in 3 seconds

Corey Hemminger · Answer 1 · Fri Nov 11 2022 01:29:11 GMT+0800 (China Standard Time)

This is a known issue due to the changes in ssh for newer Linux OSs. The latest test-kitchen 3.4.0 updated the net-ssh gem that fixes this. Latest chef-client 18 also merged an update to fix this issue. In the next chef-client and workstation releases we should see these updates pulled in.

As a workaround users have found switching to ssh user/password works in the mean time.

Closing this since this really isn’t a bento repo issue.

gabytsilver · Answer 2 · Wed Nov 16 2022 22:08:43 GMT+0800 (China Standard Time)

@Stromweld thank you for taking the time to reply with the detailed information.
I am still trying to configure my environment to use the right Chef gem, as the current Ruby version installed with the latest ChefDK is 2.7, which only supports Chef 17.
Trying to upgrade Ruby and to try again.
Unfortunately, passing the password/username does not work for me.

gabytsilver · Answer 3 · Thu Nov 17 2022 01:33:57 GMT+0800 (China Standard Time)

@Stromweld Have you found online a record to the issue with ssh for newer Linux OSs ?
Trying to see if I can workaround this manually by creating a custom Vagrantfile to provision the VMs.

Corey Hemminger · Answer 4 · Thu Nov 17 2022 01:34:14 GMT+0800 (China Standard Time)

Can you paste your kitchen.yml file in here? Also if you remove chefdk and install chef-workstation you should get the newer 3.x ruby version installed.

Corey Hemminger · Answer 5 · Thu Nov 17 2022 01:39:32 GMT+0800 (China Standard Time)

Here's a link on the changes RHEL and Ubuntu made to the crypto algorithms that affect ssh. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/considerations_in_adopting_rhel_9/index#ref_considerations-security-crypto_changes-to-security This only affects ssh key authentication if you switch to ssh username/password in the transport section it should work.

gabytsilver · Answer 6 · Thu Nov 17 2022 02:17:42 GMT+0800 (China Standard Time)

Yes, just tried to install Chef-Workstation v22.10.1013, which has:

Ruby 3.0.3.
Test Kitchen v3.3.2

"kitchen create" command failed on the same error I reported, unfortunately also for username/password (rhel9 is just the name I've given the platform in .kitchen.yml, I deleted from the log the names and paths)

>kitchen create <suit-name>-rhel9
-----> Starting Test Kitchen (v3.3.2)
Top level ::CompositeIO is deprecated, require 'multipart/post' and use `Multipart::Post::CompositeReadIO` instead!
Top level ::Parts is deprecated, require 'multipart/post' and use `Multipart::Post::Parts` instead!
-----> Creating <suit-name>-rhel9>...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Importing base box 'bento/rockylinux-9'...
==> default: Matching MAC address for NAT networking...
       ==> default: Checking if box 'bento/rockylinux-9' version '202207.20.0' is up to date...
       ==> default: Setting the name of the VM: kitchen-<cookbook><suit>-rhel9-9b87b744-7092-422e-8ef2-4dfb0f5f0fff
       ==> default: Clearing any previously set network interfaces...
       ==> default: Preparing network interfaces based on configuration...
           default: Adapter 1: nat
       ==> default: Forwarding ports...
           default: 22 (guest) => 2222 (host) (adapter 1)
       ==> default: Running 'pre-boot' VM customizations...
       ==> default: Booting VM...
       ==> default: Waiting for machine to boot. This may take a few minutes...
           default: SSH address: 127.0.0.1:2222
           default: SSH username: vagrant
           default: SSH auth method: password
           default:
           default: Inserting generated public key within guest...
           default: Removing insecure key from the guest if it's present...
           default: Key inserted! Disconnecting and reconnecting using new SSH key...
       ==> default: Machine booted and ready!
       ==> default: Checking for guest additions in VM...
           default: The guest additions on this VM do not match the installed version of
           default: VirtualBox! In most cases this is fine, but in rare cases it can
           default: prevent things such as shared folders from working properly. If you see
           default: shared folder errors, please make sure the guest additions within the
           default: virtual machine match the version of VirtualBox you have installed on
           default: your host and reload your VM.
           default:
           default: Guest Additions Version: 6.1.34
           default: VirtualBox Version: 6.0
       ==> default: Setting hostname...
       ==> default: Mounting shared folders...
       ==> default: Machine not provisioned because `--no-provision` is specified.
       Waiting for SSH service on 127.0.0.1:2222, retrying in 3 seconds`

I then upgraded to test-kitchen 3.4.0, and net-ssh to 7.0.1 (the latest), but still had failures with SSH connection. Then tried to upgrade chef gem to 18, and all went downhill from there - I now am trying to resolve gems not being found on my machine.
I've just removed the installation of Chef Workstation and starting from scratch.

Do you believe the username/password authentication would work regardless of the versions of test-kitchen mentioned above?

Below is the OS definition in .kitchen.yml:

- name: linux9
  driver:
      name: vagrant
      provider: virtualbox
      username: vagrant
      password: vagrant
      gui: true
      customize:
            cpus: 2
            clipboard: 'bidirectional'
            memory:  6200
            
  verifier:
          name: inspec
  driver_config:
        username: vagrant
        password: vagrant
        box: 'bento/rockylinux-9' 
        synced_folders:
                - ['../installers', '/installers']
    
  provisioner:
      product_name: chef
      download_url: 'http://packages.cinc.sh/files/stable/cinc/17.10.0/el/7/cinc-17.10.0-1.el7.x86_64.rpm'
      chef_omnibus_root: /opt/cinc
      chef_client_path: /opt/cinc/bin/cinc-client

Corey Hemminger · Answer 7 · Thu Nov 17 2022 02:59:08 GMT+0800 (China Standard Time)

you need to add transport: section to kitchen.yml. I just tested and verified ths works.

- name: linux9
  transport:
      name: ssh
      username: vagrant
      password: vagrant
  driver:
      name: vagrant
      provider: virtualbox
      username: vagrant
      password: vagrant
      gui: true
      customize:
            cpus: 2
            clipboard: 'bidirectional'
            memory:  6200
            
  verifier:
          name: inspec
  driver_config:
        username: vagrant
        password: vagrant
        box: 'bento/rockylinux-9' 
        synced_folders:
                - ['../installers', '/installers']
    
  provisioner:
      product_name: chef
      download_url: 'http://packages.cinc.sh/files/stable/cinc/17.10.0/el/7/cinc-17.10.0-1.el7.x86_64.rpm'
      chef_omnibus_root: /opt/cinc
      chef_client_path: /opt/cinc/bin/cinc-client

gabytsilver · Answer 8 · Thu Nov 17 2022 16:33:16 GMT+0800 (China Standard Time)

@Stromweld - thank you! I will try now, I have to fix my workstation configuration. Which test-kitchen version are you using?

Corey Hemminger · Answer 9 · Thu Nov 17 2022 22:58:23 GMT+0800 (China Standard Time)

I'm using latest chef-workstation:

❯ chef -v
Chef Workstation version: 22.10.1013
Test Kitchen version: 3.3.2
Cookstyle version: 7.32.1
Chef Infra Client version: 17.10.0
Chef InSpec version: 4.56.20
Chef CLI version: 5.6.1
Chef Habitat version: 1.6.521

gabytsilver · Answer 10 · Tue Nov 22 2022 21:36:14 GMT+0800 (China Standard Time)

@Stromweld - thank you for the information! It took me time to test different setups, eventually I was able to continue using an older version of ChefDK, with older version with Kitchen.

I will look into moving our configurations to later Ruby,Kitchen testing, Inspec - I know ChefDK is deprecated and probably bad to still be used.

Thanks again for being attentive and informative! not taken for granted.