Cephadm with v16.2.7 fails to bootstrap

Question

Cephadm with v16.2.7 fails to bootstrap

Yejneshwar opened this issue 3 years ago · comments

Bug Report

What happened:

$sudo cephadm --image quay.io/ceph/ceph:v16.2.7 bootstrap --mon-ip {ip-address} --initial-dashboard-password {password}

Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit systemd-timesyncd.service is enabled and running
Repeating the final host check...
podman|docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit systemd-timesyncd.service is enabled and running
Host looks OK
Cluster fsid: 482b9a30-58e4-11ec-92b9-39ac93e78b62
Verifying IP 192.168.1.231 port 3300 ...
Verifying IP 192.168.1.231 port 6789 ...
Mon IP 192.168.1.231 is in CIDR network 192.168.1.0/24
Pulling container image quay.io/ceph/ceph:v16.2.7...
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting mon public_network...
Creating mgr...
Verifying port 9283 ...
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Wrote config to /etc/ceph/ceph.conf
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/10)...
mgr not available, waiting (2/10)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for Mgr epoch 5...
Mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to to /etc/ceph/ceph.pub
Adding key to root@localhost's authorized_keys...
Adding host mon0...
Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.7 -e NODE_NAME=mon0 -v /var/log/ceph/482b9a30-58e4-11ec-92b9-39ac93e78b62:/var/log/ceph:z -v /tmp/ceph-tmpxwy5z61x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp21f41e1f:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.7 orch host add mon0
/usr/bin/ceph: stderr Error EINVAL: Can not automatically resolve ip address of host where active mgr is running. Please explicitly provide the address.
ERROR: Failed to add host : Failed command: /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.7 -e NODE_NAME=mon0 -v /var/log/ceph/482b9a30-58e4-11ec-92b9-39ac93e78b62:/var/log/ceph:z -v /tmp/ceph-tmpxwy5z61x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp21f41e1f:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.7 orch host add mon0

The below command successfully completes the bootstrap process
$sudo cephadm --image quay.io/ceph/ceph:v16.2.6 bootstrap --mon-ip {ip-address} --initial-dashboard-password {password}

What you expected to happen:

The successful completion of the bootstrap process while using ceph v16.2.7

How to reproduce it (minimal and precise):

Fresh install of OS

$ sudo apt-get install cephadm
$ sudo cephadm --image quay.io/ceph/ceph:v16.2.7 bootstrap --mon-ip {ip-address} --initial-dashboard-password {password}

Environment:

OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Kernel (e.g. uname -a):

Linux mon0 5.11.0-41-generic #45~20.04.1-Ubuntu SMP Wed Nov 10 10:20:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Docker version (e.g. docker version):

Client:
Version: 20.10.7
API version: 1.41
Go version: go1.13.8
Git commit: 20.10.7-0ubuntu5~20.04.2
Built: Mon Nov 1 00:34:17 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

Server:
Engine:
Version: 20.10.7
API version: 1.41 (minimum version 1.12)
Go version: go1.13.8
Git commit: 20.10.7-0ubuntu520.04.2
Built: Fri Oct 22 00:45:53 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.5-0ubuntu320.04.1
GitCommit:
runc:
Version: 1.0.1-0ubuntu2~20.04.1
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:

Ceph version (e.g. ceph -v):

ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)

Adam King · Answer 1 · Fri Dec 10 2021 04:41:23 GMT+0800 (China Standard Time)

It looks like, from my own testing, the version of cephadm that is installed using sudo apt-get install cephadm on a fresh Ubuntu 20.04 system is an older, Octopus version. I don't think this problem would happen with a recent Pacific version of the binary.

Yejneshwar · Answer 2 · Fri Dec 10 2021 17:12:07 GMT+0800 (China Standard Time)

The same cephadm command, but with Ceph v16.2.6 still works on a fresh install of Ubuntu 20.04

A little bit of insight into the error

Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.7 -e NODE_NAME=mon0 -v /var/log/ceph/482b9a30-58e4-11ec-92b9-39ac93e78b62:/var/log/ceph:z -v /tmp/ceph-tmpxwy5z61x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp21f41e1f:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.7 orch host add mon0

/usr/bin/ceph: stderr Error EINVAL: Can not automatically resolve ip address of host where active mgr is running. Please explicitly provide the address.

this needs to be :

/usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.7 -e NODE_NAME=mon0 -v /var/log/ceph/482b9a30-58e4-11ec-92b9-39ac93e78b62:/var/log/ceph:z -v /tmp/ceph-tmpxwy5z61x:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp21f41e1f:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.7 orch host add mon0 _{ip-address}_

Reason

While adopting an existing cluster to cephadm the error "Can not automatically resolve ip address of host where active mgr is running. Please explicitly provide the address." is the same error we get when running the command ceph orch host add mon0 even on Ceph v16.2.6 ( I would assume even on the other older versions ) and is resolved with the command ceph orch host add mon0 {ip_address}

Hope this helps.

stale · Answer 3 · Sun Jan 09 2022 22:39:08 GMT+0800 (China Standard Time)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.