Sykube Error while executing singularity

Question

Sykube Error while executing singularity

AroArz opened this issue 4 years ago · comments

Hi, was hoping to use Sykube to test K8s with Singularity-CRI locally but have encountered the following issues. As far as I understood it there were no pre-requisites to running Sykube other than having Singularity > 3.2 installed? Appreciative of any help

What happened

(base) aron@debian:~/Desktop/aron/sykube$ sykube init --local-image /tmp/sykube.sif
[+] Installing sykube in /var/lib/sykube
[+] Bootstrap log are available at /var/lib/sykube/bootstrap.log
[+] Create sykube nodes directory
[+] Write network config
[+] Copying Sykube /tmp/sykube.sif image to /var/lib/sykube-image/sykube.sif
[+] Starting master node
[+] Master node bootstrap
[+] Add sykube-master to /etc/hosts
[+] Running kubeadm sykube-master
[-] Error while executing singularity

--------------- LOGS -----------------
/usr/lib/python2.7/dist-packages/supervisor/options.py:297: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '

INFO:    instance started successfully
[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta2, Kind=JoinConfiguration
[init] Using Kubernetes version: v1.16.3
[preflight] Running pre-flight checks
        [WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
        [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.16.3: output: time="2020-07-21T14:48:17+02:00" level=fatal msg="pulling image failed: rpc error: code = Internal desc = could not pull image: could not pull image: could not build image: \x1b[34mINFO:   \x1b[0m Starting build...\n\x1b[31mFATAL:  \x1b[0m While performing build: conveyor failed to get: pinging docker registry returned: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 1.1.1.1:53: read udp 10.11.11.2:58529->1.1.1.1:53: i/o timeout\n"
, error: exit status 1
        [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.16.3: output: time="2020-07-21T14:48:37+02:00" level=fatal msg="pulling image failed: rpc error: code = Internal desc = could not pull image: could not pull image: could not build image: \x1b[34mINFO:   \x1b[0m Starting build...\n\x1b[31mFATAL:  \x1b[0m While performing build: conveyor failed to get: pinging docker registry returned: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 1.1.1.1:53: read udp 10.11.11.2:37874->1.1.1.1:53: i/o timeout\n"
, error: exit status 1
        [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.16.3: output: time="2020-07-21T14:48:57+02:00" level=fatal msg="pulling image failed: rpc error: code = Internal desc = could not pull image: could not pull image: could not build image: \x1b[34mINFO:   \x1b[0m Starting build...\n\x1b[31mFATAL:  \x1b[0m While performing build: conveyor failed to get: pinging docker registry returned: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 1.1.1.1:53: read udp 10.11.11.2:34788->1.1.1.1:53: i/o timeout\n"
, error: exit status 1
        [WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.16.3: output: time="2020-07-21T14:49:17+02:00" level=fatal msg="pulling image failed: rpc error: code = Internal desc = could not pull image: could not pull image: could not build image: \x1b[34mINFO:   \x1b[0m Starting build...\n\x1b[31mFATAL:  \x1b[0m While performing build: conveyor failed to get: pinging docker registry returned: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 1.1.1.1:53: read udp 10.11.11.2:60549->1.1.1.1:53: i/o timeout\n"
, error: exit status 1
        [WARNING ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: time="2020-07-21T14:49:37+02:00" level=fatal msg="pulling image failed: rpc error: code = Internal desc = could not pull image: could not pull image: could not build image: \x1b[34mINFO:   \x1b[0m Starting build...\n\x1b[31mFATAL:  \x1b[0m While performing build: conveyor failed to get: pinging docker registry returned: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 1.1.1.1:53: read udp 10.11.11.2:40678->1.1.1.1:53: i/o timeout\n"
, error: exit status 1
        [WARNING ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.15-0: output: time="2020-07-21T14:49:57+02:00" level=fatal msg="pulling image failed: rpc error: code = Internal desc = could not pull image: could not pull image: could not build image: \x1b[34mINFO:   \x1b[0m Starting build...\n\x1b[31mFATAL:  \x1b[0m While performing build: conveyor failed to get: pinging docker registry returned: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 1.1.1.1:53: read udp 10.11.11.2:55211->1.1.1.1:53: i/o timeout\n"
, error: exit status 1
        [WARNING ImagePull]: failed to pull image k8s.gcr.io/coredns:1.6.2: output: time="2020-07-21T14:50:17+02:00" level=fatal msg="pulling image failed: rpc error: code = Internal desc = could not pull image: could not pull image: could not build image: \x1b[34mINFO:   \x1b[0m Starting build...\n\x1b[31mFATAL:  \x1b[0m While performing build: conveyor failed to get: pinging docker registry returned: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 1.1.1.1:53: read udp 10.11.11.2:45831->1.1.1.1:53: i/o timeout\n"
, error: exit status 1
[kubelet-start] WARNING: unable to stop the kubelet service momentarily: [exit status 1]
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[kubelet-start] WARNING: unable to start the kubelet service: [failed to reload systemd: exit status 1]
[kubelet-start] Please ensure kubelet is reloaded and running manually.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [sykube-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.11.11.2]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [sykube-master localhost] and IPs [10.11.11.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [sykube-master localhost] and IPs [10.11.11.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
        - 'docker ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
--------------------------------------

How to reproduce it (as minimally and precisely as possible):

singularity version 3.6.0
Sykube according to https://sylabs.io/guides/cri/1.0/user-guide/sykube.html