cf dev start: unable to find VM IP address from '/var/lib/libvirt/dnsmasq/virbr0.status'

Question

cf dev start: unable to find VM IP address from '/var/lib/libvirt/dnsmasq/virbr0.status'

karizma987 opened this issue 5 years ago · comments

although i found the same error being solved but it is not working for me. i am new to cloud foundry.
i am trying to install cloud foundry dev.
i used the following command.

sudo apt install qemu-kvm libvirt-bin libvirt-doc
wget -q -O - https://packages.cloudfoundry.org/debian/cli.cloudfoundry.org.key | sudo apt-key add -
echo "deb https://packages.cloudfoundry.org/debian stable main" | sudo tee /etc/apt/sources.list.d/cloudfoundry-cli.list
# ...then, update your local package index, then finally install the cf CLI
sudo apt-get update
sudo apt-get install cf-cli

OUTOUT AFTER THE LAST COMMAND

Reading package lists... Done
Building dependency tree       
Reading state information... Done
cf-cli is already the newest version (6.45.0).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

next i run
cf install-plugin -r CF-Community cfdev

output

Plugin cfdev 0.0.16 found in: CF-Community
Attention: Plugins are binaries written by potentially untrusted authors.
Install and use plugins at your own risk.
Do you want to install the plugin cfdev? [yN]: y
Starting download of plugin binary from repository CF-Community...
 15.78 MiB / 15.78 MiB [============================================] 100.00% 2s
Installing plugin cfdev...
OK

Plugin cfdev 0.0.16 successfully installed.

**on running the command**
cf dev start
i am getting the following output

Downloading Resources...
Progress: |====================>| 100.0%
Setting State...

CF Dev collects anonymous usage data to help us improve your user experience. We intend to share these anonymous usage analytics with user community by publishing quarterly reports at :
		
https://github.com/pivotal-cf/cfdev/wiki/Telemetry
		
Are you ok with CF Dev periodically capturing anonymized telemetry [y/N]?> n
WARNING: CF Dev requires 8192 MB of RAM to run. This machine may not have enough free RAM.
[sudo] password for karishma: 
Creating the VM...
Starting the VM...
Fetching VM Address...
FAILED
cf dev start: unable to find VM IP address from '/var/lib/libvirt/dnsmasq/virbr0.status'

did i miss any step?
i am not able to rectify the error.

cf-gitbot · Answer 1 · Mon Jul 01 2019 17:12:32 GMT+0800 (China Standard Time)

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/167009440

The labels on this github issue will be updated when the story is started.

Anthony Emengo · Answer 2 · Mon Jul 01 2019 21:28:49 GMT+0800 (China Standard Time)

The following user that posted a similar issue was missing some other system packages. Could that be the case here?

#104 (comment)

karizma987 · Answer 3 · Tue Jul 02 2019 17:42:28 GMT+0800 (China Standard Time)

@aemengo
i saw the #104 (comment)
i executed the the commands

virsh net-list

output

 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes

service libvirtd status
output

 libvirtd.service - Virtualization daemon
   Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset:
   Active: active (running) since Tue 2019-07-02 19:46:27 IST; 4h 44min left
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 1034 (libvirtd)
    Tasks: 19 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─1034 /usr/sbin/libvirtd
           ├─1318 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default
           └─1319 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default

Jul 02 20:00:02 karishma-HP-Pavilion-Notebook dnsmasq[1318]: reading /etc/resolv
Jul 02 20:00:02 karishma-HP-Pavilion-Notebook dnsmasq[1318]: using nameserver 12
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: reading /etc/resolv
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: using nameserver 12
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: reading /etc/resolv
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: using nameserver 12
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: reading /etc/resolv
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: using nameserver 12
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: reading /etc/resolv
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: using nameserver 12
lines 1-22/22 (END)...skipping...
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-07-02 19:46:27 IST; 4h 44min left
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 1034 (libvirtd)
    Tasks: 19 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─1034 /usr/sbin/libvirtd
           ├─1318 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_le
           └─1319 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_le

Jul 02 20:00:02 karishma-HP-Pavilion-Notebook dnsmasq[1318]: reading /etc/resolv.conf
Jul 02 20:00:02 karishma-HP-Pavilion-Notebook dnsmasq[1318]: using nameserver 127.0.0.53#53
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: reading /etc/resolv.conf
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: using nameserver 127.0.0.53#53
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: reading /etc/resolv.conf
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: using nameserver 127.0.0.53#53
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: reading /etc/resolv.conf
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: using nameserver 127.0.0.53#53
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: reading /etc/resolv.conf
Jul 02 20:00:13 karishma-HP-Pavilion-Notebook dnsmasq[1318]: using nameserver 127.0.0.53#53

virsh net-start default
output

error: Failed to start network default
error: Requested operation is not valid: network is already active

georgi-bozhinov · Answer 4 · Tue Jul 02 2019 20:43:53 GMT+0800 (China Standard Time)

I had the same error and it turns out virtualization wasn't enabled on my machine. The issue was fixed by enabling it.

karizma987 · Answer 5 · Wed Jul 03 2019 15:42:41 GMT+0800 (China Standard Time)

@georgi-bozhinov i check my system, virtualisation is already enabled. i tried to do the whole process again. still geting the same error.
cf dev start: unable to find VM IP address from '/var/lib/libvirt/dnsmasq/virbr0.status'

Anthony Emengo · Answer 6 · Wed Jul 03 2019 23:26:07 GMT+0800 (China Standard Time)

@karizma987 Is that file present? What are the contents of /var/lib/libvirt/dnsmasq/virbr0.status. Is there any file present at this directory: /var/lib/libvirt/dnsmasq/?

Henlo · Answer 7 · Thu Jul 04 2019 02:16:10 GMT+0800 (China Standard Time)

Had the same problem as @georgi-bozhinov where virtualization was not enabled on the machine. Works now that it is enabled.

karizma987 · Answer 8 · Sat Jul 06 2019 00:18:13 GMT+0800 (China Standard Time)

@aemengo i check the file in var/lib/libvirt/dnsmasq/virbr0.status
the file is present but its empty. when i open it is shows square brackets like this
[

]

and no other content.

karizma987 · Answer 9 · Sat Jul 06 2019 00:21:57 GMT+0800 (China Standard Time)

@eddie-at-ibm i use the following command to check whether virtualization is enabled or not.

sudo modprobe msr
sudo rdmsr 0x3a
Values 3 and 5 mean it's activated.
and my system is returning 5 and even checked the system setup setting virtualization is enabled.
and the problem still persist.

Anthony Emengo · Answer 10 · Sat Jul 06 2019 02:07:32 GMT+0800 (China Standard Time)

@karizma987

Does the following command return anything? ps aux | grep linuxkit
Are there any files that can be found in: ~/.cfdev/log?

karizma987 · Answer 11 · Sun Jul 07 2019 02:21:00 GMT+0800 (China Standard Time)

@aemengo ps aux | grep linuxkit gives the below output

karishma  4353  0.0  0.0  21536   972 pts/0    S+   23:35   0:00 grep --color=auto linuxkit
root     32651  0.0  0.0 627804   108 ?        Ssl  23:16   0:00 /home/karishma/.cfdev/daemons/linuxkit
root     32659  0.0  0.0 158516   748 ?        Sl   23:16   0:00 /home/karishma/.cfdev/bin/linuxkit run qemu -cpus 4 -mem 8192 -disk size=120G,format=qcow2,file=/home/karishma/.cfdev/state/linuxkit/disk.qcow2 -fw /home/karishma/.cfdev/bin/OVMF.fd -state /home/karishma/.cfdev/state/linuxkit -networking tap,cfdevtap0 -iso -uefi /home/karishma/.cfdev/bin/cfdev-efi-v2.iso
root     32664  4.8 76.5 11140400 6162252 ?    Sl   23:16   0:56 /usr/bin/qemu-system-x86_64 -smp 4 -m 8192 -uuid 7841b966-0625-4d76-bc15-02df24205d81 -pidfile /home/karishma/.cfdev/state/linuxkit/qemu.pid -enable-kvm -machine q35,accel=kvm:tcg -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -drive file=/home/karishma/.cfdev/state/linuxkit/disk.qcow2,format=qcow2,index=0,media=disk -boot d -cdrom /home/karishma/.cfdev/bin/cfdev-efi-v2.iso -drive if=pflash,format=raw,file=/home/karishma/.cfdev/bin/OVMF.fd -device virtio-net-pci,netdev=t0,mac=02:38:a6:db:87:79 -netdev tap,id=t0,ifname=cfdevtap0,script=no,downscript=no -nographic

inside ~/.cfdev/log

linuxkit.log is present.

one new thing happened, i tried reinstalling again the cf dev.... but this time the error i got is this:-

Creating the VM...
Starting the VM...
Fetching VM Address...
FAILED
cf dev start: unexpected end of JSON input

now i don't know whether the previous error is rectified or i got a new error in addition. trying finding the solution for this error now. will update soon further about the progress.

Anthony Emengo · Answer 12 · Sun Jul 07 2019 02:22:53 GMT+0800 (China Standard Time)

What are the last few lines of the linuxkit.log file?

karizma987 · Answer 13 · Sun Jul 07 2019 02:37:51 GMT+0800 (China Standard Time)

@aemengo can you tell where to find this linuxkit.log file? i used the terminal to see the content of cfdev/log but not able to see the content of linuxkit.log

Anthony Emengo · Answer 14 · Sun Jul 07 2019 02:41:28 GMT+0800 (China Standard Time)

Can you perform the following and tell me the output?

$ tail -n 30 ~/.cfdev/log/linuxkit.log

karizma987 · Answer 15 · Sun Jul 07 2019 02:43:22 GMT+0800 (China Standard Time)

@aemengo got this as output

INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[0000] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/run/containerd/debug.sock"
INFO[0000] serving...                                    address="/run/containerd/containerd.sock"
INFO[0000] containerd successfully booted in 0.367197s  
Cleaning up stale service: "acpid"
Cleaning up stale service: "bosh-bootstrap"
Cleaning up stale service: "host-timesync-daemon"
Cleaning up stale service: "ntpd"
Cleaning up stale service: "vpnkit-forwarder"
Cleaning up stale service: "vpnkit-manager"
INFO[0003] shim containerd-shim started                  address="/containerd-shim/services.linuxkit/acpid/shim.sock" debug=false pid=519
INFO[0005] shim containerd-shim started                  address="/containerd-shim/services.linuxkit/bosh-bootstrap/shim.sock" debug=false pid=565
INFO[0006] shim containerd-shim started                  address="/containerd-shim/services.linuxkit/host-timesync-daemon/shim.sock" debug=false pid=612
INFO[0007] shim containerd-shim started                  address="/containerd-shim/services.linuxkit/ntpd/shim.sock" debug=false pid=681
INFO[0007] shim containerd-shim started                  address="/containerd-shim/services.linuxkit/vpnkit-manager/shim.sock" debug=false pid=733
[CPID] 2019/07/06 17:53:48 Initializing bosh-runc-cpid...
[ 1162.048558] nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based  firewall rule not found. Use the iptables CT target to attach helpers instead.

Anthony Emengo · Answer 16 · Tue Jul 09 2019 05:20:32 GMT+0800 (China Standard Time)

I'm really sorry for the delay in responding, but unfortunately there's still not much to go on. Something must be occurring wrongly with the libvirt interaction in a way that is not quite surfacing itself.

karizma987 · Answer 17 · Wed Jul 10 2019 15:35:04 GMT+0800 (China Standard Time)

@aemengo thank you for responding, i am working on the issue, will update incase i am able to move forward in the installation.

Richard Cox · Answer 18 · Thu Jul 11 2019 00:55:31 GMT+0800 (China Standard Time)

@karizma987 I hit the same issue today, however my virsh net-list --all was missing default. Found this and this really helpful.

karizma987 · Answer 19 · Mon Jul 15 2019 22:34:47 GMT+0800 (China Standard Time)

@richard-cox i tried virsh net-list --all

output is

 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes

so that means default is already active i guess.

Tosone · Answer 20 · Tue Sep 17 2019 17:42:53 GMT+0800 (China Standard Time)

I found this line, timeout is too short for me.

Tosone · Answer 21 · Tue Sep 17 2019 18:36:35 GMT+0800 (China Standard Time)

How to compile this repo? I want to fix some bugs and tests on my own. @aemengo

gitricko · Answer 22 · Sun Jul 05 2020 16:10:29 GMT+0800 (China Standard Time)

i have the same error... cat /var/lib/libvirt/dnsmasq/virbr. gives me 3 IPs... is it the timeout that cause the issue. Maybe the program need to wait longer ?