Network_interface.Internal_error("Not_found") - /proc/net/bonding/

[bugsyb]

For some reasons running
xe pif-scan host-uuid=....

it looks for /proc/net/bonding/<bridge_interface_name> whilst it does not exist on normal system. Tested on cleanly installed system too.
System in question: Ubuntu 14.04 LTS, Xen 4.4 (amd64).
I'm guessing it might be related to:
#38

bridge setup in standard Linux way using brctl and properly visible at Xen level.

xe network-list
uuid ( RO) : 5369a468-f639-b7ec-ddc1-b1d5861da84e
name-label ( RW): Pool-wide network associated with em1
name-description ( RW):
bridge ( RO): brem1

em1 is standard ethernet interface.

xe pif-list
uuid ( RO) : b6a218d3-8f7b-f9cd-c1f1-3f16a63cc785
device ( RO): em1
currently-attached ( RO): true
VLAN ( RO): -1
network-uuid ( RO): 5369a468-f639-b7ec-ddc1-b1d5861da84e

trying to scan for interfaces:
xe pif-scan host-uuid=0d83e699-df0b-40db-9934-b5cbc9e5eafe
The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.

Syslog messages:
Jan 2 14:06:09 silver xcp-networkd: [error|silver|174|PIF.scan R:01e4a6892f20|network_utils] Error: could not read /proc/net/bonding/brem1.

Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi||cli] Xapi_cli.exception_handler: Got exception INTERNAL_ERROR: [ Network_interface.Internal_error("Not_found") ]

Full dump:
Jan 2 14:06:09 silver xapi: [ info|silver|1989 UNIX /var/lib/xcp/xapi||cli] xe pif-scan host-uuid=0d83e699-df0b-40db-9934-b5cbc9e5eafe username=root password=(omitted)
Jan 2 14:06:09 silver xapi: [ info|silver|1989 UNIX /var/lib/xcp/xapi|session.login_with_password D:a1aba3612cfd|xapi] Session.create trackid=25343f1677c75cc6c8b5dbee602036ba pool=false uname=root originator=cli is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi|session.login_with_password D:a1aba3612cfd|mscgen] xapi=>xapi [label="session.get_uuid"];
Jan 2 14:06:09 silver xapi: [debug|silver|1990 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.get_uuid D:b7d7411afdb3 created by task D:a1aba3612cfd
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi|PIF.scan R:01e4a6892f20|audit] PIF.scan: host = '0d83e699-df0b-40db-9934-b5cbc9e5eafe (silver)'
Jan 2 14:06:09 silver xcp-networkd: [error|silver|174|PIF.scan R:01e4a6892f20|network_utils] Error: could not read /proc/net/bonding/brem1.
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi|PIF.scan R:01e4a6892f20|backtrace] Called from file "list.ml", line 55, characters 20-23\nCalled from file "xapi_pif.ml", line 261, characters 4-68\nCalled from file "xapi_pif.ml", line 479, characters 9-37\nCalled from file "message_forwarding.ml", line 233, characters 25-44\nCalled from file "rbac.ml", line 227, characters 16-23\n
Jan 2 14:06:09 silver xapi: 20150102T13:06:09.062Z|audit|silver|1989 UNIX /var/lib/xcp/xapi|PIF.scan R:01e4a6892f20|audit ]' 'API' 'PIF.scan' (('host' 'silver' '0d83e699-df0b-40db-9934-b5cbc9e5eafe' 'OpaqueRef:0d0d7bf7-4d03-afc4-4280-8724014224ba')))
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi|PIF.scan R:01e4a6892f20|backtrace] Raised at file "rbac.ml", line 236, characters 10-15\nCalled from file "server_helpers.ml", line 75, characters 11-41\n
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi|PIF.scan R:01e4a6892f20|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Network_interface.Internal_error("Not_found") ]
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi|PIF.scan R:01e4a6892f20|dispatcher] Raised at file "rbac.ml", line 236, characters 10-15\nCalled from file "server_helpers.ml", line 75, characters 11-41\n
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi|PIF.scan R:01e4a6892f20|backtrace] Raised at file "rbac.ml", line 236, characters 10-15\nCalled from file "server_helpers.ml", line 75, characters 11-41\n
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi|PIF.scan R:01e4a6892f20|xapi] Raised at file "server_helpers.ml", line 90, characters 14-15\nCalled from file "lib/pervasiveext.ml", line 22, characters 2-9\n
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi|dispatch:PIF.scan D:a8b297444b68|xapi] Raised at file "lib/pervasiveext.ml", line 26, characters 22-25\nCalled from file "lib/debug.ml", line 57, characters 17-20\nRe-raised at file "lib/debug.ml", line 62, characters 10-11\nCalled from file "lib/pervasiveext.ml", line 22, characters 2-9\n
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi|dispatch:PIF.scan D:a8b297444b68|backtrace] Raised at file "lib/pervasiveext.ml", line 26, characters 22-25\nCalled from file "lib/debug.ml", line 57, characters 17-20\nRe-raised at file "lib/debug.ml", line 62, characters 10-11\nCalled from file "lib/pervasiveext.ml", line 22, characters 2-9\nRe-raised at file "lib/pervasiveext.ml", line 26, characters 22-25\nCalled from file "server_helpers.ml", line 136, characters 10-106\nCalled from file "server.ml", line 19252, characters 23-161\nCalled from file "server_helpers.ml", line 115, characters 4-7\n
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi||xapi] Raised at file "client.ml", line 6, characters 37-75\nCalled from file "client.ml", line 18, characters 55-110\nCalled from file "client.ml", line 8682, characters 6-53\nCalled from file "cli_operations.ml", line 3672, characters 10-45\nCalled from file "xapi_cli.ml", line 112, characters 18-56\nCalled from file "lib/pervasiveext.ml", line 22, characters 2-9\n
Jan 2 14:06:09 silver xapi: [ info|silver|1989 UNIX /var/lib/xcp/xapi|session.logout D:e73c5f0ec695|xapi] Session.destroy trackid=25343f1677c75cc6c8b5dbee602036ba
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi||backtrace] Raised at file "lib/pervasiveext.ml", line 26, characters 22-25\nCalled from file "xapi_cli.ml", line 111, characters 2-138\nCalled from file "xapi_cli.ml", line 221, characters 7-44\nCalled from file "xapi_cli.ml", line 273, characters 4-23\n
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi||cli] Xapi_cli.exception_handler: Got exception INTERNAL_ERROR: [ Network_interface.Internal_error("Not_found") ]
Jan 2 14:06:09 silver xapi: [debug|silver|1989 UNIX /var/lib/xcp/xapi||cli] Raised at file "lib/pervasiveext.ml", line 26, characters 22-25\nCalled from file "xapi_cli.ml", line 111, characters 2-138\nCalled from file "xapi_cli.ml", line 221, characters 7-44\nCalled from file "xapi_cli.ml", line 273, characters 4-23\n

[Si Beaumont]

There is a problem with 3.12 and newer kernels. We are working on a fix but essentially (quoting @dvrabel)...

This is because xcp-networkd assumes that the presence /sys/class/net/<dev>/master implies that is a bond. But as of Linux 3.12, openvswitch ports have a master link to the bridge device.

I think xcp-networkd should check /sys/class/net/<dev>/master/bonding/slaves exists and contains before treating as a Linux bond slave.

This bug breaks XenServer installations because one of the first boot scripts does a xe pif-scan and this fails.

[bugsyb]

Thanks!

Good to know it's known issue, so hopefully will be fixed some time in the
future.

Regards,
Dawid

2015-01-07 15:37 GMT+01:00 Si Beaumont [e-mail]:

There is a problem with 3.12 and newer kernels. We are working on a fix
but essentially (quoting @dvrabel https://github.com/dvrabel)...

This is because xcp-networkd assumes that the presence
/sys/class/net//master implies that is a bond. But as of Linux 3.12,
openvswitch ports have a master link to the bridge device.

I think xcp-networkd should check
/sys/class/net//master/bonding/slaves exists and contains before
treating as a Linux bond slave.

This bug breaks XenServer installations because one of the first boot
scripts does a xe pif-scan and this fails.

—
Reply to this email directly or view it on GitHub
#48 (comment)
.

[Si Beaumont]

@bugsyb said:

hopefully will be fixed some time in the future.

It's being worked on right now in fact :)