Support infiniband device

Question

Support infiniband device

jcatana opened this issue 5 years ago · comments

Trying to use a downsteam package that references this repo. I ran into an issue using IP over Infiniband device type in a mix ethernet/infiniband environment.

 ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc mq state UP mode DEFAULT group default qlen 256
    link/infiniband 20:00:00:67:fe:80:00:00:00:00:00:00:e0:07:1b:ff:ff:7d:e2:e8 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

Infiniband devices use 12 digit mac addresses and this probably unnecessary sanity check in packet.go causes error.
Lines 90-92:

	if len(srcHW) != len(dstHW) {
		return nil, ErrInvalidHardwareAddr
	}

It assumes all MAC addresses are 6 character.
Removing the check and recompiling allowed downstream Metallb to work on my IP over Infiniband interconnects.

Matt Layher · Answer 1 · Wed Jun 05 2019 02:54:23 GMT+0800 (China Standard Time)

Neat. I'm good with that. Mind sending a PR?

Matt Layher · Answer 2 · Wed Jun 05 2019 02:55:50 GMT+0800 (China Standard Time)

Actually, why would that check cause an issue? The surrounding code just checks for MAC addresses of at least length 6.

jcatana · Answer 3 · Wed Jun 05 2019 02:59:18 GMT+0800 (China Standard Time)

Yeah, that is fine, but the if len(srcHW) != len(dstHW) fails it . My source is 12 and dst is 6

Matt Layher · Answer 4 · Wed Jun 05 2019 03:01:27 GMT+0800 (China Standard Time)

Apologies, I haven't ever tried that. How does that work? I'd expect that you'd be ARPing to another IPoIB device on the other side that also has a 12 byte address.

The ARP packet allocates enough space for the length of the source address at the moment, so the 6 empty bytes on the destination are just ignored?

I'm happy to loosen the restriction, I'd just like to understand what's going on here. If you have some links, that'd be helpful.

Matt Layher · Answer 5 · Wed Jun 05 2019 03:07:50 GMT+0800 (China Standard Time)

In addition, this RFC seems to indicate both addresses would be 20 bytes: https://tools.ietf.org/html/rfc4391#section-9.2.

jcatana · Answer 6 · Sat Jun 08 2019 08:03:04 GMT+0800 (China Standard Time)

Maybe it's upstream for you in "net" because I think that's how you are getting MAC address.
How can I help debug this and see what is being compared at that point? Just add fmt module and print it out? I've tried but I'm fairly bad at go.

Nasser Alansari · Answer 7 · Fri Dec 13 2019 03:59:52 GMT+0800 (China Standard Time)

From MetalLB:
https://github.com/danderson/metallb/blob/30e6f68bade331367ddb3a772593d137af21e4d8/internal/layer2/arp.go#L52

pkt, err := arp.NewPacket(op, a.hardwareAddr, ip, ethernet.Broadcast, ip)

As from the above, it use ethernet.Broadcast as dstHW (Gratuitous ARP). And that's why it always fail for IP over Infiniband( 20bytes)

I belief removing the check would resolve the issue or check for destHW is not equal to ethernet.Broadcast too.

EDIT: MetalLB in L2 mode uses gratuitous ARP.

Matt Layher · Answer 8 · Fri Dec 13 2019 22:35:46 GMT+0800 (China Standard Time)

Let me know if this is all set now, thanks!