Mellanox Infiniband / Ethernet - Mellanox Technologies MT25408A0-FCC-QI ConnectX

[jadsy2107]

Hi All,

if your installing XCP-ng from 7.2 onwards (tested this with 8.2.1) - and using these old NIC's they do not detect and go into Ethernet mode as they did <= 7.2

It's a quick fix for anyone facing this issue.

The newer cards can be flashed with a simple program mlxconfig and set the default mode permanently on hardware, but I coudlnt find any software that worked with these cards, maybe it was designed after this version ConnectX2
mlxconfig supports ConnectX3 and above

The implemented fix:

We need to echo ‘eth’ in the device mode, it would be currently ‘ib’

Step 1. Create script to set mode
vim /etc/infiniband/startup.sh

Paste this: changing your device number if needed (lspci |grep Mellanox will show you yours)
echo eth > /sys/bus/pci/devices/0000:15:00.0/mlx4_port1
echo eth > /sys/bus/pci/devices/0000:15:00.0/mlx4_port2

Save the file and run (to make the new file executable)
chmod +x /etc/infiniband/startup.sh
This will make the device visable in the system, but unfortunately with a random name so we need to tell the linux system what interface name we want by adding a new file at /etc/udev/rules.d/70-persistent-net.rules but before we do that we need the MAC address of the devices to set the name for them

‘ip link’ will show your MAC addresses for your new interfaces as side-XXXX-eth2 and eth3

Step 2..
vim /etc/udev/rules.d/70-persistent-net.rules

Paste this in there (changing your MAC Address to that in step 1 and save:
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="f4:52:14:cf:06:44", ATTR{type}=="1", KERNEL=="eth", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="f4:52:14:cf:06:45", ATTR{type}=="1", KERNEL=="eth", NAME="eth3"

Step 3.
This is not persistent on reboots, so we need to make sure that the script is run;
crontab -e

add:
@reboot /etc/infiniband/startup.sh

Reboot the server, and you should see the new NIC 2 and NIC 3 in XCP-ng Center after a rescan

[Olivier Lambert]

Pinging @Fohdeesha about this

[Jon Sands]

These cards are incredibly old (the original ConnectX series), and will probably be dropped by the mlx4 driver within a year, so I don't think it's worth modifying anything in the XCP-ng project to do this automatically - the two or three users still using these original CX cards can follow his guide above to get them into ETH mode - alternatively as he mentions MFT/mlxconfig no longer supports these cards to "properly" put them into ETH only mode on the card's EEPROM, however older versions of MFT/mlxconfig should be able to - then no OS tuning is necessary