Hardware messages monitoring
It is possible to monitor all messages driver is sending down to hardware and
messages coming from hardware back to the driver. This is implemented
as a kernel tracepoint. The tracepoint name is devlink:devlink_hwmsg.
There is a simple python script called devlink-hwmsg.py that hooks up
on this tracepoint and converts events into PCAP output.
You can get it like this:
sw:~$ git clone https://github.com/jpirko/hwmsg_tracing.git
Then just use it from your desktop connecting it to Wireshark for easier view like this:
desktop:~$ ssh sw sudo ~/hwmsg_tracing/devlink-hwmsg.py | sudo wireshark -k -i -
In case the devlink-hwmsg.py complains about missing tracepoint object,
you need to update python-perf package. For example like this:
sw:~$ sudo dnf install https://kojipkgs.fedoraproject.org//packages/kernel/4.8.0/0.rc7.git4.1.fc25/x86_64/python-perf-4.8.0-0.rc7.git4.1.fc25.x86_64.rpm -y
Also make sure that devlink module is loaded before you run the script.
Processing hardware messages
Hardware message packets contain data in a custom TLV-based format. The tool
bwz.py can be used to filter this data based on values inside individual TLV
fields, and to slice the data (i.e. omit some TLV records or remove the TLV
headers altogether).
bwz includes a simple language for expressing filtering (-f) and slicing
(-s). See --help for details of operation. E.g. to include just the payload
of messages that the kernel sends out, you could pipe devlink-hwmsg.py output
as follows:
sw:~$ devlink-hwmsg.py | bwz.py -f outgoing -s buf | ...
To select complete messages from a particular driver, you could do this:
sw:~$ devlink-hwmsg.py | bwz.py -f 'driver == "mlxsw_spectrum"' | ...
It's possible to query substrings of individual values using Python array subscript syntax. E.g. if you know the message buffer is in Mellanox EMAD format, you can do the following to select all messages related to the RAUHT register (whose ID is 0x8014):
sw:~$ ... | bwz.py -f 'buf[0x14:0x16] == "\x80\x14"' | ...
One can combine the conditions using & and | for "and" and "or". Make sure
you parenthesize the combined expressions, & and | don't have the right
precedence in Python. When in doubt, use the command line argument --show to
have bwz dump how it understands the task.
sw:~$ bwz.py --show -f '(driver == "mlxsw_spectrum") & outgoing & \
(buf[0x14:0x16] == "\x80\x14")'
filter=(((driver == 'mlxsw_spectrum') & (~incoming)) & ((buf[slice(20, 22, None)]) == '\x80\x14'))
slice=tlv(bus, dev, driver, incoming, type, buf)
bwz depends on pcapy, a Python package for processing pcap files.