The idea is to connect FPGA boards using USB to a raspberry.
The raspberry provides the USB devices using USBIP to a client.
We have the following components on the server (raspberry):
- udev rules to enable provision of specific device over usbip
- systemd config files
- kernel configuration: default dwc-otg module crashes and needs to be replaced by dwc2
For the client we have:
- a python script to drive uspip for a set of devices
- a recompilation/reconfiguration of the vhci-hcd kernel module since it provides too few USB virtual devices
Configuring a usb device while the the server is running is different as compared to booting the server while the devices are already connected. The udev rules are invoked early in the boot process while the usbip daemon is required to provide IP access to the usb device. In fact systemd-udevd.service is started well before usbipd.service.
So, a rule like
SUBSYSTEM=="usb" ATTR{idVendor}=="0403" ATTR{idProduct}=="6001" \
ACTION=="add" \
RUN+="/usr/bin/usbip bind -b $KERNEL"
fails since the usbip daemon is not yet started. To avoid possible dependency issues I decided to postpone the usb binding process using systemd.
Instead, such a rule is now formulated as:
SUBSYSTEM=="usb" ATTR{idVendor}=="0403" ATTR{idProduct}=="6001" \
ACTION=="add" \
TAG+="systemd" ENV{SYSTEMD_WANTS}+="usbip-bind@%k.service"
Please note the following:
- before usbip bind one needs to start usbipd to service the request.
- for the USB device use ATTR and not ATTRS for idVendor and idProduct to avoid multiple invocations of the udev rule
- TAG+="systemd" is required to enable SYSTEMD_WANTS correctly.
As a consequence we need to define two systemd services. The first one is usbipd.service to provide the USB devices over IP. It only starts the usbipd daemon after the network is available (which actually might only occur after completing systemd-udev.service).
The second service is a template service named usbip-bind@.service. This is invoked using the udev rule above using the variable SYSTEMD_WANTS. In this second service there exists a dependency that is only starts after usbipd.service has been started. Effectively delaying the binding of the USB device.
Use dwc2 instead of otg_dwc
On the raspberry, add the following line to /boot/config
dtoverlay=dwc2
Also add the folowing option to the kernel cmdline in /boot/cmdline.txt
modules-load=dwc2
That will replace the dwc-otg driver by dwc2, since the dwc-otg does not work at this time.
The directories systemd and udev can be rsync-ed to the server to provide the udev rules and the systemd services.
The kernel configuration needs to be done by hand.
On the client you can find what USB devices are exported from the server using usbip
$ usbip list -r <server>
Then you can attach a USB device with
$ usbip attach -r <server> -b <busid>
With for example being 1-1.2.1, i.e. the busid on the server. Using usbview you can find a USB/IP Virtual Host Controller under which the attached USB devices can be found. There exist default 2 controller, one for USB2 and one for USB3, i.e. 480Mb/s and 5GB/s. Each virtual host controller allows for maximal eight attached devices.
Often, having 8 virtual USB devices is not sufficient. One would like to increase the number of devices which can be attached at the same time. Sadly, the kernel module in which the max number of devices is defined can only be re-configured at compile time. I.e. a module parameter to dynamically extend the amount of devices does not exist at this time.
Therefore, the kernel module vhci_hcd needs to be recompiled to increase the maximum number of virtual USB devices. Two variables are used for this:
CONFIG_USBIP_VHCI_HC_PORTS=10
CONFIG_USBIP_VHCI_NR_HCS=2
The first increases the maximum number of devices to 10 for a virtual host controller. With the second parameters one can increase the number of host controllers. Filling in 2 means here that in total 4 virtual host controllers are created, 2 for 480Mb/s and 2 for 5Gb/s.
Bear in mind that possibly in other parts of the kernel the amount of USB devices is limited as well: I did not investigate this further.
If needed the kernel module re-compilation needs to be done by hand. Further no extras need to be installed.
On Altera boards the default device id for a JTAG-blaster is 09fb:6810. This is the situation when the board is powered up. However, when you start the jtagd using jtagconfig the device is being re-programmed and turned into a device with id 09fb:6010.
This kills the usbip connection.
So after starting the jtagd one has to re-attach the USB device as describe above. This operation needs be to repeated after the FPGA board went through a power cycle. The modified device seems to have become a dual uart of which one is used for the JTAG programmer.
As inital step a python script in scripts helps in performing the attachement of USB devices which needs to be extende to become more easily usable.
In this script a number of devices and the server are predefined, this needs to be done using an .ini file or similar.