MassimoGirondi / rdma-example-cuda

RDMA exmaple. Now with CUDA!

RDMA exmaple

A simple RDMA server client example. The code contains a lot of comments. Here is the workflow that happens in the example:

Client:

1. setup RDMA resources
2. connect to the server
3. receive server side buffer information via send/recv exchange
4. do an RDMA write to the server buffer from a (first) local buffer. The content of the buffer is the string passed with the -s argument.
5. do an RDMA read to read the content of the server buffer into a second local buffer.
6. compare the content of the first and second buffers, and match them.
7. disconnect

Server:

1. setup RDMA resources
2. wait for a client to connect
3. allocate and pin a server buffer
4. accept the incoming client connection
5. send information about the local server buffer to the client
6. wait for disconnect

How to run

git clone https://github.com/animeshtrivedi/rdma-example.git
cd ./rdma-example
cmake .
make

server

./bin/rdma_server

client

atr@atr:~/rdma-example$ ./bin/rdma_client -a 127.0.0.1 -s textstring 
Passed string is : textstring , with count 10 
Trying to connect to server at : 127.0.0.1 port: 20886 
The client is connected successfully 
---------------------------------------------------------
buffer attr, addr: 0x5629832e22c0 , len: 10 , stag : 0x1617b400 
---------------------------------------------------------
...
SUCCESS, source and destination buffers match 
Client resource clean up is complete 
atr@atr:~/rdma-example$ 

Does not have an RDMA device?

In case you do not have an RDMA device to test the code, you can setup SofitWARP software RDMA device on your Linux machine. Follow instructions here: https://github.com/animeshtrivedi/blog/blob/master/post/2019-06-26-siw.md.

CUDA Support

This can be used to test GPU direct support on your hardware!

Compile with -DCUDA=1, the server should be compiled through the Nvidia toolchain, and linked to CUDA libraries (which must be installed beforehand). You also probably need OFED and a Mellanox adapter.

Then the -g 0 flag will instruct the server to allocate memory on the GPU, instead of a standard RAM region.

How to check that the CUDA driver is doing its job

Instead of using the nvidia_peermem driver shipped with the recent driver releases, you can use nv_peer_memory, an older but almost equivalent version of the driver. Before compiling change line 59 of nv_peer_mem.c to print messages, or use the following:

sed -i "s/enable_dbg = 0/ebable_dbg = 1//"  nv_peer_mem.c

Then compile it with make -j and load with sudo insmod nv_peer_mem.ko. Observe the kernel messages with sudo dmesg -w and you should spot operations linked to this module.