Failed to disconnect RDMA CM connection with ib_write_lat test

[Anumula-Murali-Mohan-Reddy]

issue: client invokes rdma_disconnect prior to server in ib_write_lat test, client qp state changes from idle to closing and it will send FIN to the server, and on server side, peer_close will change qp state from RTS to closing and close_con_rpl will change qp state from closing to idle. Now server qp state is idle, server is calling rdma_disconnect and trying to change server qp state from idle to closing, but this is not allowed, which results in an RDMA CM connection error. Therefore server and client are not closing synchronously.

Server
[root@beagle5 perftest]# ib_write_lat -n 10 -R -s 2048 -D 10 -p 10000
Waiting for client to connect
RDMA_Write Latency Test
Dual-port : OFF Device : cxgb4_0
Number of qps : 1 Transport type : IW
Connection type : RC Using SRQ : OFF
PCIe relax order: OFF
ibv_wr* API : OFF
Mtu : 1024[B]
Link type : Ethernet
GID index : 0
Max inline data : 220[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
Waiting for client rdma_cm QP to connect
Please run the same command with the IB/RoCE interface IP
local address: LID 0000 QPN 0x0402 PSN 0xee0d9a
GID: 00:07:67:60:177:64:00:00:00:00:00:00:00:00:00:00
remote address: LID 0000 QPN 0x040a PSN 0x430988
GID: 00:07:67:57:141:96:00:00:00:00:00:00:00:00:00:00
#bytes #iterations t_avg[usec] tps average
Conflicting CPU frequency values detected: 3800.000000 != 2987.817000. CPU Frequency is not max.
Conflicting CPU frequency values detected: 3626.812000 != 3800.000000. CPU Frequency is not max.
2048 957120 3.13 159520.05
Failed to disconnect RDMA CM connection.
ERRNO: Invalid argument.
Failed to disconnect RDMA CM nodes.
[root@beagle5 perftest]#

Client
[root@beagle6 perftest]# ib_write_lat -n 10 -R -s 2048 -D 10 -p 10000 91.91.91.6
RDMA_Write Latency Test
Dual-port : OFF Device : cxgb4_0
Number of qps : 1 Transport type : IW
Connection type : RC Using SRQ : OFF
PCIe relax order: OFF
ibv_wr* API : OFF
TX depth : 1
Mtu : 1024[B]
Link type : Ethernet
GID index : 0
Max inline data : 220[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
local address: LID 0000 QPN 0x040a PSN 0x430988
GID: 00:07:67:57:141:96:00:00:00:00:00:00:00:00:00:00
remote address: LID 0000 QPN 0x0402 PSN 0xee0d9a
GID: 00:07:67:60:177:64:00:00:00:00:00:00:00:00:00:00
#bytes #iterations t_avg[usec] tps average
Conflicting CPU frequency values detected: 3800.000000 != 2768.750000. CPU Frequency is not max.
Conflicting CPU frequency values detected: 3800.000000 != 1323.708000. CPU Frequency is not max.
2048 957119 3.13 159520.77