prov/verbs: memory consistency with MOFED 5.8-3.0.7.0

[thomasgillis]

Hi all,

I am reaching out as I have a issues with verbs following an update of MOFED.
I am not sure how to approach this, any advise/idea is welcome

Describe the bug
I have a client on top of ofi using verbs;ofi_rxm, which works very well on MLNX_OFED_LINUX-5.8-2.0.3.0 (OFED-5.8-2.0.3).
On a similar system, they have bumped the version of ofed: MLNX_OFED_LINUX-5.8-3.0.7.0 (OFED-5.8-3.0.7): and since then, I have issues. When using fi_write with FI_REMOTE_CQ_DATA, I observe that even after the entry in the target cq has been read, the memory is not in the expected state.

Would you have any idea on how to move forward with this? (I don't think that versioning down is an option for the moment)

[chien-intel]

Please reach out to Nvidia support. The problem sounds severe so most likely it is already fixed.

[thomasgillis]

I could modify fabtest and get the following reproducer:

Waiting for CQ data from client
data[0] = 1
data[1] = 2
data[2] = 3
data[3] = 4
Posting write with CQ data: 0x89abcdef
sending 1
sending 2
len of RMA = 8, offset = 0x240b040, data = 0
sending 3
sending 4
len of RMA = 8, offset = 0x240b048, data = 1
Done
received 1 cq-data: 1/2: len = 8, data=0, buf=(nil)
received 1 cq-data: 2/2: len = 8, data=1, buf=(nil)
fi_cq_data_entry.len verify: success
error, value of 0x240b040 + 2 =0x240b048 data[2] should be 3 instead of 8064

Here is the patch to it: thomasgillis@c0fad63

@shefty could you take a look at the modified test and confirm that it is a valid test for the cq_data? I plan to submit it with the issue to NVidia.

[shefty]

@thomasgillis, I recommend creating a reproducer that calls libibverbs directly. It's difficult to read the fabtests code and determine if it's correct because of the internal fabtests abstractions.