Multiple providers (e.g. Verbs + Shm) using the same memory buffer?
rdfriese opened this issue · comments
Is it possible for multiple providers to have access to the same underlying memory buffer?
For example I have set up a verbs instance and registered a memory buffer, I then set up a shm instance and register to the same memory buffer. Everything seems okay until trying to actually access the memory using verbs (shm operations seem to work fine).
If using different memory buffers, everything works as expected.
All other resources like eq, cq, cntrs, etc. are unique to each provider.
Thanks!
@rdfriese Yup! That is totally acceptable.
@aingerson Great news! I must be doing it wrong then hahah. Are the steps I'm doing above the right way to go about it or am I missing something?
Namely it seems like after registering the memory buffer to shm, if I try to perform a verbs operation like a put, followed by fi_cq_read() I get:
fi_cq_read failed: Error available (-259) and Error: Work Request Flushed Error
Make sure you are using MR registered with verbs on verbs calls. Underlying HW does not know anything about the MR registered with SHM.
Sorry, I think that is my question, can I register the same MR with both providers, currently it seems like if I try to register it with both, the provider that registered it last will be the only one that works.
(I have no issues using different MRs registered to only a single provider)
Ahhhh I think I see what I mean, I may be reusing the same MR handle, effectively overwriting which ever provider registered first. I'll double check and test and close the issue if that's the case!
I'm guessing you are using 1 MR variable and registering it with SHM and Verbs. It will behave as you stated, the last one works by overwriting the previous registration.
Yup! That was the issue, I had to refactor a bit to manage two MR variables, but I just tested with that in place and everything seems to be working as expected, thanks!