Is it possible to use this over EFA?

Question

Is it possible to use this over EFA?

VoVAllen opened this issue 4 years ago · comments

Jinjing Zhou commented 4 years ago

Hi,

I'm wondering whether it's possible for this to utilize elastic fabric adapter.

Thanks

Rashika Kheria · Answer 1 · Thu Oct 29 2020 01:17:53 GMT+0800 (China Standard Time)

Yes, it is possible and the README mentions that the plugin has been tested with EFA provider. Here is a getting started doc on EFA over P3dn which uses the plugin: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start-nccl-base.html and https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-efa-using.html

Jinjing Zhou · Answer 2 · Thu Oct 29 2020 01:28:39 GMT+0800 (China Standard Time)

Hi,

Is there any code I can refer to that uses libfabric RDM endpoint primitive? (i.e. fi_send) Also I found there's code in libfabric repo about FI_HMEM. Does it mean I can receive data using efa to a registered cuda memory directly with something like fi_mr and fi_msg?

Rashika Kheria · Answer 3 · Thu Oct 29 2020 01:34:13 GMT+0800 (China Standard Time)

This plugin repository uses fi_tsend for libfabric RDM endpoints. Yes, with FI_HMEM, you can use cuda buffers directly. This plugin also has code for that.