aws / aws-ofi-nccl

This is a plugin which lets EC2 developers use libfabric as network provider while running NCCL applications.

aws/aws-ofi-nccl Issues

Propagate "Invalid address" to NCCL communicator
Closed 4 months ago
Segfault after/during finalize with OpenMPI
Updated 5 months ago2
NCCL internal error after aws-ofi-nccl upgrade to version 1.7.4
Updated 5 months ago5
GPU direct
Updated 6 months ago1
Topology Discovery Regression
Updated 6 months ago2
Add more examples with more recent cuda versions
Closed 7 months ago2
Support Red Hat Enterprise Linux 9+
Updated 8 months ago4
Unable to force FI_HMEM to be used and FI_OPT_CUDA_API_PERMITTED is not respected by config scripts
Closed 8 months ago4
Plugin fails if compiled against Libfabric 1.18 but run against Libfabric 1.17 or older.
Closed 9 months ago11
Support Amazon Linux 2023 (AL2023)
Updated 9 months ago
What are some AI/ML workloads users can utilize to test performance of the plugin?
Closed 9 months ago
Unable to find libcudart.so (1.7.1)
Updated 9 months ago5
Misleading comparison on unsigned integer
Closed 9 months ago
Running nccl-perf tests documentation is missing MPI instructions
Closed 9 months ago3
Support Ubuntu 22.04
Updated 10 months ago3
Support FI_CONTEXT2
Closed 10 months ago2
aws branch does not build on centos 7 with gcc 4.8.5
Closed a year ago2
Restore PSM3 transport for libfabric in .travis.yml
Closed a year ago4
potential reoccurrence of https://github.com/aws/aws-ofi-nccl/issues/69
Closed a year ago1
NCCL WARN NET/OFI Only EFA provider is supported
Closed a year ago2
PyTorch Distributed Training crashes with "Cannot allocate memory (-12)"
Closed a year ago14
Performance degradation for buffer size starting from ~46MB
Closed a year ago10
[Feature Request]Allow custom NCCL_TOPO_FILE location
Closed a year ago2
Mellanox and EFA in Docker Image
Closed a year ago6
aws-ofi-nccl makes unnecessary calls to ofi_iflush() when using the PSM3 transport.
Closed a year ago2
WARNING: unrecognized options: --with-nccl when attempting to install
Closed a year ago10
NCCL WARN NET/OFI Request completed with error. RC: 21. Error: unknown error
Closed a year ago2
Error running NCCL Tests via MPiJob: "prov_err Not a Directory"
Closed 2 years ago2
How does ofi_iflush() work?
Closed 2 years ago6
Question - difference between main vs aws branches
Closed 2 years ago1
Training performance on p3dn.24xlarge on Amazon Linux 2 is worse than on Ubuntu 20.04 (with and without EFA)
Closed 2 years ago2
Error (and crash) when using EFA from docker running on ubuntu AMI
Closed 2 years ago2
Crash with multirail providers.
Closed 2 years ago1
[Question] Is RDMA available on p3dn instances?
Closed 2 years ago7
DataLoader crash when using FI_EFA_USE_DEVICE_RDMA=1
Closed 2 years ago40
Missed ability to not use memory registering if provider does not request it
Closed 2 years ago6
Deadlock for rendezvous providers
Closed 2 years ago2
Handling GDR-capable providers not requested for memory registration
Closed 2 years ago3
Clarification needed for NCCL setting recommendation
Closed 3 years ago2
Is it safe to use OFI_NCCL_GDR_FLUSH_DISABLE?
Closed 3 years ago2
Couldn't allocate endpoint. RC: -22, ERROR: Invalid argument
Closed 3 years ago2
Unable to register memory (register_mr_buffers:465)
Closed 3 years ago5
EFA is not enabled in P4DN after upgrading NCCL v 2.10.3 and PyTorch v1.10 (master)
Closed 3 years ago3
bad performance on two p4 instances (nccl-v2.7.8, aws-ofi-nccl-v1.1.2)
Closed 3 years ago14
Can't create p4d instance with multiple EFA network interfaces.
Closed 4 years ago1
Throughput decrease
Closed 4 years ago9
Is it possible to use this over EFA?
Closed 4 years ago3
try horovod: create_nccl_ofi_component:496 NCCL WARN NET/OFI Couldn't enable endpoint. RC: -12, ERROR: Cannot allocate memory
Closed 4 years ago3
[7] create_nccl_ofi_component:459 NCCL WARN NET/OFI Couldn't open AV. RC: -22, ERROR: Invalid argument
Closed 4 years ago7
./configure does not recognize home directory "~/"
Closed 5 years ago2