MPI support
kailaix opened this issue · comments
add MPI adjoint features.
For scientific computing, a good algorithm should have small communication overhead. Therefore, blocking send and receive should not work too worse than nonblocking ones. The MPI adjoint feature focuses only on blocking send and receive.
The idea is to implement 6 functions that are implemented with custom operators.
mpi_send
mpi_recv
mpi_sum
mpi_bcast
mpi_init
mpi_finalize
The first four functions should implement gradient backprop
It's not that simple. The problem of blocking comms is that this implies a synchronization (unless buffering is used). So even if the comm is fast there may be substantial delays. But regardless it's much simpler so it's a good starting point.