PyTorch DTensor
xzyaoi opened this issue · comments
PyTorch now has native support for distributed tensor, might be a better way to do TP than megatron's MPU.
eth-easl / fmengine
Utilities for Training Very Large Models
Utilities for Training Very Large Models
xzyaoi opened this issue · comments
PyTorch now has native support for distributed tensor, might be a better way to do TP than megatron's MPU.