TODO: Make DP + EP available
hyunwoongko opened this issue · comments
Describe a TODO feature
- The Expert Parallelism (MoE) feature we currently have cannot be used with data parallelism. we'll make it can be worked with Data Parallelism and reflects a new design that can further reduce the communication amount by 1.5 times.