lucidrains / mixture-of-experts

A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models

lucidrains/mixture-of-experts Issues

Load balancing loss?
Closed 9 months ago2
Would you elaborate more on the enhancement?
Updated a year ago
Regarding experts.w1 and experts.w2 gradients
Updated a year ago1
convolution operation
Updated 2 years ago
implicit inplace operation '*=' cause an error when deriving the back gradient in pytorch
Closed 3 years ago1
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
Updated 3 years ago
question about why need an exclusive cumsum in gating method?
Closed 3 years ago
Error reported under FP16 training
Closed 4 years ago1
RuntimeError: expected backend CPU and dtype Float but got backend CPU and dtype Long
Closed 4 years ago1
Segmentation Fault?
Closed 4 years ago1