MoE support

[edvardasast]

Hi, is it possible to add MoE model support?

[Yehor Smoliakov]

This project uses transformers. The documentation says this:

This implementation contains the dense models available on release. Let us know via a GitHub issue if you would like to see the MoE models as well.

So you can create an issue in theirs repository - https://github.com/huggingface/transformers