Add Basic RWKV Block to GPT-NeoX
Quentin-Anthony opened this issue · comments
We want to add RWKV to gpt-neox:
- Add basic RWKV block, without kernels, from https://github.com/BlinkDL/RWKV-LM to https://github.com/EleutherAI/gpt-neox/tree/main/megatron/model
- Add rwkv kernels
- Add config options for rwkv
- Add assertions to gpt-neox so that parallelism schemes and other architectures are disabled when rwkv is enabled in config