hpcaitech / Titans

A collection of models built with ColossalAI

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[FEATURE] Settings related to common layers.

BoxiangW opened this issue · comments

Describe the feature

Compared to vanilla PyTorch, Titans right now includes many unnecessary codes for example multiple files for MLP.
We could provide common MLP, Attention ... modules for users to avoid redundancy in both our repo and users' codes.
I have attached an simple example of PyTorch code as well.
image