eugenesiow / LLM-Insights

Insights gleaned from work on each stage of Large Language Model (LLM) development.

LLM Insights

1. Infrastructure/Platform
   • Hardware
   • Alternative Accelerators
2. Data Engineering
3. Pre-Training Stage
   • Tokenization
   • Model Architecture
   • Multi-Node
   • Training
   • Base Model Evaluation
4. Fine-Tuning Stage
   • Adapters
5. Alignment Stage
   • Reducing Hallucination
6. Inference Stage
   • Estimating Model Size
   • Speed
   • Throughput
7. Applications

Acknowledgements

I'm grateful to my employers for trusting me to lead the team that built the GPU supercompute platform/infrastructure and to co-lead the team doing LLM pre-training. This allowed me to work on large on-premise GPU compute clusters with A100s and then H100s, which is certainly a privilege. Hopefully sharing some of these notes and insights helps the community.