Awesome-LLM

🔥 Large Language Models(LLM) have taken the ~~NLP community~~ the Whole World by storm. Here is a curated list of papers about large language models, especially relating to ChatGPT. It also contains frameworks for LLM training, tools to deploy LLM, courses and tutorials about LLM and all publicly available LLM checkpoints and APIs:

Awesome-LLM

Milestone Papers

Date	keywords	Institute	Paper	Publication
2017-06	Transformers	Google	Attention Is All You Need	NeurIPS
2018-06	GPT 1.0	OpenAI	Improving Language Understanding by Generative Pre-Training
2018-10	BERT	Google	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	NAACL
2019-02	GPT 2.0	OpenAI	Language Models are Unsupervised Multitask Learners
2019-09	Megatron-LM	NVIDIA	Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
2019-10	T5	Google	Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	JMLR
2020-01	Scaling Law	OpenAI	Scaling Laws for Neural Language Models
2020-05	GPT 3.0	OpenAI	Language models are few-shot learners	NeurIPS
2020-12	LM-BFF	Princeton	Making Pre-trained Language Models Better Few-shot Learners	ACL
2021-01	Switch Transformers	Google	Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	JMLR
2021-08	Codex	OpenAI	Evaluating Large Language Models Trained on Code
2021-08	Foundation Models	Stanford	On the Opportunities and Risks of Foundation Models
2021-09	FLAN	Google	Finetuned Language Models are Zero-Shot Learners	ICLR
2021-12	GLaM	Google	GLaM: Efficient Scaling of Language Models with Mixture-of-Experts	ICML
2021-12	WebGPT	OpenAI	WebGPT: Improving the Factual Accuracy of Language Models through Web Browsing
2021-12	Retro	DeepMind	Improving language models by retrieving from trillions of tokens	ICML
2021-12	Gopher	DeepMind	Scaling Language Models: Methods, Analysis & Insights from Training Gopher
2022-01	COT	Google	Chain-of-Thought Prompting Elicits Reasoning in Large Language Models	NeurIPS
2022-01	LaMDA	Google	LaMDA: Language Models for Dialog Applications
2022-01	Megatron-Turing NLG	Microsoft&NVIDIA	Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
2022-03	InstructGPT	OpenAI	Training language models to follow instructions with human feedback
2022-04	PaLM	Google	PaLM: Scaling Language Modeling with Pathways
2022-04	Chinchilla	DeepMind	An empirical analysis of compute-optimal large language model training	NeurIPS
2022-05	OPT	Meta	OPT: Open Pre-trained Transformer Language Models
2022-06	Emergent Abilities	Google	Emergent Abilities of Large Language Models	TMLR
2022-06	BIG-bench	Google	Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
2022-06	METALM	Microsoft	Language Models are General-Purpose Interfaces
2022-09	Sparrow	DeepMind	Improving alignment of dialogue agents via targeted human judgements
2022-10	Flan-T5	Google	Scaling Instruction-Finetuned Language Models
2022-10	GLM-130B	Tsinghua	GLM-130B: An Open Bilingual Pre-trained Model	ICLR
2022-11	HELM	Stanford	Holistic Evaluation of Language Models
2022-11	BLOOM	BigScience	BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
2022-11	Galactica	Meta	Galactica: A Large Language Model for Science
2023-01	Flan 2022 Collection	Google	The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

LLM Leaderboard

Model	Size	Training Corpus	Architecture	API/CKPT	Date	Origin
ChatGPT	-	-	Decoder	api	2022-11	Blog
Switch Transformer	1.6T	multilingual	Decoder(MOE)	-	2021-01	Paper
GLaM	1.2T	English	Decoder(MOE)	-	2021-12	Paper
PaLM	540B	multilingual,code	Decoder	-	2022-04	Paper
MT-NLG	530B	English	Decoder	-	2022-01	Paper
J1-Jumbo	178B	English	Decoder	api	2021-08	Paper
OPT	175B	primarily English	Decoder	api \| ckpt	2022-05	Paper
BLOOM	176B	multilingual,code	Decoder	api \| ckpt	2022-11	Paper
GPT 3.0	175B	primarily English	Decoder	api	2020-05	Paper
LaMDA	137B	Dialogue	Decoder	-	2022-01	Paper
GLM	130B	English,Chinese	Decoder	ckpt	2022-10	Paper
Galactica	120B	English,code,Latex,DNA,etc.	Decoder	ckpt	2022-11	Paper
YaLM	100B	English,Russian	Decoder	ckpt	2022-06	Blog
GPT-NeoX	20B	English	Decoder	ckpt	2022-04	Paper
UL2	20B	English	agnostic	ckpt	2022-05	Paper
Flan-T5	11B	English	Encoder-Decoder	ckpt	2022-10	Paper
鹏程.盘古α	13B	Chinese	Decoder	ckpt	2021-04	Paper
T0	11B	English	Encoder-Decoder	ckpt	2021-10	Paper
T5	11B	English	Encoder-Decoder	ckpt	2019-10	Paper
CPM-Bee	10B	English,Chinese	Decoder	api	2022-10	Paper
rwkv-4	7B	English	RWKV	ckpt	2022-09	Github
GPT-J	6B	English	Decoder	ckpt	2022-09	Github
GPT-Neo	2.7B	English	Decoder	ckpt	2021-03	Github
GPT-Neo	1.3B	English	Decoder	ckpt	2021-03	Github

ChatGPT Evaluation

Is ChatGPT a General-Purpose Natural Language Processing Task Solver? Link
Is ChatGPT A Good Translator? A Preliminary Study Link

LLM Training Frameworks

Serving OPT-175B, BLOOM-176B and CodeGen-16B using Alpa

Alpa is a system for training and serving large-scale neural networks. Scaling neural networks to hundreds of billions of parameters has enabled dramatic breakthroughs such as GPT-3, but training and serving these large-scale neural networks require complicated distributed system techniques. Alpa aims to automate large-scale distributed training and serving with just a few lines of code.

Megatron-LM GPT2 tutorial

DeepSpeed is an easy-to-use deep learning optimization software suite that enables unprecedented scale and speed for DL Training and Inference. Visit us at deepspeed.ai or our Github repo.

pretrain_gpt3_175B.sh

Megatron-LM could be visited here. Megatron (1, 2, and 3) is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA. This repository is for ongoing research on training large transformer language models at scale. We developed efficient, model-parallel (tensor, sequence, and pipeline), and multi-node pre-training of transformer based models such as GPT, BERT, and T5 using mixed precision.

Open source solution replicates ChatGPT training process! Ready to go with only 1.6GB GPU memory and gives you 7.73 times faster training!

Colossal-AI provides a collection of parallel components for you. We aim to support you to write your distributed deep learning models just like how you write your model on your laptop. We provide user-friendly tools to kickstart distributed training and inference in a few lines.

BMTrain is an efficient large model training toolkit that can be used to train large models with tens of billions of parameters. It can train models in a distributed manner while keeping the code as simple as stand-alone training.

Mesh TensorFlow (mtf) is a language for distributed deep learning, capable of specifying a broad class of distributed tensor computations. The purpose of Mesh TensorFlow is to formalize and implement distribution strategies for your computation graph over your hardware/processors. For example: "Split the batch over rows of processors and split the units in the hidden layer across columns of processors." Mesh TensorFlow is implemented as a layer over TensorFlow.

This tutorial discusses parallelism via jax.Array.

Tools for using LLM

🦜️🔗 LangChain

Large language models (LLMs) are emerging as a transformative technology, enabling developers to build applications that they previously could not. But using these LLMs in isolation is often not enough to create a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge. This library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include ❓ Question Answering over specific documents, 💬 Chatbots and 🤖 Agents.

👋 wechat-chatgpt

Use ChatGPT On Wechat via wechaty

Tutorials about LLM

[ICML 2022] Welcome to the "Big Model" Era: Techniques and Systems to Train and Serve Bigger Models Link
[NeurIPS 2022] Foundational Robustness of Foundation Models Link
[Andrej Karpathy] Let's build GPT: from scratch, in code, spelled out. Video|Code
[DAIR.AI] Prompt Engineering Guide Link
[邱锡鹏] 大型语言模型的能力分析与应用 Slides | Video
[Philipp Schmid] Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers Link

Courses about LLM

[OpenBMB] 大模型公开课主页
[Stanford] CS224N-Lecture 11: Prompting, Instruction Finetuning, and RLHF Slides
[Stanford] CS324-Large Language Models Homepage
[Stanford] CS25-Transformers United V2 Homepage
[Stanford Webinar] GPT-3 & Beyond Video
[李沐] InstructGPT论文精读 Bilibili Youtube
[李沐] HELM全面语言模型评测 Bilibili
[李沐] GPT，GPT-2，GPT-3 论文精读 Bilibili Youtube
[Aston Zhang] Chain of Thought论文 Bilibili Youtube

Opinions about LLM

Towards ChatGPT and Beyond [2023-02-20][知乎][欧泽彬]
追赶ChatGPT的难点与平替 [2023-02-19][李rumor]
对话旷视研究院张祥雨｜ChatGPT的科研价值可能更大 [2023-02-16][知乎][旷视科技]
关于ChatGPT八个技术问题的猜想 [2023-02-15][知乎][张家俊]
ChatGPT发展历程、原理、技术架构详解和产业未来 [2023-02-15][知乎][陈巍谈芯]
What Is ChatGPT Doing … and Why Does It Work? [2023-02-14][Stephen Wolfram]
对ChatGPT的二十点看法 [2023-02-13][知乎][熊德意]
Why did all of the public reproduction of GPT-3 fail? [2023-02-12][Jingfeng Yang]
ChatGPT-所见、所闻、所感 [2023-02-11][知乎][刘聪NLP]
The Next Generation Of Large Language Models [2023-02-07][Forbes]
Large Language Model Training in 2023 [2023-02-03][Cem Dilmegani]
What Are Large Language Models Used For? [2023-01-26][NVIDIA]
通向AGI之路：大型语言模型(LLM)技术精要 [2023-01-18][知乎][张俊林]
Major LLMs + Data Availability [2023-01-06][Shayne Longpre]
How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources [2022-12-11][Yao Fu]
ChatGPT (可能)是怎麼煉成的 - GPT 社會化的過程 [2022-12-07][Hung-yi Lee]
Large Language Models: A New Moore's Law [2021-10-26][Huggingface]

Other Awesome Lists

Awesome ChatGPT Prompts - A collection of prompt examples to be used with the ChatGPT model.
awesome-chatgpt-prompts-zh - A Chinese collection of prompt examples to be used with the ChatGPT model.
Awesome ChatGPT - Curated list of resources for ChatGPT and GPT-3 from OpenAI.
Chain-of-Thoughts Papers - A trend starts from "Chain of Thought Prompting Elicits Reasoning in Large Language Models.

Other Useful Resources

ShareGPT - Share your wildest ChatGPT conversations with one click.

Contributing

This is an active repository and your contributions are always welcome!

I will keep some pull requests open if I'm not sure if they are awesome for LLM, you could vote for them by adding 👍 to them.

If you have any question about this opinionated list, do not hesitate to contact me chengxin1998@stu.pku.edu.cn.

crazyofapple / Awesome-LLM