tma15

tuning_playbook

A playbook for systematically maximizing the performance of deep learning models.

self_supervised

A Pytorch-Lightning implementation of self-supervised algorithms

olm-training

Repo for training MLMs, CLMs, or T5-type models on the OLM pretraining data, but it should work with any hugging face text dataset.

data-preparation

Code used for sourcing and cleaning the BigScience ROOTS corpus

olm-datasets

Pipeline for pulling and processing online language model pretraining data from the web

cramming

Cramming the training of a (BERT-type) language model into limited compute.

PaLM-rlhf-pytorch

Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

pytorch_scatter

PyTorch Extension Library of Optimized Scatter Operations

pyserini

Pyserini is a Python toolkit for reproducible information retrieval research with sparse and dense representations.

TRIME

[EMNLP 2022] Training Language Models with Memory Augmentation https://arxiv.org/abs/2205.12674

Laboro-DistilBERT-Japanese

EMAT

Efficient Memory-Augmented Transformers

FasterTransformer

Transformer related optimization, including BERT, GPT

LibVQ

KnnContrastiveForOOD

torchprofile

A general and accurate MACs / FLOPs profiler for PyTorch models

Transkimmer

Code for ACL2022 publication Transkimmer: Transformer Learns to Layer-wise Skim

intel-extension-for-transformers

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

dataloader

The merlin dataloader lets you rapidly load tabular data for training deep leaning models with TensorFlow, PyTorch or JAX

length-adaptive-transformer

Official Pytorch Implementation of Length-Adaptive Transformer (ACL 2021)

dse

SentAugment

SentAugment is a data augmentation technique for NLP that retrieves similar sentences from a large bank of sentences. It can be used in combination with self-training and knowledge-distillation, or for retrieving paraphrases.

voltaML

⚡VoltaML is a lightweight library to convert and run your ML/DL deep learning models in high performance inference runtimes like TensorRT, TorchScript, ONNX and TVM.

kernl

Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackable.

model-soups

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

transformers-bloom-inference

Fast Inference Solutions for BLOOM

DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

AITemplate

AITemplate is a Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.

pytorch-lightning

Pretrain, finetune and deploy AI models on multiple GPUs, TPUs with zero code changes.