Takuya Makino's starred repositories
tuning_playbook
A playbook for systematically maximizing the performance of deep learning models.
self_supervised
A Pytorch-Lightning implementation of self-supervised algorithms
olm-training
Repo for training MLMs, CLMs, or T5-type models on the OLM pretraining data, but it should work with any hugging face text dataset.
data-preparation
Code used for sourcing and cleaning the BigScience ROOTS corpus
olm-datasets
Pipeline for pulling and processing online language model pretraining data from the web
PaLM-rlhf-pytorch
Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM
pytorch_scatter
PyTorch Extension Library of Optimized Scatter Operations
FasterTransformer
Transformer related optimization, including BERT, GPT
torchprofile
A general and accurate MACs / FLOPs profiler for PyTorch models
Transkimmer
Code for ACL2022 publication Transkimmer: Transformer Learns to Layer-wise Skim
intel-extension-for-transformers
⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡
dataloader
The merlin dataloader lets you rapidly load tabular data for training deep leaning models with TensorFlow, PyTorch or JAX
length-adaptive-transformer
Official Pytorch Implementation of Length-Adaptive Transformer (ACL 2021)
SentAugment
SentAugment is a data augmentation technique for NLP that retrieves similar sentences from a large bank of sentences. It can be used in combination with self-training and knowledge-distillation, or for retrieving paraphrases.
model-soups
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
transformers-bloom-inference
Fast Inference Solutions for BLOOM
DeepSpeed-MII
MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
AITemplate
AITemplate is a Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.
pytorch-lightning
Pretrain, finetune and deploy AI models on multiple GPUs, TPUs with zero code changes.