adamlin120

ai-workshop-code

Code I wrote for my AI & LLM workshops

llm.c

LLM training in simple, raw C/CUDA

tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

awesome-synthetic-datasets

awesome synthetic (text) datasets

cohere-toolkit

Cohere Toolkit is a collection of prebuilt components enabling users to quickly build and deploy RAG applications.

shisa-v2

Japanese / English Bilingual LLM

text-dedup

All-in-one text de-duplication

SemDeDup

Code for "SemDeDup", a simple method for identifying and removing semantic duplicates from a dataset (data pairs which are semantically similar, but not exactly identical).

chat-ui

Open source codebase powering the HuggingChat app

axolotl

Go ahead and axolotl questions

research-career-tools

BTX

Medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

composer

Supercharge Your Model Training

lighteval

LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.

echo

Sensei

Generate Synthetic Data Using OpenAI, MistralAI or AnthropicAI

ml-engineering

Machine Learning Engineering Open Book

unsloth

Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

tabbyAPI

An OAI compatible exllamav2 API that's both lightweight and fast

LLaMA-Factory

Unify Efficient Fine-Tuning of 100+ LLMs

dsir

DSIR large-scale data selection framework for language model training

datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

mergekit

Tools for merging pretrained large language models.

ocotillo

Performant and accurate speech recognition built on Pytorch

DL-Art-School

TorToiSe fine-tuning with DLAS

tortoise-tts

A multi-voice TTS system trained with an emphasis on quality

insanely-fast-whisper

Incredibly fast Whisper-large-v3

RedPajama-Data

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs