wavy-jung

awesome-mixture-of-experts

A collection of AWESOME things about mixture-of-experts

JetMoE

Reaching LLaMA2 Performance with 0.1M Dollars

GLMKD

Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method ; GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model

repo-level-codegen-papers

Repo-Level Code generation papers

tree-sitter

An incremental parsing system for programming tools

devdocs

API Documentation Browser

axlearn

An Extensible Deep Learning Library

Yuan2.0-M32

Mixture-of-Experts (MoE) Language Model

learning_ray

Notebooks for the O'Reilly book "Learning Ray"

llama-agentic-system

Agentic components of the Llama Stack APIs

open_lm

A repository for research on medium sized language models.

faker

Faker is a Python package that generates fake data for you.

LLM101n

LLM101n: Let's build a Storyteller

NeMo-Aligner

Scalable toolkit for efficient model alignment

datacomp

DataComp: In search of the next generation of multimodal datasets

dclm

DataComp for Language Models

NeMo-Framework-Launcher

Provides end-to-end model development pipelines for LLMs and Multimodal models that can be launched on-prem or cloud-native.

Skywork-MoE

Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models

text-clustering

Easily embed, cluster and semantically label text datasets

maxtext

A simple, performant and scalable Jax LLM!

unsloth

Finetune Llama 3.1, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

SimPO

SimPO: Simple Preference Optimization with a Reference-Free Reward

dbrx

Code examples and resources for DBRX, a large language model developed by Databricks

mistral-inference

Official inference library for Mistral models

llama-moe

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

llm-swarm

Manage scalable open LLM inference endpoints in Slurm clusters

cosmopedia

Bend

A massively parallel, high-level programming language

llama-recipes

Scripts for fine-tuning Meta Llama3 with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supporting a number of candid inference solutions such as HF TGI, VLLM for local or cloud deployment. Demo apps to showcase Meta Llama3 for WhatsApp & Messenger.

distilabel

Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.