looput

micrograd

A tiny scalar-valued autograd engine and a neural net library on top of it with PyTorch-like API

LookaheadDecoding

BIG-bench

Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models

crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

reader

Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/

edna

Note taking for developers and power users

LiveBench

LiveBench: A Challenging, Contamination-Free LLM Benchmark

FlagEmbedding

Retrieval and Retrieval-augmented LLMs

MiniCPM-V

MiniCPM-Llama3-V 2.5: A GPT-4V Level Multimodal LLM on Your Phone

llama-moe

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

LLaMA-Factory

A WebUI for Efficient Fine-Tuning of 100+ LLMs (ACL 2024)

llm.c

LLM training in simple, raw C/CUDA

dbrx

Code examples and resources for DBRX, a large language model developed by Databricks

LLM-Agent-Paper-List

The paper list of the 86-page paper "The Rise and Potential of Large Language Model Based Agents: A Survey" by Zhiheng Xi et al.

opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

Efficient-LLMs-Survey

[TMLR 2024] Efficient Large Language Models: A Survey

deita

Deita: Data-Efficient Instruction Tuning for Alignment [ICLR2024]

nlpaug

Data augmentation for NLP

data-juicer

A one-stop data processing system to make data higher-quality, juicier, and more digestible for (multimodal) LLMs! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷为大模型提供更高质量、更丰富、更易”消化“的数据！

Pai-Megatron-Patch

The official repo of Pai-Megatron-Patch for LLM & VLM large scale training developed by Alibaba Cloud.

SemDeDup

Code for "SemDeDup", a simple method for identifying and removing semantic duplicates from a dataset (data pairs which are semantically similar, but not exactly identical).

Conditional-Pretraining-of-Large-Language-Models

finngen-tools

Tools for training causal language models for Finnish

pdf2htmlEX

Convert PDF to HTML without losing text or format.

pretraining-with-human-feedback

Code accompanying the paper Pretraining Language Models with Human Preferences

HanLP

中文分词 词性标注 命名实体识别 依存句法分析 成分句法分析 语义依存分析 语义角色标注 指代消解 风格转换 语义相似度 新词发现 关键词短语提取 自动摘要 文本分类聚类 拼音简繁转换 自然语言处理

MNBVC

MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、wiki、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。

LLMSurvey

The official GitHub page for the survey paper "A Survey of Large Language Models".

AutoChain

AutoChain: Build lightweight, extensible, and testable LLM Agents

WebShop

[NeurIPS 2022] 🛒WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents