nezhazheng

datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

AISystem

AISystem 主要是指AI系统，包括AI芯片、AI编译器、AI推理和训练框架等AI全栈底层技术

InsTag

InsTag: A Tool for Data Analysis in LLM Supervised Fine-tuning

deita

Deita: Data-Efficient Instruction Tuning for Alignment [ICLR2024]

dataverse

The Universe of Data. All about data, data science, and data engineering

llm.c

LLM training in simple, raw C/CUDA

grok-1

Grok open release

Agent-Pro

The Code Repo for Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization

Cradle

The Cradle framework is a first attempt at General Computer Control (GCC). Cradle supports agents to ace any computer task by enabling strong reasoning abilities, self-improvment, and skill curation, in a standardized general environment with minimal requirements.

DouZero

[ICML 2021] DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning | 斗地主AI

rlcard

Reinforcement Learning / AI Bots in Card (Poker) Games - Blackjack, Leduc, Texas, DouDizhu, Mahjong, UNO.

Open-Sora

Open-Sora: Democratizing Efficient Video Production for All

transformer-debugger

trafilatura

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

boilerpipe

Work in progress transmit from Google Code

jusText

Heuristic based boilerplate removal tool

readability

A standalone version of the readability lib

Html2Article

Html网页正文提取

html-extractor

基于行块分布函数的通用网页正文抽取算法优化，Python实现

AlphaCodium

Official implementation for the paper: "Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering""

ungoliant

:spider: The pipeline for the OSCAR corpus

TransformerEngine

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.

pythia

The hub for EleutherAI's work on interpretability and learning dynamics

leptonai

A Pythonic framework to simplify AI service building

List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words

List of Dirty, Naughty, Obscene, and Otherwise Bad Words

ai-notes

notes for software engineers getting up to speed on new AI developments. Serves as datastore for https://latent.space writing, and product brainstorming, but has cleaned up canonical references under the /Resources folder.

amber-data-prep

Data preparation code for Amber 7B LLM

ChineseWebText

SDV

Synthetic data generation for tabular data

MNBVC

MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、wiki、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。