ccx06

Cherry_LLM

[NAACL'24] Self-data filtering of LLM instruction-tuning data using a novel perplexity-based difficulty score, without using any other models

datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

cc_net

Tools to download and cleanup Common Crawl data

RedPajama-Data

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

OpenWebMath

MathPile

Generative AI for Math: MathPile

awesome-LLM-resourses

🧑‍🚀 全世界最好的LLM资料总结 | Summary of the world's best LLM resources.

Awesome-Tabular-LLMs

We collect papers about "large language models (LLM) for table-related tasks", e.g., using LLM for Table QA task. “表格+LLM”相关论文整理

agentscope

Start building LLM-empowered multi-agent applications in an easier way.

Chinese-Tiny-LLM

doremi

Pytorch implementation of DoReMi, a method for optimizing the data mixture weights in language modeling datasets

awesome-pretrained-chinese-nlp-models

Awesome Pretrained Chinese NLP Models，高质量中文预训练模型&大模型&多模态模型&大语言模型集合

MNBVC

MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、wiki、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。

zero_nlp

中文nlp解决方案(大模型、数据、模型、训练、推理)

GPT2-Chinese

Chinese version of GPT2 training code, using BERT tokenizer.

Open-GPT

GPT is a belief. This project provides a code library for efficiently training Chinese GPT. This project uses the nanoGPT framework and the novel optimizer algorithm sophia.

nlp_chinese_corpus

大规模中文自然语言处理语料 Large Scale Chinese Corpus for NLP

gpt-neo

An implementation of model parallel GPT-2 and GPT-3-style models using the mesh-tensorflow library.

gpt-2

Code for the paper "Language Models are Unsupervised Multitask Learners"

RAG-Survey

Telechat

HPT

This repository implements a prompt tuning model for hierarchical text classification. This work has been accepted as the long paper "HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text Classification" in EMNLP 2022.

FlagEmbedding

Retrieval and Retrieval-augmented LLMs

Chinese-Keyphrase-Extraction

无监督中文关键词抽取（Keyphrase Extraction），基于统计，基于图【LDA与PageRank（TextRank， TPR， Salience Rank， Single TPR等）】，基于嵌入【SIFRank等】，开箱即用！

Baichuan2

A series of large language models developed by Baichuan Intelligent Technology

multi-label-classification

基于tf.keras的多标签多分类模型

LLaMA-Factory

Efficiently Fine-Tune 100+ LLMs in WebUI (ACL 2024)

albert_pytorch

A Lite Bert For Self-Supervised Learning Language Representations

Pytorch-NLU

Pytorch-NLU，一个中文文本分类、序列标注工具包，支持中文长文本、短文本的多类、多标签分类任务，支持中文命名实体识别、词性标注、分词、抽取式文本摘要等序列标注任务。 Ptorch NLU, a Chinese text classification and sequence annotation toolkit, supports multi class and multi label classification tasks of Chinese long text and short text, and supports sequence annotation tasks such as Chinese named entity recognition, part of spee

ChatRWKV

ChatRWKV is like ChatGPT but powered by RWKV (100% RNN) language model, and open source.