looput

quivr

Open-source RAG Framework for building GenAI Second Brains 🧠 Build productivity assistant (RAG) ⚡️🤖 Chat with your docs (PDF, CSV, ...) & apps using Langchain, GPT 3.5 / 4 turbo, Private, Anthropic, VertexAI, Ollama, LLMs, Groq that you can share with users ! Efficient retrieval augmented generation framework

HanLP

中文分词 词性标注 命名实体识别 依存句法分析 成分句法分析 语义依存分析 语义角色标注 指代消解 风格转换 语义相似度 新词发现 关键词短语提取 自动摘要 文本分类聚类 拼音简繁转换 自然语言处理

unilm

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

pdf2htmlEX

Convert PDF to HTML without losing text or format.

LLMSurvey

The official GitHub page for the survey paper "A Survey of Large Language Models".

CodeGeeX

CodeGeeX: An Open Multilingual Code Generation Model (KDD 2023)

imagen-pytorch

Implementation of Imagen, Google's Text-to-Image Neural Network, in Pytorch

CodeGeeX2

CodeGeeX2: A More Powerful Multilingual Code Generation Model

LLM-Agent-Paper-List

The paper list of the 86-page paper "The Rise and Potential of Large Language Model Based Agents: A Survey" by Zhiheng Xi et al.

nlpaug

Data augmentation for NLP

VisualGLM-6B

Chinese and English multimodal conversational language model | 多模态中英双语对话语言模型

opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

MNBVC

MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、wiki、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。

dbrx

Code examples and resources for DBRX, a large language model developed by Databricks

AgentBench

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)

data-juicer

A one-stop data processing system to make data higher-quality, juicier, and more digestible for (multimodal) LLMs! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷为大模型提供更高质量、更丰富、更易”消化“的数据！

AutoChain

AutoChain: Build lightweight, extensible, and testable LLM Agents

Chrome-GPT

An AutoGPT agent that controls Chrome on your desktop

bigscience

Central place for the engineering/scaling WG: documentation, SLURM scripts and logs, compute environment and data.

Efficient-LLMs-Survey

[TMLR 2024] Efficient Large Language Models: A Survey

Pai-Megatron-Patch

The official repo of Pai-Megatron-Patch for LLM & VLM large scale training developed by Alibaba Cloud.

deita

Deita: Data-Efficient Instruction Tuning for Alignment [ICLR2024]

WebShop

[NeurIPS 2022] 🛒WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

pretraining-with-human-feedback

Code accompanying the paper Pretraining Language Models with Human Preferences

ERNIE-Layout-Pytorch

An unofficial Pytorch implementation of ERNIE-Layout which is originally released through PaddleNLP.

SemDeDup

Code for "SemDeDup", a simple method for identifying and removing semantic duplicates from a dataset (data pairs which are semantically similar, but not exactly identical).

LATIN-Prompt

Conditional-Pretraining-of-Large-Language-Models

finngen-tools

Tools for training causal language models for Finnish