zye1996

jieba

结巴中文分词

LLaMA-Factory

Unify Efficient Fine-Tuning of 100+ LLMs

unilm

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

NeMo

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

surya

OCR, layout analysis, reading order, line detection in 90+ languages

ai

Build AI-powered applications with React, Svelte, Vue, and Solid

speechbrain

A PyTorch-based Speech Toolkit

magika

Detect file content types with deep learning

OLMo

Modeling, training, eval, and inference code for OLMo

VAR

[GPT beats diffusion🔥] [scaling laws in visual generation📈] Official impl. of "Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction". An *ultra-simple, user-friendly yet state-of-the-art* codebase for autoregressive image generation!

trt-llm-rag-windows

A developer reference project for creating Retrieval Augmented Generation (RAG) chatbots on Windows using TensorRT-LLM

vearch

Distributed vector search for AI-native applications

AutoPrompt

A framework for prompt tuning using Intent-based Prompt Calibration

CDial-GPT

A Large-scale Chinese Short-Text Conversation Dataset and Chinese pre-training dialog models

whisper-timestamped

Multilingual Automatic Speech Recognition with word-level timestamps and confidence

BitNet

Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch

conversational-datasets

Large datasets for conversational AI

GaLore

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Chinese-medical-dialogue-data

Chinese medical dialogue data 中文医疗对话数据集

PaddleOCR2Pytorch

PaddleOCR inference in PyTorch. Converted from [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)

CnSTD

CnSTD: 基于 PyTorch/MXNet 的 中文/英文 场景文字检测（Scene Text Detection）、数学公式检测（Mathematical Formula Detection, MFD）、篇章分析（Layout Analysis）的Python3 包

DocumentLayoutAnalysis

Document Layout Analysis resources repos for development with PdfPig.

Medical-Dialogue-System

MuTual

A Dataset for Multi-Turn Dialogue Reasoning

LESS

[ICML 2024] LESS: Selecting Influential Data for Targeted Instruction Tuning

clean-dialog

A framework for cleaning Chinese dialog data

gutenberg-dialog

Build a dialog dataset from online books in many languages

TikTalk

tvsub

TVsub: DCU-Tencent Chinese-English Dialogue Corpus

LPMDataset