Heng-xiu

flores

The FLORES+ Machine Translation Benchmark

slidespeech_dl

LLMs-from-scratch

Implementing a ChatGPT-like LLM in PyTorch from scratch, step by step

Stirling-PDF

#1 Locally hosted web application that allows you to perform various operations on PDF files

mtdata

A tool that locates, downloads, and extracts machine translation corpora

Chinese_spelling_Correction

Chinese Grammar Error and Spelling Error Correction System - 中文文法錯誤及錯別字校正系統

AdvancedLiterateMachinery

A collection of original, innovative ideas and algorithms towards Advanced Literate Machinery. This project is maintained by the OCR Team in the Language Technology Lab, Tongyi Lab, Alibaba Group.

Large_dataset_translator

Translate large dataset to any language with google translation api and multithread processing, no key required !

Adaptive-MT-LLM-Fine-tuning

Fine-tuning Mistral LLM for Adaptive Machine Translation

Whisper

High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model

mt-metrics-eval

Tools for evaluating the performance of MT metrics on data from recent WMT metrics shared tasks.

bitextor

Bitextor generates translation memories from multilingual websites

VoiceStreamAI

Near-Realtime audio transcription using self-hosted Whisper and WebSocket in Python/JS

deep-learning-pytorch-huggingface

Transformers-Tutorials

This repository contains demos I made with the Transformers library by HuggingFace.

transformers_tasks

⭐️ NLP Algorithms with transformers lib. Supporting Text-Classification, Text-Generation, Information-Extraction, Text-Matching, RLHF, SFT etc.

minbpe

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

aya-annotations-ui

Web UI & Backend for Data Annotations in Aya

HanLP

中文分词 词性标注 命名实体识别 依存句法分析 成分句法分析 语义依存分析 语义角色标注 指代消解 风格转换 语义相似度 新词发现 关键词短语提取 自动摘要 文本分类聚类 拼音简繁转换 自然语言处理

instructor

structured outputs for llms

how-to-train-tokenizer

怎么训练一个LLM分词器

MoneyPrinter

Automate Creation of YouTube Shorts using MoviePy.

ALMA

State-of-the-art LLM-based translation models.

OneRingTranslator

Simple REST service to translate texts. Plugins. Automatic calculate BLEU/COMET metrics of translation quality.

self-translate

Do Multilingual Language Models Think Better in English?

List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words

List of Dirty, Naughty, Obscene, and Otherwise Bad Words

inference

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.

xorbits

Scalable Python DS & ML, in an API compatible & lightning fast way.

augmentoolkit

Convert Compute And Books Into Instruct-Tuning Datasets

llama.cpp

LLM inference in C/C++