jklj077

candle

Minimalist ML framework for Rust

open-webui

User-friendly WebUI for LLMs (Formerly Ollama WebUI)

hugo-PaperMod

A fast, clean, responsive Hugo theme.

nanotron

Minimalistic large language model 3D-parallelism training

template

This is the repository for the distill web framework

locust

Write scalable load tests in plain Python 🚗💨

mediawiki-services-parsoid

This is a mirror from https://gerrit.wikimedia.org/g/mediawiki/services/parsoid/. See https://www.mediawiki.org/wiki/Developer_access for contributing.

crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

marlin

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

python-magic

A python wrapper for libmagic

pandoc

Universal markup converter

marker

Convert PDF to markdown quickly with high accuracy

dify

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.

HunyuanDiT

Hunyuan-DiT : A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding

data-juicer

A one-stop data processing system to make data higher-quality, juicier, and more digestible for LLMs! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷为大语言模型提供更高质量、更丰富、更易”消化“的数据！

MAP-NEO

internetarchive

A Python and Command-Line Interface to Archive.org

ia-download

Internet archive downloader

llama-cpp-python

Python bindings for llama.cpp

dash-cookbook

Receipts for creating AI Applications with APIs from DashScope (and friends)!

python-markdownify

Convert HTML to Markdown

the-stack-v2

Code for the curation of The Stack v2 and StarCoder2 training data

octopack

🐙 OctoPack: Instruction Tuning Code Large Language Models

unsloth

Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

llama3

The official Meta Llama 3 GitHub site

CodeQwen1.5

CodeQwen1.5 is the code version of Qwen, the large language model series developed by Qwen team, Alibaba Cloud.

web-content-extraction-benchmark

Web Content Extraction Benchmark

yt-dlp

A feature-rich command-line audio/video downloader