kenhktsui

sgt

Sequence Graph Transform

Megatron-LM

Ongoing research training transformer models at scale

OpenDevin

🐚 OpenDevin: Code Less, Make More

Phi-3CookBook

This is a Phi-3 book for getting started with Phi-3. Phi-3, a family of open AI models developed by Microsoft. Phi-3 models are the most capable and cost-effective small language models (SLMs) available, outperforming models of the same size and next size up across a variety of language, reasoning, coding, and math benchmarks.

transformer-debugger

turndown

🛏 An HTML to Markdown converter written in JavaScript

firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

SeeAct

[ICML'24] SeeAct is a system for generalist web agents that autonomously carry out tasks on any given website, with a focus on large multimodal models (LMMs) such as GPT-4V(ision).

tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

cosmopedia

KoPA

[Paper][Preprint 2023] Making Large Language Models Perform Better in Knowledge Graph Completion

reader

Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/

LLMTest_NeedleInAHaystack

Doing simple retrieval from LLM models at various context lengths to measure accuracy

corenet

CoreNet: A library for training deep neural networks

aws-neuron-sdk

Powering AWS purpose-built machine learning chips. Blazing fast and cost effective, natively integrated into PyTorch and TensorFlow and integrated with your favorite AWS services

lighteval

LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.

public-apis

A collective list of free APIs

rho

Repo for Rho-1: Token-level Data Selection & Selective Pretraining of LLMs.

nanotron

Minimalistic large language model 3D-parallelism training

trafilatura

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

gradio

Build and share delightful machine learning apps, all in Python. 🌟 Star to support our work!

llama.cpp

LLM inference in C/C++

Olive

Olive is an easy-to-use hardware-aware model optimization tool that composes industry-leading techniques across model compression, optimization, and compilation.

WeightWatcher

The WeightWatcher tool for predicting the accuracy of Deep Neural Networks

ELR

Official Implementation of Early-Learning Regularization Prevents Memorization of Noisy Labels

pycma

Python implementation of CMA-ES

evojax

Open-Sora

Open-Sora: Democratizing Efficient Video Production for All

semantic-router

Superfast AI decision making and intelligent processing of multi-modal data.