DavidHerel

collapse-lm-iclr

"Collapse of Self-trained Language Models" codebase for ICLR 2024

fundus

A very simple news crawler with a funny name

minbpe

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

API

Documentation and Samples for the Official HN API

GNews

A Happy and lightweight Python Package that Provides an API to search for articles on Google News and returns a JSON response.

litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

llama.cpp

LLM inference in C/C++

llama2.c

Inference Llama 2 in one file of pure C

gigaGPT

a small code base for training large models

preprocessing_reddit

arctic_shift

Making Reddit data accessible to researchers, moderators and everyone else. Interact with the data through large dumps, an API or web interface.

nanoRWKV

RWKV in nanoGPT style

sota_lm

Repository containing state-of-the-art ensemble for several language modelling benchmarks

lit-llama

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

TinyLlama

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

TAADpapers

Must-read Papers on Textual Adversarial Attack and Defense

EvolvingModularRobots_Unity

Software for Evolving Modular Robots in Unity

nanoGPT

The simplest, fastest repository for training/finetuning medium-sized GPTs.

llama

Inference code for Llama models

cc_net

Tools to download and cleanup Common Crawl data

awd-lstm-lm

LSTM and QRNN Language Model Toolkit for PyTorch

why-I-hate-wow-private-servers

Reasons why most of WoW private servers sucks

davidherel.github.io

Source code for my website: https://davidherel.com

semantics-preserving-encoder

Python library providing a simple, fully supervised sentence embedding technique for textual adversarial attacks.

TabPFN

Official implementation of the TabPFN paper (https://arxiv.org/abs/2207.01848) and the tabpfn package.

From-Simple-Transformations-to-Highly-Efficient-Jobs

Spark Training

tabular-benchmark

LaMDA-rlhf-pytorch

Open-source pre-training implementation of Google's LaMDA in PyTorch. Adding RLHF similar to ChatGPT.

deep_learning_curriculum

Language model alignment-focused deep learning curriculum

RWKV-LM

RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.