hongsunjang

finetune-gpt2xl

Guide: Finetune GPT2-XL (1.5 Billion Parameters) and finetune GPT-NEO (2.7 B) on a single GPU with Huggingface Transformers using DeepSpeed

gpt-2

Code for the paper "Language Models are Unsupervised Multitask Learners"

metaseq

Repo for external large-scale work

Pretrained-Language-Model

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

Ok-Topk

Ok-Topk is a scheme for distributed training with sparse gradients. Ok-Topk integrates a novel sparse allreduce algorithm (less than 6k communication volume which is asymptotically optimal) with the decentralized parallel Stochastic Gradient Descent (SGD) optimizer, and its convergence is proved theoretically and empirically.

computer-science

:mortar_board: Path to a free self-taught education in Computer Science!

SimpleSSD-FullSystem

Open-Source Licensed Educational SSD Simulator for High-Performance Storage and Full-System Evaluations

torchrec

Pytorch domain library for recommendation systems

Vitis-AI-Tutorials

SQuant

SQuant [ICLR22]

lora

Using Low-rank adaptation to quickly fine-tune diffusion models.

LoRA

Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"

open-llms

📋 A list of open LLMs available for commercial use.

improved-diffusion

Release for Improved Denoising Diffusion Probabilistic Models

ChatGPT-as-a-server

Using ChatGPT as a real backend

lm-evaluation-harness

A framework for few-shot evaluation of language models.

KDiskMark

A simple open-source disk benchmark tool for Linux distros

CrystalDiskInfo

CrystalDiskInfo

ssd-benchmark-rs

Super Simple Disk Benchmark - benchmarks the writing performance of your disk

sc22-ae

Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Cosmos-OpenSSD

MQSim

MQSim is a fast and accurate simulator modeling the performance of modern multi-queue (MQ) SSDs as well as traditional SATA based SSDs. MQSim faithfully models new high-bandwidth protocol implementations, steady-state SSD conditions, and the full end-to-end latency of requests in modern SSDs. It is described in detail in the FAST 2018 paper by Arash Tavakkol et al., "MQSim: A Framework for Enabling Realistic Studies of Modern Multi-Queue SSD Devices" (https://people.inf.ethz.ch/omutlu/pub/MQSim-SSD-simulation-framework_fast18.pdf)

Vitis-Tutorials

Vitis In-Depth Tutorials

GenGNN

GCoD

[HPCA 2022] GCoD: Graph Convolutional Network Acceleration via Dedicated Algorithm and Accelerator Co-Design

Literatures-on-GNN-Acceleration

A reading list for deep graph learning acceleration.

GenStore

GenStore is the first in-storage processing system designed for genome sequence analysis that greatly reduces both data movement and computational overheads of genome sequence analysis by exploiting low-cost and accurate in-storage filters. Described in the ASPLOS 2022 paper by Mansouri Ghiasi et al. at https://people.inf.ethz.ch/omutlu/pub/GenStore_asplos22-arxiv.pdf

llama

Inference code for Llama models