Phil Wang's repositories
reformer-pytorch
Reformer, the efficient Transformer, in Pytorch
nuwa-pytorch
Implementation of NÜWA, state of the art attention network for text to video synthesis, in Pytorch
slot-attention
Implementation of Slot Attention from GoogleAI
segformer-pytorch
Implementation of Segformer, Attention + MLP neural network for segmentation, in Pytorch
se3-transformer-pytorch
Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repository is geared towards integration with eventual Alphafold2 replication.
electra-pytorch
A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch
flash-cosine-sim-attention
Implementation of fused cosine similarity attention in the same style as Flash Attention
res-mlp-pytorch
Implementation of ResMLP, an all MLP solution to image classification, in Pytorch
graph-transformer-pytorch
Implementation of Graph Transformer in Pytorch, for potential use in replicating Alphafold2
chroma-pytorch
Implementation of Chroma, generative models of protein using DDPM and GNNs, in Pytorch
gated-state-spaces-pytorch
Implementation of Gated State Spaces, from the paper "Long Range Language Modeling via Gated State Spaces", in Pytorch
perceiver-ar-pytorch
Implementation of Perceiver AR, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch
rvq-vae-gpt
My attempts at applying Soundstream design on learned tokenization of text and then applying hierarchical attention to text generation
memory-compressed-attention
Implementation of Memory-Compressed Attention, from the paper "Generating Wikipedia By Summarizing Long Sequences"
n-grammer-pytorch
Implementation of N-Grammer, augmenting Transformers with latent n-grams, in Pytorch
adjacent-attention-network
Graph neural network message passing reframed as a Transformer with local attention
equiformer-diffusion
Implementation of Denoising Diffusion for protein design, but using the new Equiformer (successor to SE3 Transformers) with some additional improvements
isab-pytorch
An implementation of (Induced) Set Attention Block, from the Set Transformers paper
flash-genomics-model
My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hierarchical methods)
einops-exts
Implementation of some personal helper functions for Einops, my most favorite tensor manipulation library ❤️
autoregressive-linear-attention-cuda
CUDA implementation of autoregressive linear attention, with all the latest research findings
memory-editable-transformer
My explorations into editing the knowledge and memories of an attention network
bitsandbytes
8-bit CUDA functions for PyTorch
nucleotide-transformer
🧬 Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics
RFdiffusion
Code for running RFdiffusion