There are 3 repositories under attention-mechanisms topic.
Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM
An ultimately comprehensive paper list of Vision Transformer/Attention, including papers, codes, and related websites
Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch
Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch
Implementation of Toolformer, Language Models That Can Use Tools, by MetaAI
Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch
Awesome List of Attention Modules and Plug&Play Modules in Computer Vision
Implementation of Muse: Text-to-Image Generation via Masked Generative Transformers, in Pytorch
Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch
Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch
PyTorch Dual-Attention LSTM-Autoencoder For Multivariate Time Series
Implementation of MeshGPT, SOTA Mesh generation using Attention, in Pytorch
Implementation of MagViT2 Tokenizer in Pytorch
Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch
Official PyTorch Implementation for "Rotate to Attend: Convolutional Triplet Attention Module." [WACV 2021]
Implementation of RT1 (Robotic Transformer) in Pytorch
An implementation of local windowed attention for language modeling
Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group
Implementation of ChatGPT, but tailored towards primary care medicine, with the reward being able to collect patient histories in a thorough and efficient manner and come up with a reasonable differential diagnosis
Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind
🦖Pytorch implementation of popular Attention Mechanisms, Vision Transformers, MLP-Like models and CNNs.🔥🔥🔥
Neat (Neural Attention) Vision, is a visualization tool for the attention mechanisms of deep-learning models for Natural Language Processing (NLP) tasks. (framework-agnostic)
Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs
Implementation of the Equiformer, SE3/E3 equivariant attention network that reaches new SOTA, and adopted for use by EquiFold for protein folding
Sparse and structured neural attention mechanisms
Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch
Learning YOLOv3 from scratch 从零开始学习YOLOv3代码
Implementation of Block Recurrent Transformer - Pytorch
Implementation of Mega, the Single-head Attention with Multi-headed EMA architecture that currently holds SOTA on Long Range Arena
Implementation of fused cosine similarity attention in the same style as Flash Attention
Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT
Implementation of Recurrent Interface Network (RIN), for highly efficient generation of images and video without cascading networks, in Pytorch
Implementation of Flash Attention in Jax
Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind
This repository contain various types of attention mechanism like Bahdanau , Soft attention , Additive Attention , Hierarchical Attention etc in Pytorch, Tensorflow, Keras