A collection of resources and papers on Vector Quantized Variational Autoencoder (VQ-VAE) and its application
Understanding VQ-VAE (DALL-E Explained Pt. 1)
Charlie Snell
[Website]
9 Feb 2021
How is it so good ? (DALL-E Explained Pt. 2)
Charlie Snell
[Website]
7 Apr 2021
VQ-VAE: A brief introduction
Jianlin Su
[Website]
24 June 2019
Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization
Mengqi Huang, Zhendong Mao, Zhuowei Chen, Yongdong Zhang
CVPR 2023 Highlight. [Paper]
Not All Image Regions Matter: Masked Vector Quantization for Autoregressive Image Generation
Mengqi Huang, Zhendong Mao, Quan Wang, Yongdong Zhang
CVPR 2023. [Paper]
MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis
Tianhong Li, Huiwen Chang, Shlok Kumar Mishra, Han Zhang, Dina Katabi, Dilip Krishnan
CVPR 2023. [Paper]
Regularized Vector Quantization for Tokenized Image Synthesis
Jiahui Zhang, Fangneng Zhan, Christian Theobalt, Shijian Lu
CVPR 2023. [Paper]
Peco: Perceptual Codebook for Bert Pre-training of Vision Transformers
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu, Baining Guo
AAAI 2023. [Paper]
Designing a Better Asymmetric VQGAN for StableDiffusion
Zixin Zhu, Xuelu Feng, Dongdong Chen, Jianmin Bao, Le Wang, Yinpeng Chen, Lu Yuan, Gang Hua
arXiv 2023. [Paper]
LLM Itself Can Read and Generate CXR Images
Suhyeon Lee, Won Jun Kim, Jong Chul Ye
arXiv 2023. [Paper]
SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David A. Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, Kevin Murphy, Alexander G. Hauptmann, Lu Jiang
arXiv 2023. [Paper]
Scaling Laws for Generative Mixed-Modal Language Models
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, Luke Zettlemoyer
arXiv 2023. [Paper]
MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation
Chuanxia Zheng, Long Tung Vuong, Jianfei Cai, Dinh Phung
NeurIPS 2022. [Paper]
Vector-quantized Image Modeling with Improved VQGAN
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, Yonghui Wu
ICLR 2022. [Paper]
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, Furu Wei
arXiv 2022. [Paper]
BEiT: BERT Pre-Training of Image Transformers
Hangbo Bao, Li Dong, Songhao Piao, Furu Wei
ICLR 2022. [Paper]
High-Resolution Image Synthesis with Latent Diffusion Models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer
CVPR 2022 [Paper]
Vector Quantized Diffusion Model for Text-to-Image Synthesis
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, Baining Guo
CVPR 2022 [Paper]
MaskGIT: Masked Generative Image Transformer
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman
CVPR 2022. [Paper]
Autoregressive Image Generation using Residual Quantization
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, Wook-Shin Han
CVPR 2022. [Paper]
VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder
Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, Ming-Ming Cheng
ECCV 2022. [Paper]
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, Aniruddha Kembhavi
arXiv 2022. [Paper]
Improved Vector Quantized Diffusion Models
Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, Fang Wen
arXiv 2022. [Paper]
CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers
Ming Ding, Wendi Zheng, Wenyi Hong, Jie Tang
arXiv 2022. [Paper]
DiVAE: Photorealistic Images Synthesis with Denoising Diffusion Decoder
Jie Shi, Chenfei Wu, Jian Liang, Xiang Liu, Nan Duan
arXiv 2022. [Paper]
CogView: Mastering Text-to-Image Generation via Transformers
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, Jie Tang
NeurIPS 2021. [Paper]
Zero-Shot Text-to-Image Generation
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, Ilya Sutskever
ICML 2021. [Paper]
Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE
Jialun Peng, Dong Liu, Songcen Xu, Houqiang Li
CVPR 2021. [Paper]
Taming Transformers for High-Resolution Image Synthesis
Patrick Esser, Robin Rombach, B. Ommer
CVPR 2020. [Paper]
Generating Diverse High-Fidelity Images with VQ-VAE-2
Ali Razavi, Aäron van den Oord, Oriol Vinyals
NeurIPS 2019. [Paper]
DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
Arash Vahdat, W. Macready, Zhengbing Bian, Amir Khoshaman
ICML 2018. [Paper]
Neural Discrete Representation Learning
Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu
NeurIPS 2017. [Paper]
CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, Jie Tang
ICLR 2023. [Paper]
MAGVIT: Masked Generative Video Transformer
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, Lu Jiang
CVPR 2023. [Paper]
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, Devi Parikh
ECCV 2022. [Paper]
Latent Video Transformer
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, Evgeny Burnaev
VISIGRAPP 2021. [Paper]
Predicting Video with VQVAE
Jacob Walker, Ali Razavi, Aäron van den Oord
arXiv 2021. [Paper]
VideoGPT: Video Generation using VQ-VAE and Transformers
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, Aravind Srinivas
arXiv 2021. [Paper]
SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation
Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander Schwing, Liangyan Gui
CVPR 2023. [Paper]
AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, Shubham Tulsiani
CVPR 2022. [Paper]
Human Pose as Compositional Tokens
Zigang Geng, Chunyu Wang, Yixuan Wei, Ze Liu, Houqiang Li, Han Hu
CVPR 2023. [Paper]
Vector Quantized Diffusion Model with CodeUnet for Text-to-Sign Pose Sequences Generation
Pan Xie, Qipeng Zhang, Zexian Li, Hao Tang, Yao Du, Xiaohui Hu
arXiv 2022. [Paper]