Media formats are moving beyond 1D audio and 2D images which are viewed at static viewing angles and scale to 3D content which are viewed from dynamic viewing angles and scale. This has broad implications in gaming, virtual concerts, television, augmented reality, and more. Compression and streaming are at the core of the design of the next generation of media systems, and 3D brings new challenges which require innovations in algorithms as well as systems.
This repository collects papers and talks and blog posts which broadly discuss compression and streaming for 3D content. This is an exciting field which combines computer graphics, computer vision, machine learning, multimedia, systems, signal processing, information theory, security, and more!
This is very much work in progress and missings lots of work. The categorizations etc are very likely to change.
Compression is Again Driving the Evolution of Media
Philip A. Chou
State-of-the-art in Compressed GPU-Based Direct Volume Rendering
Marcos Balsa Rodríguez, Enrico Gobbetti, Jose Antonio Iglesias Guitian, Maxim Makhinya, Fabio Marton, Renato Pajarola, Susanne K Suter
Eurographics State-of-the-Art Reports 2014
3D Mesh Compression: Survey, Comparisons, and Emerging Trends
Adrien Maglo, Guillaume Lavoue, Florent Dupont, Celine Hudelot
Eurographics State-of-the-Art Reports 2015
Nonlinear Transform Coding
Johannes Ballé, Philip A. Chou, David Minnen, Saurabh Singh, Nick Johnston, Eirikur Agustsson, Sung Jin Hwang, George Toderici
IEEE Journal of Selected Topics in Signal Processing 2020
Toward Truly Immersive Holographic-Type Communication: Challenges and Solutions
Alexander Clemm, Maria Torres Vega, Hemanth Kumar Ravuri, Tim Wauters, Filip De Turkc
IEEE Communications Magazine 2020
Extremely Interactive and Low-Latency Services in 5G and Beyond Mobile Systems
Tarik Taleb, Zinelaabidine Nadir, Hannu Flinck, JaeSeung Song
IEEE Communications Standards Magazine 2021
Neural Fields in Visual Computing and Beyond
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, Srinath Sridhar
Eurographics State-of-the-Art Reports 2022
MPEG-I: Coded Representation of Immersive Media
Emerging MPEG Standards for Point Cloud Compression
Sebastian Schwarz, Marius Preda, Madhukar Budagavi, Pablo Cesar, Philip A. Chou, Robert A. Cohen, Maja Krivokuca, Sebastien Lasserre, Zhu Li, Joan Llach, Khaled Mammou, Rufael Mekuria, Ohji Nakagami, Ernestasia Siahaan, Ali Tabatabai, Alexis M. Tourapis, Vladyslav Zakharchenko
IEEE JETCAS 2019
A Comprehensive Study and Comparison of Core Technologies for MPEG 3-D Point Cloud Compression
Hao Liu, Hui Yuan, Qi Liu, Junhui Hou, Ju Liu
IEEE Transactions on Broadcasting 2019
An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC)
D. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, A. Tabatabai
APSIPA Transactions on Signal and Information Processing 2020
GigaVoxels: ray-guided streaming for efficient and detailed voxel rendering
Cyril Crassin, Fabrice Neyret, Sylvain Lefebvre, Elmar Eisemann
I3D 2009
Holoportation: Virtual 3D Teleportation in Real-time
Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, Shahram Izadi
ACM UIST 2016
Project Startline: A high-fidelity telepresence system
Jason Lawrence Dan B Goldman Supreeth Achar Gregory Major Blascovich Joseph G. Desloge Tommy Fortes Eric M. Gomez Sascha Häberling Hugues Hoppe Andy Huibers Claude Knaus Brian Kuschak Ricardo Martin-Brualla Harris Nover Andrew Ian Russell Steven M. Seitz Kevin Tong
ACM Transactions on Graphics 2021
Nanite: A Deep Dive
Brian Karis, Rune Stubbe, Graham Wihlidal
ACM SIGGRAPH 2021, Advances in Real-time Rendering in Games
Self-similarity for accurate compression of point sampled surfaces
Julie Digne, Raphaelle Chaine, Sebastien Valette
Eurographics 2014
3D point cloud compression using conventional image compression for efficient data transmission
Hamidreza Houshiar, Andreas Nuchter
IEEE ICAT 2015
Compression and Rendering of Textured Point Clouds via Sparse Coding
Kersten Schuster, Philip Trettner, Patric Schmitz, Julian Schakib, Leif Kobbelt
HPG 2021
We draw a slight distinction between point clouds and voxelized point clouds. The latter is quantized into integer coordinates, while the general class of point clouds are still often quantized but not necessarily into a regular grid.
A Volumetric Approach to Point Cloud Compression—Part I: Attribute Compression
Philip A. Chou, Maxim Koroteev, Maja Krivokuca
IEEE Transactions of Image Processing
A Volumetric Approach to Point Cloud Compression—Part II: Geometry Compression
Maja Krivokuca, Philip A. Chou, Maxim Koroteev
IEEE Transactions of Image Processing
Point Cloud Attribute Compression with Graph Transform
Cha Zhang, Dinei Florencio, Charles Loop
IEEE ICIP 2014
Compression of 3D Point Clouds Using a Region-Adaptive Hierarchical Transform
Ricardo L. de Queiroz, Philip A. Chou
IEEE Transactions on Image Processing 2016
Geometry and attribute compression for voxel scenes
Bas Dado, Timothy R. Kol, Pablo Bauszat, Jean-Marc Thiery, Elmar Eisemann
Eurographics 2016
Real-time compression and streaming of 4D performances
Danhang Tang, Mingsong Dou, Peter Lincoln, Philip Davidson, Kaiwen Guo, Jonathan Taylor, Sean Fanello, Cem Keskin, Adarsh Kowdle, Sofien Bouaziz, Shahram Izadi, Andrea Tagliasacchi
SIGGRAPH Asia 2018
Deep Implicit Volume Compression
Danhang Tang, Saurabh Singh, Philip A. Chou, Christian Haene, Mingsong Dou, Sean Fanello, Jonathan Taylor, Philip Davidson, Onur G. Guleryuz, Yinda Zhang, Shahram Izadi, Andrea Tagliasacchi, Sofien Bouaziz, Cem Keskin
CVPR 2020
OctSqueeze: Octree-Structured Entropy Model for LiDAR Compression
Lila Huang, Shenlong Wang, Kelvin Wong, Jerry Liu, Raquel Urtasun
CVPR 2020
MuSCLE: Multi Sweep Compression of LiDAR using Deep Entropy Models
Sourav Biswas, Jerry Liu, Kelvin Wong, Shenlong Wang, Raquel Urtasun
NeurIPS 2020
Region adaptive graph fourier transform for 3d point clouds
Eduardo Pavez, Benjamin Girault, Antonio Ortega, Philip A. Chou
IEEE ICIP 2020
VoxelContext-Net: An Octree based Framework for Point Cloud Compression
Zizheng Que, Guo Lu, Dong Xu
CVPR 2021
Moving Basis Decomposition for Precomputed Light Transport
Ari Silvennoinen, Peter-Pike Sloan
EGSR 2021
Volumetric Attribute Compression for 3D Point Clouds using Feedforward Network with Geometric Attention
Tam Thuc Do, Philip A. Chou, Gene Cheung
ICASSP 2023
Deep Geometry Post-Processing for Decompressed Point Clouds
Xiaoqing Fan, Ge Li, Dingquan Li, Yurui Ren, Wei Gao, Thomas H. Li
ArXiv, 2022
Rate-Utility Optimized Streaming of Volumetric Media for Augmented Reality
Jounsup Park, Philip A. Chou, Jenq-Neng Hwang
IEEE JETCAS 2019
Towards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression
Jeroen van der Hooft, Tim Wauters, Filip De Turck, Christian Timmerer, Hermann Hellwagner
ACM MM 2019
Neural fields offer a new paradigm for compression which is based on function fitting (akin to some earlier works on compression via polynomial fitting, etc). Eventually I will write something which explains why neural fields might be a cool alternative / complementary to traditional voxel-adjacent compression methods.
There is a lot of work in this area. Here, we only list papers which make an experimental comparison in terms of compression (broadly defined).
On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes
Thomas Davies, Derek Nowrouzezahrai, Alec Jacobson
Technical Report
Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Surfaces
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, Sanja Fidler
CVPR 2021
ACORN: Adaptive Coordinate Networks for Neural Scene Representation
Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, Gordon Wetzstein
SIGGRAPH 2021
BACON: Band-limited Coordinate Networks for Multiscale Scene Representation
David B. Lindell, Dave Van Veen, Jeong Joon Park, Gordon Wetzstein
ArXiv 2021
Compressive Neural Representations of Volumetric Scalar Fields
Yuzhe Lu, Kairong Jiang, Joshua A. Levine, Matthew Berger
EuroVis 2021
3D Scene Compression through Entropy Penalized Neural Representation Functions
Thomas Bird, Johannes Balle, Saurabh Singh, Philip A. Chou
IEEE PCS 2021
NeRV: Neural Representations for Videos
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, Abhinav Shrivastava
NeurIPS 2021
COIN: COmpression with Implicit Neural Representations
Emilien Dupont, Adam Goliński, Milad Alizadeh, Yee Whye Teh, Arnaud Doucet
ICLR 2021 Neural Compression Workshop
Implicit Neural Representations for Image Compression
Yannick Strümpler, Janis Postels, Ren Yang, Luc Van Gool, and Federico Tombari
ECCV 2022
COIN++: Data Agnostic Neural Compression
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Goliński, Yee Whye Teh, Arnaud Doucet
ArXiv 2022
LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks
Berivan Isik, Philip A. Chou, Sung Jin Hwang, Nick Johnston, George Toderici
ArXiv 2021
Implicit Neural Video Compression
Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, Taco Cohen
ArXiv 2021
Neural Motion Compression with Frequency-adaptive Fourier Feature Network
Kenji Tojo, Yifei Chen, Nobuyuki Umetani
Eurographics 2022
Variable Bitrate Neural Fields
Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Muller, Morgan McGuire, Alec Jacobson, Sanja Fidler
SIGGRAPH 2022
Streamable Neural Fields
Junwoo Cho, Seungtae Nam, Daniel Rho, Jong Hwan Ko, Eunbyung Park
ECCV 2022
On Quantizing Implicit Neural Representations
Cameron Gordon, Shin-Fang Chng, Lachlan MacDonald, Simon Lucey
WACV 2023
Masked Wavelet Representation for Compact Neural Radiance Fields
Daniel Rho, Byeonghyeon Lee, Seungtae Nam, Joo Chan Lee, Jong Hwan Ko, Eunbyung Park
CVPR 2023
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes
Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P. Srinivasan, Ben Mildenhall, Andreas Geiger, Jonathan T. Barron, Peter Hedman
ArXiv 2023