Awesome Implicit Neural Representations

A curated list of resources on implicit neural representations, inspired by awesome-computer-vision. Work-in-progress.

This list does not aim to be exhaustive, as implicit neural representations are a rapidly evolving & growing research field with hundreds of papers to date.

Instead, this list aims to list papers introducing key concepts & foundations of implicit neural representations across applications. It's a great reading list if you want to get started in this area!

For most papers, there is a short summary of the most important contributions.

Disclosure: I am an author on the following papers:

What are implicit neural representations?

Implicit Neural Representations (sometimes also referred to coordinate-based representations) are a novel way to parameterize signals of all kinds. Conventional signal representations are usually discrete - for instance, images are discrete grids of pixels, audio signals are discrete samples of amplitudes, and 3D shapes are usually parameterized as grids of voxels, point clouds, or meshes. In contrast, Implicit Neural Representations parameterize a signal as a continuous function that maps the domain of the signal (i.e., a coordinate, such as a pixel coordinate for an image) to whatever is at that coordinate (for an image, an R,G,B color). Of course, these functions are usually not analytically tractable - it is impossible to "write down" the function that parameterizes a natural image as a mathematical formula. Implicit Neural Representations thus approximate that function via a neural network.

Why are they interesting?

Implicit Neural Representations have several benefits: First, they are not coupled to spatial resolution anymore, the way, for instance, an image is coupled to the number of pixels. This is because they are continuous functions! Thus, the memory required to parameterize the signal is independent of spatial resolution, and only scales with the complexity of the underyling signal. Another corollary of this is that implicit representations have "infinite resolution" - they can be sampled at arbitrary spatial resolutions.

This is immediately useful for a number of applications, such as super-resolution, or in parameterizing signals in 3D and higher dimensions, where memory requirements grow intractably fast with spatial resolution.

However, in the future, the key promise of implicit neural representations lie in algorithms that directly operate in the space of these representations. In other words: What's the "convolutional neural network" equivalent of a neural network operating on images represented by implicit representations? Questions like these offer a path towards a class of algorithms that are independent of spatial resolution!

Papers

Implicit Neural Representations of Geometry

The following three papers first (and concurrently) demonstrated that implicit neural representations outperform grid-, point-, and mesh-based representations in parameterizing geometry and seamlessly allow for learning priors over shapes.

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation (Park et al. 2019)
Occupancy Networks: Learning 3D Reconstruction in Function Space (Mescheder et al. 2019)
IM-Net: Learning Implicit Fields for Generative Shape Modeling (Chen et al. 2018)

Since then, implicit neural representations have achieved state-of-the-art-results in 3D computer vision:

Sal: Sign agnostic learning of shapes from raw data (Atzmon et al. 2019) shows how we may learn SDFs from raw data (i.e., without ground-truth sigend distance values)
Implicit Geometric Regularization for Learning Shapes (Gropp et al. 2020) shows how we may learn SDFs from raw data (i.e., without ground-truth sigend distance values)
Local Implicit Grid Representations for 3D Scenes, Convolutional Occupancy Networks, Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction concurrently proposed hybrid voxelgrid/implicit representations to fit large-scale 3D scenes.
Implicit Neural Representations with Periodic Activation Functions (Sitzmann et al. 2020) demonstrates how we may parameterize room-scale 3D scenes via a single implicit neural representation by leveraging sinusoidal activation functions.

Implicit representations of Geometry and Appearance

From 2D supervision only (“inverse graphics”)

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations proposed to learn an implicit representations of 3D shape and geometry given only 2D images, via a differentiable ray-marcher, and generalizes across 3D scenes for reconstruction from a single image via hyper-networks. This was demonstrated for single-object scenes, but also for simple room-scale scenes (see talk).
Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision (Niemeyer et al. 2020), replaces LSTM-based ray-marcher in SRNs with a fully-connected neural network & analytical gradients, enabling easy extraction of the final 3D geometry.
Neural Radiance Fields (NeRF) (Mildenhall et al. 2020) proposes positional encodings, volumetric rendering & ray-direction conditioning for high-quality reconstruction of single scenes, and has spawned a large amount of follow-up work on volumetric rendering of 3D implicit representations. For a curated list of NeRF follow-up work specifically, see awesome-NeRF
SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images (Lin et al. 2020), demonstrates how we may train Scene Representation Networks from a single observation only.
Pixel-NERF (Yu et al. 2020) proposes to condition a NeRF on local features lying on camera rays, extracted from contact images, as proposed in PiFU (see "from 3D supervision").
Multiview neural surface reconstruction by disentangling geometry and appearance (Yariv et al. 2020) demonstrates sphere-tracing with positional encodings for reconstruction of complex 3D scenes, and proposes a surface normal and view-direction dependent rendering network for capturing view-dependent effects.

With view-dependent effects

Neural Radiance Fields (NeRF) (Mildenhall et al. 2020)
Multiview neural surface reconstruction by disentangling geometry and appearance (Yariv et al. 2020)

From 3D supervision

Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization (Saito et al. 2019) Pifu first introduced the concept of conditioning an implicit representation on local features extracted from context images. Follow-up work achieves photo-realistic, real-time re-rendering.
Texture Fields: Learning Texture Representations in Function Space (Oechsle et al.)

For dynamic scenes

Occupancy flow: 4d reconstruction by learning particle dynamics (Niemeyer et al. 2019) first proposed to learn a space-time neural implicit representation by representing a 4D warp field with an implicit neural representation.

The following papers concurrently proposed to leverage a similar approach for the reconstruction of dynamic scenes from 2D observations only via Neural Radiance Fields.

Hybrid implicit / explicit (condition implicit on local features)

The following three papers concurrently proposed to condition an implicit neural representation on local features stored in a voxelgrid:

Local Implicit Grid Representations for 3D Scenes
Convolutional Occupancy Networks
Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction
Neural Sparse Voxel Fields Applies a similar concept to neural radiance fields.

The following papers condition a deep signed distance function on local patches:

Representation learning with implicit neural representations for downstream tasks

Inferring Semantic Information with 3D Neural Scene Representations leverages features learned by Scene Representation Networks for weakly supervised semantic segmentation of 3D objects.

Generalization & Meta-Learning with Neural Implicit Representations

DeepSDF, Occupancy Networks, IM-Net concurrently proposed conditioning via concatenation.
Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization (Saito et al. 2019) proposed to locally condition implicit representations on ray features extracted from context images.
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (Sitzmann et al. 2019) proposed meta-learning via hypernetworks.
MetaSDF: MetaSDF: Meta-Learning Signed Distance Functions (Sitzmann et al. 2020) proposed gradient-based meta-learning for implicit neural representations
SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images (Lin et al. 2020) show how to learn 3D implicit representations from single-image supervision only.
Learned Initializations for Optimizing Coordinate-Based Neural Representations (Tancik et al. 2020) explored gradient-based meta-learning for NeRF.

Fitting high-frequency detail with positional encoding & periodic nonlinearities

Neural Radiance Fields (NeRF) (Mildenhall et al. 2020) proposed positional encodings.
Implicit Neural Representations with Periodic Activation Functions (Sitzmann et al. 2020) proposed implicit representations with periodic nonlinearities.
Fourier features let networks learn high frequency functions in low dimensional domains (Tancik et al. 2020) explores positional encodings in an NTK framework.

Implicit Neural Representations of Images

Compositional Pattern-Producing Networks: Compositional pattern producing networks: A novel abstraction of development (Stanley et al. 2007) first proposed to parameterize images implicitly via neural networks.
Implicit Neural Representations with Periodic Activation Functions (Sitzmann et al. 2020) proposed to generalize across implicit representations of images via hypernetworks.
X-Fields: Implicit Neural View-, Light- and Time-Image Interpolation (Bemana et al. 2020) parameterizes the Jacobian of pixel position with respect to view, time, illumination, etc. to naturally interpolate images.
Learning Continuous Image Representation with Local Implicit Image Function (Chen et al. 2020) proposed a hypernetwork-based GAN for images.

Composing implicit neural representations

The following papers propose to assemble scenes from per-object 3D implicit neural representations.

Implicit Representations for Partial Differential Equations & Boundary Value Problems

Implicit Geometric Regularization for Learning Shapes (Gropp et al. 2020) learns SDFs by enforcing constraints of the Eikonal equation via the loss.
Implicit Neural Representations with Periodic Activation Functions (Sitzmann et al. 2020) proposes to leverage the periodic sine as an activation function, enabling the parameterization of functions with non-trivial higher-order derivatives and the solution of complicated PDEs.
AutoInt: Automatic Integration for Fast Neural Volume Rendering (Lindell et al. 2020)

Generative Adverserial Networks with Implicit Representations

For 3D

Generative Radiance Fields for 3D-Aware Image Synthesis (Schwarz et al. 2020)
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (Chan et al. 2020)

For 2D

Adversarial Generation of Continuous Images (Skorokhodov et al. 2020)
Learning Continuous Image Representation with Local Implicit Image Function (Chen et al. 2020)
Image Generators with Conditionally-Independent Pixel Synthesis (Anokhin et al. 2020)

Talks

Links

awesome-NeRF - List of implicit representations specifically on neural radiance fields (NeRF)

License

License: MIT

About

A curated list of resources on implicit neural representations.

MIT License