AndrewHazelden / Rust-WASM-Gauzilla-Renderer

Gauzilla: a 3D Gaussian Splatting renderer written in Rust for WebAssembly with lock-free multithreading

Gauzilla

A 3D Gaussian Splatting (3DGS) renderer written in Rust for platform-agnostic WebAssembly (WASM) with lock-free multithreading.

• Uses WebGL and CPU splat sorting (based on splat) for high compatibility among web browsers
• Circumvents WASM's limitations in multithreading via the use of the lock-free bus mechanism
• Uses rfd to securely load PLY files that are stored locally on the host machine

Introduction

The innovative rendering technique known as 3D Gaussian Splatting (3DGS) [1] represents a Machine Learning-oriented approach to 3D rendering, specifically designed for Novel View Synthesis (NVS). It facilitates the real-time photorealistic rendering of scenes reconstructed from images and videos captured using conventional smartphone cameras. Since its release in 2023 there has been a Cambrian explosion of 3DGS applications and extensions, such as 4DGS [2], D3GA [3], SLAM [4], SC-GS [5], GPS [6], GHA [7], etc.

One of the orignal ideas of splatting Gaussian ellipses goes back to 2002 [8] and 3DGS uses almost the same approach for its efficient forward rendering on the GPU.

Briefly, a 3D elliptical Gaussian centered at a point p with a covariance matirx V is defined as:

The key is to compute the 3x3 covariance matrix in ray coordinates:

where W is the view transformation (camera) matrix, V_k = R * S * S^T * R^T (which can be computed from quaternions and scales stored in the PLY file), and

is a Jacobian or the partial derivative of the camera-ray mapping at the point t_k in camera space.

Once the covariance matrix in ray coordinates is obtained via the simple matrix multiplications above, one can compute from the top-left 2x2 component of the matrix the major and minor axes representing the size and orientation of Gaussian ellipses, which can subsequently be rendered on the GPU efficiently using Geometry Instancing.

How to Build & Use

1. Install the nightly version of Rust: rustup toolchain install nightly
2. Install wasm-pack and sfz
3. Run ./build.sh and open the locally-served URL in a web browser
4. Open a PLY file formatted for 3DGS (eg. download the official pre-trained models)

Orbit Camera Controls:

Left mouse button   - Rotate view
Middle mouse button - Zoom in/out

Fly Camera Controls:

Left mouse button   - Rotate view
Middle mouse button - Move forward/backward
Right mouse button  - Move left/right/up/down

ToDo

• Optimize Scene::sort() and Scene::generate_texture() (eg. parallelize using wasm-bindgen-rayon)
• Show the averages of FPS and CPU sort time instead of raw values
• Display a progress bar for PLY file loading
• Allow camera controls with keyboard
• Write a WebGPU render path (cf. splatter)

References

1. Kerbl et al., "3D Gaussian Splatting for Real-Time Radiance Field Rendering," 2023.
2. Wu et al., "4D Gaussian Splatting for Real-Time Dynamic Scene Rendering," 2023.
3. Zielonka et al., "D3GA - Drivable 3D Gaussian Avatars," 2023.
4. Matsuki et al., "Gaussian Splatting SLAM," 2023.
5. Huang et al., "SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes," 2023.
6. Zheng et al., "GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis," 2023.
7. Xu et al., "Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians," 2022.
8. Zwicker et al., "EWA Splatting," 2002.