apple / ml-pointersect

Pointersect: Neural Rendering with Cloud-Ray Intersection

[Website] [Paper] [Docs] [Examples] [Videos]

Pointersect is a plug-and-play method for rendering point clouds. It is differentiable and does not require per-scene optimization.

Usage

Given a point cloud and a ray, pointersect returns:

• the intersection point between the ray and the underlying surface represented by the point cloud;
• surface normal at the intersection point; and
• color/material interpolation weights of neighboring points.

You can use point clouds containing only xyz---neither color nor vertex normal is needed.

Examples

1. We use the surface normal estimated by pointersect to relight a point cloud.

2. Even though pointersect is designed to render clean point clouds, here is an example where we use a pretrained pointersect model to render a lidar-scanned point cloud without any optimization.

3. Edit and render without re-optimization.

4. We use pointersect with path tracing to render the global illumination of a scene.

How to use

You can use pointersect by installing the pypi package:

pip install pointersect

or

# in the repo root 
pip install .

We provide API, command line tool, and training script for using pointersect. See the documentation for instructions.

We also provide a few examples if you want to jump in directly :)

Citation

If you use this software package, please cite our paper:

@inproceedings{chang2023pointersect,
  author={Jen-Hao Rick Chang and Wei-Yu Chen and Anurag Ranjan and Kwang Moo Yi and Oncel Tuzel},
  title={Pointersect: Neural Rendering with Cloud-Ray Intersection},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year = {2023}
}