fastai_sparse
This is an extension of the fast.ai library to train Submanifold Sparse Convolution Networks that apply to 2D/3D sparse data, such as 3D geometric meshes or point clouds in euclidian space
Currently, this library has SparseConvNet under the hood which is the best in 3D (ScanNet benchmark, ShapeNet workshop) so far.
Installation
fastai_sparse is compatible with: Python 3.6, PyTorch 1.0+
Some key dependences:
- Fast.ai
- PyTorch sparse convolution models: SparseConvNet.
- PLY file reader and 3D geometry mesh transforms are implemented by trimesh.
- ipyvolume is used for interactive visualisation in jupyter notebooks examples.
See details in INSTALL.md
Features:
- fast.ai train/inference loop concept (Model + DataBunch --> Learner) Classes overview
- model training best practices provided by fast.ai (Learning Rate Finder, One Cycle policy)
- 3D transforms for data preprocessing and augmentation:
- mesh-level transforms and features extraction (surface normals, triangle area,...)
- points-level spatial transforms (affine, elastic,...)
- points-level features (color, brightness)
- mesh to points
- points to sparse voxels
- metrics (IoU, avgIoU, ) calculation and tracking
- visualization utils (batch generator output)
Notebooks with examples
- 3D Transformation examples notebook
- ScanNet 3D indoor scene semantic segmentation detailed notebook
- ScanNet 3D example with surface normals notebook
- ShapeNet 3D semantic segmentation detailed notebook
TODO
Priority 1:
- Separate 3D augmentation library with key points, spatial targets
- Prediction pipeline
- Classification/regression examples
- Spatial targets (bounding box, key points, axes)
Priority 2:
- TTA
- Multi-GPU
- PointNet-like feature extraction layer ("VoxelNet" architecture)
- Confidence / heatmap / kernels visualization