Multi-Sensor-Fusion-Scene-Segmentation
ENPM673 Spring 2022: Perception for Autonomous Robots
Project 4
RGB segmentation and lidar fusion
Optical flow segmentation and lidar fusion
Instructions -
FusionNet
Semantic segmentation using RGB, LiDAR and Optical flow sensor data Trained on vkitti dataset.
Data Preparation
Download vkitti_1.3.1_depthgt.tar, vkitti_1.3.1_flowgt.tar, vkitti_1.3.1_rgb.tar, vkitti_1.3.1_scenegt.tar
from this link. Extract
the tarballs into one folder. Use the path of this folder for <path_to_dataset> argument throughout the following
instruction.
Training
To train, run the following commands:
python main.py --dataset_path <path_to_dataset> [--lidar] [--optical_flow]
The --lidar and --optical_flow flags are optional. Specifying the flags will train the network with corresponding
LiDAR and oflow data. Without the flags, the network will be trained only on RGB data.
Testing
To test, run the following commands:
python recons.py --dataset_path <path_to_dataset> --checkpoint_path <path_checkpoint_file> [--lidar] [--optical_flow]
Specify one or both of --lidar and --optical_flow flags if you had specified it during training. A pretrained checkpoint file
can be downloaded from here. Note that if you're using
our checkpoint file, do not specify the --lidar and --optical_flow flags.
Results
The below image shows the results of FusionNet. The pictures on the left column are the actual segmentation ground truth and the pictures on the right column are the segmentation masks obtained from our net.
BiSeNetv2:
For training and testing RGB and Optical flow data follow the instrcutions on the ReadMe of BiSeNet Download initial training weights from *here
Lidar Fusion
To fuse segmentation results from optical flow or rgb with lidar points, modify the input data here https://github.com/mjoshi07/Multi-Sensor-Fusion-Scene-Segmentation/blob/36d1966809ec5a2b19349d9fd629ec5e007ecbb5/lidar_fusion/Code/lidar_fusion.py#L87
Convert lidar .bin files to .pcd using the function
https://github.com/mjoshi07/Multi-Sensor-Fusion-Scene-Segmentation/blob/36d1966809ec5a2b19349d9fd629ec5e007ecbb5/lidar_fusion/Code/Utils.py#L11
Run the following command after setting all the data
python3 lidar_fusion.py
The output after execution will look something like this
Resources
- Raft: optical flow: learnopencv
- Raft: optical flow: paperswithcode
- Flownet 2.0: optical flow
- ppr: SegFlow: Segmentation + optical flow
- git: SegFlow: Segmentation + optical flow
- KITTI Scene Flow dataset
Authors -
Naitri Rajyaguru (117361919)
Mayank Joshi (117555264)
Nitesh Jha (117525366)
Aneesh Dandime (118418506)
Tanuj Thakkar (117817539)