Multi modal fusion

[Tom Sanitz]

Is there any example on how to do a multi modal approach in Det3d? Like how to adjust the data loading pipeline and so on? I am mainly using the nuScenes dataset for now.

[Tom Sanitz]

Never mind, found a way to access the camera images by extending the nuscenes.py

[Tom Sanitz]

So other people might struggle less in the future, you need to add your custom entry like "camera_images" to the list in: det3d/torchie/parallel/collate.py.py:
e.g.:
if key in ["voxels", "num_points", "num_gt", "voxel_labels", "num_voxels",
"cyv_voxels", "cyv_num_points", "cyv_num_voxels", "camera_images"]:

Otherwise they will be converted back to a numpy array even if they were torch.Tensors before.