Distributed Dynamic Map Fusion with Federated Knowledge Distill in CARLA Simulation

TODO

SUMO Co-simulation Ph.I
- know how SUMO works, and how with CARLA
- collect related examples as initialization
- ?
SUMO Co-simulation Ph.II
SUMO Co-simulation Ph.III
SUMO Co-simulation Ph.IV

Citation

@INPROCEEDINGS{INVS2021
	author={Z. {Zhang} and S. {Wang} and Y. {Hong} and L. {Zhou} and Q. {Hao},
	title={Distributed dynamic map fusion via federated learning for intelligent networked vehicles},
	booktitle={2021 {IEEE} International Conference on Robotics and Automation, {ICRA}
               2021, Xi’an, China, May 2021},
	year={2021}
}

Dependency

Ubuntu 18.04
Python 3.7+
CARLA >= 0.9.8, <=0.9.10
CUDA>=10.0
pytorch<=1.4.0
llvm>=10.0

Installation

Clone this repository to your workspace

git clone https://github.com/lasso-sustech/CARLA_INVS.git --branch=main --depth=1

Enter the directory "CARLA_INVS" and install dependencies with make
```
make dependency
```
It uses apt and pip3 with network access. You can try speed up downloading with fast mirror sites.
Download and extract the CARLA simulator somewhere (e.g., ~/CARLA), and update CARLA_PATH in params.py with absolute path to the CARLA folder location.

This repository is composed of three components: gen_data for dataset generation and visualization, PCDet for training and testing, fusion for global map fusion and visualization.

The three components share the same configuration file params.py.

Custom Dataset Generation

Features: 1) LiDAR/camera raw data collection in multi-agent synchronously; 2) data transformation to KITTI format; 3) data visualization with Open3d library.

Tuning: Tune the configurations as you like in params.py file under the gen_data section.

start up CarlaUE4.sh in your CARLA_PATH firstly and run the following script in shell to look for vehicles spawn points with point_id.
```
python3 gen_data/Scenario.py spawn
```
run the following script to generate multi-agent raw data
```
python3 gen_data/Scenario.py record [x_1,...x_N] [y_1,...y_M]
```
where x_1,...,x_N is list of point_ids (separated by comma) for human-driven vehicles, and y_1,...,y_M for autonomous vehicles with sensors installation.

The recording process would stop when Ctrl+C triggered, and the generated raw data will be put at $ROOT_PATH/raw_data.
Run the following script to transform raw data to KITTI format
```
python3 gen_data/Process.py raw_data/record2020_xxxx_xxxx
```
and the cooked KITTI format data will be put at $ROOT_PATH/dataset

(Optional) run the following script to view KITTI Format data sample with Open3D

# The vehicle_id is the intelligent vehicle ID, and the frame_ID is the index of dataset.
python3 gen_data/Visualization.py dataset/record2020_xxxx_xxxx vehicle_id frame_id

Training Procedures

Training for federated model

prepare sample dataset in $ROOT_PATH/data (link)
run python3 PCDet/INVS_main.py

Training for federated distill

To be updated.

Evaluation Procedures

View local map

cd fusion;
python3 visualization/Visualization_local.py ../data/record2020_1027_1957 713 39336

View fusion map

cd fusion;
python3 visualization/Visualization_fusion_map.py ../data/record2020_1027_1957 38549

Contact

Should you have any question, please create issues, or contact Shuai Wang.

Appendix

Raw Data Format

tmp
   +- record2020_xxxx_xxxx
      +- label                #tmp labels
      +- vhicle.xxx.xxx_xxx
          +- sensor.camera.rgb_xxx
              +- 0000.jpg
              +- 0001.jpg
          +- sensor.camera.rgb_xxx_label
              +- 0000.txt
              +- 0001.txt
          +- sensor.lidar.rgb_cast_xxx
              +- 0000.ply
              +- 0001.ply
      +- vhicle.xxx.xxx_xxx

label is the directory to save the tmp labels.

KITTI Format

dataset
   +- record2020_xxxx_xxxx
      +- global_label          #global labels
      +- vhicle.xxx.xxx_xxx
          +- calib00
              +- 0000.txt
              +- 0001.txt
          +- image00
              +- 0000.jpg
              +- 0001.jpg
          +- label00
              +- 0000.txt
              +- 0001.txt
          +- velodyne
              +- 0000.bin
              +- 0001.bin
      +- vhicle.xxx.xxx_xxx

label is the directory to save the ground truth labels.
calib is the calibration matrix from point cloud to image.

iamhyc / CARLA_INVS