carlaILTrainer
Carla Imitation Learning Trainer
The main project belongs to Carla Team, and I just wrote a training system for the main project. Here you can find more information about the original project.
Please check out issue and issue to find out why this code was published.
TODO:
- Run all branches , but just back propagate on one ( use a mask for that). @merryHunter solved this in the python file.
- Use the Tensorflow queues for data loading
- Compatibility fix for carla checkpoint loading
- Fix the loss function (Use only one loss)
- Balance the dataset according to labels in training mode
- Test the whole system with the recent benchmark
Contribution:
Please feel free to pull a request. Any features and any changes are welcome.
All Contributors:
- Mojtaba Va'lipour: @mvpcom
- Ivan Chernuha : @merryHunter
Paper Results:
This is an auto table generator for the paper writing purpose. Tables are based on the CoRL 2017 Carla Team Paper. This can help me to have a gorgeous table easier and with a lot of fewer efforts. Currently, Text, LaTex, and HTML are supported. Before you are going to run this code, you have to prepare the results for three different runs for each town in Carla. You can see an example in the following passage.
Output Support: Text, Latex, HTML
Example: python calc_CORL2017_Tables.py --path "./CarlaPaper_ReExperiment/" -v -n "CoRL-2017 Carla Paper"
"./CarlaPaper_ReExperiment/" contains the following folders:
CarlaPaper_ReExperiment/
├── CarlaPaperModel_Test01_CoRL2017_Town01
│ ├── log_201807010542
│ ├── log_201807011830
│ ├── log_201807020224
│ ├── measurements.csv
│ ├── metrics.json
│ ├── res
│ └── summary.csv
├── CarlaPaperModel_Test01_CoRL2017_Town02
│ ├── log_201807031155
│ ├── log_201807032256
│ ├── log_201807050552
│ ├── measurements.csv
│ └── summary.csv
├── CarlaPaperModel_Test02_CoRL2017_Town01
│ ├── log_201807020250
│ ├── log_201807020739
│ ├── log_201807021521
│ ├── measurements.csv
│ └── summary.csv
├── CarlaPaperModel_Test02_CoRL2017_Town02
│ ├── log_201807050555
│ ├── measurements.csv
│ └── summary.csv
├── CarlaPaperModel_Test03_CoRL2017_Town01
│ ├── log_201807022050
│ ├── log_201807030243
│ ├── log_201807030738
│ ├── measurements.csv
│ └── summary.csv
├── CarlaPaperModel_Test03_CoRL2017_Town02
│ ├── log_201807052015
│ ├── log_201807060744
│ ├── measurements.csv
│ └── summary.csv
├── results.html
└── results.laTex
P.S: results.html and results.laTex are produced by this code after execution. These are a little bit different because of different configuration and different Carla version. In these results, bikes were in the test only. Please share your code and enhancements by a pull request with the world if you use this in your papers and add a feature.
Example Output:
MODEL: CoRL-2017 Carla Paper_ReExperiment
| Tasks | Training Conditions | New Town | New Weather | New Town & Weather |
|---|---|---|---|---|
| Straight | 0.98 +/- 0.0 | 0.97 +/- 0.0 | 0.97 +/- 0.01 | 0.95 +/- 0.0 |
| One Turn | 0.94 +/- 0.01 | 0.68 +/- 0.01 | 0.94 +/- 0.02 | 0.7 +/- 0.0 |
| Navigation | 0.89 +/- 0.0 | 0.42 +/- 0.0 | 0.83 +/- 0.01 | 0.46 +/- 0.02 |
| Nav. Dynamic | 0.88 +/- 0.0 | 0.42 +/- 0.02 | 0.8 +/- 0.03 | 0.46 +/- 0.03 |
| Infractions | Training Conditions | New Town | New Weather | New Town & Weather |
|---|---|---|---|---|
| Opposite Lane | 16.78 +/- 0.04 | 2.89 +/- 0.0 | 8.13 +/- 0.13 | 3.89 +/- 0.5 |
| Sidewalk | 7.46 +/- 1.32 | 2.65 +/- 0.34 | 8.13 +/- 0.13 | 2.26 +/- 0.2 |
| Collision-static | 16.78 +/- 0.04 | 8.67 +/- 0.0 | 8.13 +/- 0.13 | 12.73 +/- 0.01 |
| Collision-car | 16.78 +/- 0.04 | 8.67 +/- 0.0 | 8.13 +/- 0.13 | 12.73 +/- 0.01 |
| Collision-pedestrian | 16.78 +/- 0.04 | 8.67 +/- 0.0 | 8.13 +/- 0.13 | 12.73 +/- 0.01 |
| Infractions | Training Conditions | New Town | New Weather | New Town & Weather |
|---|---|---|---|---|
| Opposite Lane | 13.09 +/- 2.68 | 1.03 +/- 0.11 | 16.86 +/- 0.09 | 1.35 +/- 0.1 |
| Sidewalk | 11.22 +/- 0.1 | 0.9 +/- 0.05 | 16.86 +/- 0.09 | 0.95 +/- 0.04 |
| Collision-static | 33.66 +/- 0.3 | 17.68 +/- 0.26 | 16.86 +/- 0.09 | 26.88 +/- 0.7 |
| Collision-car | 33.66 +/- 0.3 | 17.68 +/- 0.26 | 16.86 +/- 0.09 | 26.88 +/- 0.7 |
| Collision-pedestrian | 33.66 +/- 0.3 | 17.68 +/- 0.26 | 16.86 +/- 0.09 | 26.88 +/- 0.7 |
| Infractions | Training Conditions | New Town | New Weather | New Town & Weather |
|---|---|---|---|---|
| Opposite Lane | 17.4 +/- 3.7 | 2.26 +/- 0.05 | 23.47 +/- 8.47 | 3.12 +/- 0.17 |
| Sidewalk | 16.27 +/- 4.14 | 0.62 +/- 0.03 | 12.71 +/- 3.75 | 0.68 +/- 0.03 |
| Collision-static | 66.63 +/- 0.2 | 24.88 +/- 0.5 | 35.12 +/- 0.28 | 40.43 +/- 0.45 |
| Collision-car | 66.63 +/- 0.2 | 24.88 +/- 0.5 | 35.12 +/- 0.28 | 40.43 +/- 0.45 |
| Collision-pedestrian | 66.63 +/- 0.2 | 24.88 +/- 0.5 | 35.12 +/- 0.28 | 40.43 +/- 0.45 |
| Infractions | Training Conditions | New Town | New Weather | New Town & Weather |
|---|---|---|---|---|
| Opposite Lane | 20.17 +/- 2.74 | 1.5 +/- 0.18 | 24.95 +/- 12.95 | 1.66 +/- 0.04 |
| Sidewalk | 13.05 +/- 2.98 | 0.65 +/- 0.06 | 4.26 +/- 0.87 | 0.68 +/- 0.07 |
| Collision-static | 4.51 +/- 1.07 | 0.38 +/- 0.05 | 2.15 +/- 0.55 | 0.41 +/- 0.05 |
| Collision-car | 1.19 +/- 0.14 | 0.28 +/- 0.03 | 1.48 +/- 0.28 | 0.27 +/- 0.02 |
| Collision-pedestrian | 14.14 +/- 5.64 | 1.66 +/- 0.15 | 7.35 +/- 0.97 | 1.83 +/- 0.14 |
| Number of Infractions | Training Conditions | New Town | New Weather | New Town & Weather |
|---|---|---|---|---|
| Opposite Lane | 0.0 +/- 0.0 | 3.0 +/- 0.0 | 0.0 +/- 0.0 | 3.33 +/- 0.47 |
| Sidewalk | 2.33 +/- 0.47 | 3.33 +/- 0.47 | 0.0 +/- 0.0 | 5.67 +/- 0.47 |
| Collision-static | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 |
| Collision-car | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 |
| Collision-pedestrian | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 |
| Number of Infractions | Training Conditions | New Town | New Weather | New Town & Weather |
|---|---|---|---|---|
| Opposite Lane | 2.67 +/- 0.47 | 17.33 +/- 1.7 | 0.33 +/- 0.47 | 20.0 +/- 1.41 |
| Sidewalk | 3.0 +/- 0.0 | 19.67 +/- 1.25 | 0.33 +/- 0.47 | 28.33 +/- 0.94 |
| Collision-static | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 |
| Collision-car | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 |
| Collision-pedestrian | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 |
| Number of Infractions | Training Conditions | New Town | New Weather | New Town & Weather |
|---|---|---|---|---|
| Opposite Lane | 4.0 +/- 0.82 | 11.0 +/- 0.0 | 1.67 +/- 0.47 | 13.0 +/- 0.82 |
| Sidewalk | 4.33 +/- 0.94 | 40.33 +/- 1.25 | 3.0 +/- 0.82 | 59.33 +/- 1.7 |
| Collision-static | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 |
| Collision-car | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 |
| Collision-pedestrian | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 | 0.0 +/- 0.0 |
| Number of Infractions | Training Conditions | New Town | New Weather | New Town & Weather |
|---|---|---|---|---|
| Opposite Lane | 3.33 +/- 0.47 | 16.67 +/- 1.89 | 2.33 +/- 1.89 | 22.67 +/- 0.47 |
| Sidewalk | 5.33 +/- 1.25 | 38.33 +/- 3.4 | 8.33 +/- 1.89 | 56.33 +/- 6.85 |
| Collision-static | 15.33 +/- 3.09 | 67.33 +/- 10.4 | 17.0 +/- 4.97 | 94.0 +/- 13.49 |
| Collision-car | 56.0 +/- 6.38 | 90.33 +/- 11.67 | 24.0 +/- 5.72 | 139.67 +/- 8.73 |
| Collision-pedestrian | 5.33 +/- 1.7 | 15.0 +/- 1.41 | 4.67 +/- 0.47 | 20.67 +/- 1.7 |