Source code for the paper: Robust Monocular Depth Estimation under Challenging Conditions
Authors: Stefano Gasperini*, Nils Morbitzer*, HyunJun Jung, Nassir Navab, and Federico Tombari
*equal contribution
[Project Page] [ICCV Paper (CVF)] [arXiv] [Translated Images (Google Form)]
md4all is a simple and effective solution that works reliably under both adverse and ideal conditions and for different types of learning supervision. We achieve this by exploiting the already proven efficacy of existing architectures and losses under perfect settings. Therefore, we strive to provide valid training signals independently of what is given as input.
Please get in touch with Stefano Gasperini (stefano.gasperini@tum.de) or Nils Morbitzer (nils.morbitzer@tum.de) if you have any questions!
This repository provides the PyTorch implementation for our self-supervised md4all model based on Monodepth2. Soon, we will add the code for the fully-supervised version based on AdaBins.
Soon we will be able to provide commercial licenses. Please reach out to us if you are interested.
In the meantime, this repository comes for non-commercial use with a CC BY-NC-SA 4.0 (Creative Commons) license.
This code was developed with Python3 (python3) with Cuda 11.3. All models were trained on a single NVIDIA RTX 3090 (or RTX 4090) GPU with 24GB of memory.
Installation steps:
-
We recommend using Docker to set up your environment for better reproducibility.
- To make it as easy as possible, we provide a Makefile that needs to be changed at three locations:
- l. 26: Change <USER_ID>:<GROUP_ID> to your user and group id.
- l. 33: Change <PATH_TO_DATAROOT> to your host path of the data folder.
- l. 34: Change <PATH_TO_MD4ALL> to your host path of the md4all code folder. Then you can run the commands below.
- Change the directory:
cd <PATH_TO_MD4ALL>
- Build the docker image:
make docker-build NAME=build
-
If you do not want to use Docker, here are the installation steps with Anaconda and pip:
- Create a conda environment:
conda create -n md4all python=<PYTHON_VERSION>
- Activate the environment:
conda activate md4all
- Change the directory:
cd <PATH_TO_MD4ALL>
- Install the requirements:
Or with specific package versions:
pip install -r requirements.txt
pip install -r requirements_w_version.txt
- Create a conda environment:
-
Download the nuScenes trainval dataset (v1.0) i.e. the 10 file blobs and the metadata from here (nuScenes). Optionally, you can also download the nuScenes test set from the same location.
-
Download the translated images and the 'train_samples_dynamic.json' file from here (our Google Form).
-
Set everything up such that your file structure looks similar to:
-
Download the recorded data of the left stereo camera and the front LMS laser sensor for the following scenes from here (RobotCar website):
- 2014/12/09 for day
- 2014/12/16 for night
-
Download the translated images, the computed poses, and the split files from here (our Google Form). The link takes to the same Google Form as for nuScenes, so if you already filled it up for nuScenes, no need to fill it up again for RobotCar as the download link is the same.
-
Download the RobotCar SDK from here (GitHub repo). The repository contains the extrinsics files.
-
Set everything up such that your file structure looks similar to:
-
Undistort and demoisaic the images from the left stereo camera (Attention: Using those commands will replace the original distorted and mosaiced images of the left stereo camera):
- Docker:
make docker-precompute-rgb-images-robotcar NAME=precompute-rgb-images-robotcar
- Conda:
python data/robotcar/precompute_rgb_images.py --dataroot <PATH_TO_DATAROOT> --scenes 2014-12-09-13-21-02 2014-12-16-18-44-24 --camera_sensor stereo/left --out_dir <PATH_TO_DATAROOT>
- Docker:
-
Precompute the ground truth depth data by projecting the point cloud of the lms front sensor to the images:
- Docker:
make docker-precompute-pointcloud-robotcar NAME=precompute-pointcloud-robotcar
- Conda:
python data/robotcar/precompute_depth_gt.py --dataroot <PATH_TO_DATAROOT> --scenes 2014-12-09-13-21-02 2014-12-16-18-44-24 --mode val test
- Docker:
We provide pre-trained models, namely md4allDD, the baseline used for knowledge distillation (for nuScenes not the same baseline as reported in Table 1 originating from an older code base version), and md2 for both nuScenes and RobotCar here (Google Drive). Download the files to the checkpoints folder. To evaluate the pre-trained models (associated with their respective .yaml config files), run the following commands:
-
nuScenes:
- Docker:
make docker-eval-md4allDDa-80m-nuscenes-val NAME=eval-md4allDDa-80m-nuscenes-val
- Conda:
python evaluation/evaluate_depth.py --config <PATH_TO_MD4ALL>/config/eval_md4allDDa_80m_nuscenes_val.yaml
- Docker:
-
RobotCar:
- Docker:
make docker-eval-md4allDDa-50m-robotcar-test NAME=eval-md4allDDa-50m-robotcar-test
- Conda:
python evaluation/evaluate_depth.py --config <PATH_TO_MD4ALL>/config/eval_md4allDDa_50m_robotcar_test.yaml
- Docker:
The provided models and configuration files lead to the results of the tables in our paper.
To train a model e.g., the baseline (associated with its .yaml config file), run:
-
nuScenes:
- Docker:
make docker-train-baseline-nuscenes NAME=train-baseline-nuscenes
- Conda:
python train.py --config <PATH_TO_MD4ALL>/config/train_baseline_nuscenes.yaml
- Docker:
-
RobotCar:
- Docker:
make docker-train-baseline-robotcar NAME=train-baseline-robotcar
- Conda:
python train.py --config <PATH_TO_MD4ALL>/config/train_baseline_robotcar.yaml
- Docker:
To predict the depth for custom images, you can use one of the commands below. Please remember that our models were trained on a single dataset so we provide no performance guarantees on the transfer to out-of-distribution data. This script is meant for simplifying quick tests.
-
nuScenes (using model trained on nuScenes):
- Docker (For Docker, you need to adapt the image path and output path written in the Makefile to customize the behavior of test_simple.py):
make docker-test-simple-md4allDDa-nuscenes NAME=test-simple-md4allDDa-nuscenes
- Conda:
python test_simple.py --config <PATH_TO_MD4ALL>/config/test_simple_md4allDDa_nuscenes.yaml --image_path <PATH_TO_MD4ALL>/resources/n015-2018-11-21-19-21-35+0800__CAM_FRONT__1542799608112460.jpg --output_path <PATH_TO_MD4ALL>/output
- Docker (For Docker, you need to adapt the image path and output path written in the Makefile to customize the behavior of test_simple.py):
-
RobotCar (using model trained on RobotCar):
- Docker (For Docker, you need to adapt the image path and output path written in the Makefile to customize the behavior of test_simple.py):
make docker-test-simple-md4allDDa-robotcar NAME=test-simple-md4allDDa-robotcar
- Conda:
python test_simple.py --config <PATH_TO_MD4ALL>/config/test_simple_md4allDDa_robotcar.yaml --image_path <PATH_TO_MD4ALL>/resources/1418756721422679.png --output_path <PATH_TO_MD4ALL>/output
- Docker (For Docker, you need to adapt the image path and output path written in the Makefile to customize the behavior of test_simple.py):
We provide the pre-trained ForkGAN models for both nuScenes (day-to-night, day-to-rain) and RobotCar (day-to-night) here (Google Drive). Download the ForkGAN folders (e.g. forkgan_nuscenes_day_night) to the checkpoints folder. To predict the translation for custom images, you can use one of the commands below. Please remember that our models were trained on a single dataset so we provide no performance guarantees on the transfer to out-of-distribution data. This script is meant for simplifying quick tests.
-
nuScenes day-to-night translations (using forkgan model trained on nuScenes day and night images):
- Docker (For Docker, you need to adapt the image path, checkpoint directory, and output path written in the Makefile to customize the behavior of translate_simple.py):
make docker-translate-simple-md4allDDa-nuscenes-day-night NAME=translate-simple-md4allDDa-nuscenes-day-night
- Conda:
python translate_simple.py --image_path <PATH_TO_MD4ALL>/resources/n008-2018-07-26-12-13-50-0400__CAM_FRONT__1532621809112404.jpg --checkpoint_dir <PATH_TO_MD4ALL>/checkpoints/forkgan_nuscenes_day_night --model_name forkgan_nuscenes_day_night --resize_height 320 --resize_width 576 --output_dir <PATH_TO_MD4ALL>/output
- Docker (For Docker, you need to adapt the image path, checkpoint directory, and output path written in the Makefile to customize the behavior of translate_simple.py):
-
nuScenes day-to-rain translations (using forkgan model trained on nuScenes clear and rainy day images):
- Docker (For Docker, you need to adapt the image path, checkpoint directory, and output path written in the Makefile to customize the behavior of translate_simple.py):
make docker-translate-simple-md4allDDa-nuscenes-day-rain NAME=translate-simple-md4allDDa-nuscenes-day-rain
- Conda:
python translate_simple.py --image_path <PATH_TO_MD4ALL>/resources/n008-2018-07-26-12-13-50-0400__CAM_FRONT__1532621809112404.jpg --checkpoint_dir <PATH_TO_MD4ALL>/checkpoints/forkgan_nuscenes_day_rain --model_name forkgan_nuscenes_day_rain --resize_height 320 --resize_width 576 --output_dir <PATH_TO_MD4ALL>/output
- Docker (For Docker, you need to adapt the image path, checkpoint directory, and output path written in the Makefile to customize the behavior of translate_simple.py):
-
RobotCar day-to-night translations (using forkgan model trained on RobotCar day and night images):
- Docker (For Docker, you need to adapt the image path, checkpoint directory and output path written in the Makefile to customize the behavior of translate_simple.py):
make docker-translate-simple-md4allDDa-robotcar-day-night NAME=translate-simple-md4allDDa-robotcar-day-night
- Conda:
python translate_simple.py --image_path <PATH_TO_MD4ALL>/resources/1418132504537582.png --checkpoint_dir <PATH_TO_MD4ALL>/checkpoints/forkgan_robotcar_day_night --model_name forkgan_robotcar_day_night --crop_height 768 --crop_width 1280 --resize_height 320 --resize_width 544 --output_dir <PATH_TO_MD4ALL>/output
- Docker (For Docker, you need to adapt the image path, checkpoint directory and output path written in the Makefile to customize the behavior of translate_simple.py):
- Permission Denied error when running docker without sudo: To resolve the problem follow the steps here (Docker docs).
- ModuleNotFoundError: no module named Python Error => Make sure to update your PYTHONPATH accordingly:
- Docker:
export PYTHONPATH="${PYTHONPATH}:/mnt/code/md4all"
- Conda:
export PYTHONPATH="${PYTHONPATH}:/path/to/md4all"
- Docker:
- FileNotFoundError: [Errno 2] No such file or directory: '<PATH_TO_RESOURCE>' => If you use Conda you have to adapt the paths to the model checkpoint, dataset etc. according to your file system (as they are configured for Docker).
If you find our code useful for your research, please cite:
@inproceedings{gasperini_morbitzer2023md4all,
title={Robust Monocular Depth Estimation under Challenging Conditions},
author={Gasperini, Stefano and Morbitzer, Nils and Jung, HyunJun and Navab, Nassir and Tombari, Federico},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
year={2023},
pages={8177-8186}
}
Our implementation is based on the PackNet-SfM repository (GitHub) and follows their code structure. It also incorporates parts of the Monodepth2 repository (GitHub).
To perform day-to-adverse image translations, we used a PyTorch implementation of ForkGAN (GitHub) (original implementation can be found here (GitHub)).
We want to thank the authors for their great contribution! :)