xiaofeng94 / BeDDE-for-defogging

BeDDE, a real-world benchmark dataset for dehazing, for "Dehazing Evaluation: Real-world Benchmark Datasets, New Criteria and Baselines" (TIP 2020)

BeDDE & exBeDDE for dehazing evaluation

BeDDE

Reference image	Light	Light	Medium	Heavy

BeDDE (read as /ˈbedi/) is a real-world benchmark dataset for evaluations of dehazing methods. It consists of 208 pairs of hazy images and clear refernece images. For each pair, a manually labelled mask is provided to delineate regions with the same contents. We evaluate dehazing results on those regions.

exBeDDE

exBeDDE is an extension of BeDDE, designed to measure the performance of dehazing evaluation metrics. It contains 167 hazy images and 1670 dehazed images with mean opinion scores labeled by people. Its hazy images come from BeDDE, and the dehazed images are generated by 10 dehazing methods.

Visibility Index (VI) and Realness Index (RI)

We find it is more reasonable to evaluate dehazed reuslts from two separate aspects, i.e., visibility and realness, and acorrdingly propose two criteria, i.e., visibility index (VI) and realness index (RI). More details can be found in the paper (or this link) titled as Dehazing Evaluation: Real-World Benchmark Datasets, New Criteria and Baselines. (Presented in Trans. Image Process.)

Download

You can download BeDDE and exBeDDE on Google drive.

You may also get access to the two datasets (key: m53p) on BaiduYun disk.

Testing

Our environment

• Matlab 2017b

Test on BeDDE

1. Download BeDDE.rar and unzip it to ./Defogging_eval

2. Set Matlab work folder to ./Defogging_eval

3. Run eval_defog_method.m

Then, you will see the VI score of hazy images. You may modify the variables to try other dehazing methods or metrics. Moreover, the score for each image will be stored in ./statistics/<your_method_name>_<metric>_eval.mat.

4. Run eval_defog_method_stat_by_level.m to get evaluations on level.

This script requires the stored scores generated in last step, but you may use the example scores for RefineDNet. For your case, you should modify variables method_names and metric_name.

The difficulty levels are stored in ./fog_level_labels/fog_level.mat.

Test on exBeDDE

1. Download exBeDDE.rar and unzip it to ./Defogging_eval

2. Set Matlab work folder to ./Defogging_eval

3. Run assess_IQA_metric.m

It will take a while. After that, you will see the performance of our VI on hazy groups. You may modify the variables to test on dehazed groups or assess other metrics.

Test your own dehazing method on BeDDE

1. Create a folder named after your method in each city folder of BeDDE. Put the images results of each city to the corresponding folder you just created. Name all the dehazed images after their original hazy images or make sure the name of a dehazed image starts with the name of its original hazy image. Your directory tree may look like,

BeDDE
├── beijing
│   ├── fog
│   │   ├── beijing_1.png
│   │   ├── beijing_2.png
│   │   ...
│   │   └── beijing_15.png
│   ├── gt
│   ├── mask
│   ├── <your_method_name>
│   │   ├── beijing_1_<your_method_name>.png
│   │   ├── beijing_2_<your_method_name>.png
│   │   ...
│   │   └── beijing_15_<your_method_name>.png
├── changsha
│   ...
├── chengdu
│   ...
...

2. Set the variable method_name in eval_defog_method.m to the name of your method.

3. Set the variable eval_method to VI, RI, VSI or other metrics and then run the script to get the score for your method on BeDDE.

Benchmarks

Dehazing methods

All methods were evaluated on BeDDE. VI shows their abilities to restore visibility, RI and LPIPS refer to realness of the results.

Dehazing metrics

All metrics were assessed on exBeDDE's hazy groups for the visibility evaluation and on dehazing groups for the realness evaluation.

Visibility evaluation	Realness evaluation

Dehazing methods

The selected 14 dehazing methods adopted by exBeDDE are Fast Visibility Restoration (FVR), Dark Channel Prior (DCP), Bayesian Defogging (BayD), Color Attenuation Prior (CAP), Non-Local image Dehazing (NLD), MSCNN, DehazeNet, AOD-Net, DCPDN, GFN, DisentGAN, Learning a patch quality comparator (PQC), EPDN, and GridDehazeNet.

We used offical code for all dehazing methods (except DCP). Please try the code link for those methods listed as follows: FVR (Bottom), DCP, CAP, NLD, MSCNN, DehazeNet, AOD-Net, DCPDN (Also try this code with less issues to fix), GFN, DisentGAN, PQC, EPDN, GridDehazeNet

References

If our datasets and criteria are helpful, please consider citing the following papers. [1] and [2] for BeDDE. [2] for exBeDDE, VI, and RI.

[1] S. Zhao, L. Zhang, et al. Evaluation of defogging: A real-world benchmark dataset, a new criterion and baselines. In ICME, pp.1840-1845, 2019.

[2] S. Zhao, L. Zhang, et al. Dehazing Evaluation: Real-world Benchmark Datasets, Criteria and Baselines. IEEE Trans. Image Process., 28(1):6947-6962, 2020.