1canapecoo / DC-ShadowNet-Hard-and-Soft-Shadow-Removal

[ICCV2021]DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network, https://arxiv.org/pdf/2207.10434.pdf

DC-ShadowNet (ICCV'2021)

Introduction

This is an implementation of the following paper.

DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network. International Conference on Computer Vision (ICCV'2021)

Yeying Jin, Aashish Sharma and Robby T. Tan

[Paper Link] [Supplement] [Slides] [Video]

Abstract

Shadow removal from a single image is generally still an open problem. Most existing learning-based methods use supervised learning and require a large number of paired images (shadow and corresponding non-shadow images) for training. A recent unsupervised method, Mask-ShadowGAN, addresses this limitation. However, it requires a binary mask to represent shadow regions, making it inapplicable to soft shadows. To address the problem, in this paper, we propose an unsupervised domain-classifier guided shadow removal network, DC-ShadowNet. Specifically, we propose to integrate a shadow/shadow-free domain classifier into a generator and its discriminator, enabling them to focus on shadow regions. To train our network, we introduce novel losses based on physics-based shadow-free chromaticity, shadow-robust perceptual features, and boundary smoothness. Moreover, we show that our unsupervised network can be used for test-time training that further improves the results. Our experiments show that all these novel components allow our method to handle soft shadows, and also to perform better on hard shadows both quantitatively and qualitatively than the existing state-of-the-art shadow removal methods.

Prerequisites

conda env create -f shadow_env.yml

Datasets

1. SRD (please download train BaiduNetdisk and test from the authors). Extracted Shadow Masks in the SRD Dataset

2. AISTD

3. LRSS: Soft Shadow Dataset

4. ISTD

5. USR: Unpaired Shadow Removal Dataset

Shadow Removal Results:

1. SRD Dataset Results:

DC-ShadowNet-SRD, SRD-Results

2. AISTD/ISTD+ Dataset Results:

DC-ShadowNet-AISTD, AISTD-Results

3. LRSS Soft Shadow Dataset Results:

DC-ShadowNet-LRSS, LRSS-Results

4. ISTD Dataset Results:

DC-ShadowNet-ISTD, ISTD-Results

5. USR Dataset Results:

DC-ShadowNet-USR, USR-Results

Evaluation

The default root mean squared error (RMSE) evaluation code used by all methods (including ours) actually computes mean absolute error (MAE).

1. The faster version MAE evaluation code
2. The original version MAE evaluation code

1. SRD Dataset Evaluation

set the paths of the shadow removal result and the dataset in demo_srd_release.m and then run it.

demo_srd_release.m

Get the following Table 1 in the main paper on the SRD (size: 256x256).

Method	Training	All	Shadow	Non-Shadow
DC-ShadowNet	Unpaired	4.66	7.70	3.39
Input Image	N/A	13.77	37.40	3.96

For SRD (size: 640x840)

Method	Training	All	Shadow	Non-Shadow
DC-ShadowNet	Unpaired	6.57	9.84	5.52

2. AISTD Dataset Evaluation

set the paths of the shadow removal result and the dataset in demo_aistd_release.m and then run it.

demo_aistd_release.m

Get the following Table 2 in the main paper on the AISTD (size: 256x256).

Method	Training	All	Shadow	Non-Shadow
DC-ShadowNet	Unpaired	4.6	10.3	3.5

For AISTD (size: 480x640)

Method	Training	All	Shadow	Non-Shadow
DC-ShadowNet	Unpaired	6.33	11.37	5.38

3. LRSS Soft Shadow Dataset Evaluation

set the paths of the shadow removal result and the dataset in demo_lrss_release.m and then run it.

demo_lrss_release.m

Get the following Table 3 in the main paper on the LRSS dataset (size: 256x256).

Method	Training	All
DC-ShadowNet	Unpaired	3.48
Input Image	N/A	12.26

Pre-trained Model

1. Download the pre-trained SRD model, put in results/SRD/model/

2. Download the pre-trained AISTD model, put in results/AISTD/model/

3. Download the pre-trained ISTD model, put in results/ISTD/model/

Test

python main_test.py --dataset SRD --datasetpath [path_to_SRD dataset] --use_original_name False

rename to the original name, please change the suffix of test images accordingly (.jpg or .png)

python main_test.py --dataset SRD --datasetpath [path_to_SRD dataset] --use_original_name True --im_suf_A .jpg

Results in: results/SRD/[iteration]/outputB; results/SRD/[iteration]/inputA_outputB

Train

1. Implement the papers On the removal of shadows from images (TPAMI,05) and Recovery of Chromaticity Image Free from Shadows via Illumination Invariance (ICCV,03)

2. Download Datasets and run 1. get the Shadow-Free Chromaticity Maps after Illumination Compensation, and put them in the trainC folder, you should see the following directory structure.

${DC-ShadowNet-Hard-and-Soft-Shadow-Removal}
|-- dataset
    |-- SRD
      |-- trainA ## Shadow 
      |-- trainB ## Shadow-free 
      |-- trainC ## Shadow-Free Chromaticity Maps after Illumination Compensation
      |-- testA  ## Shadow 
      |-- testB  ## Shadow-free 
...

3. python main_train.py --dataset SRD --datasetpath [path_to_SRD dataset] --iteration [iteration]

Shadow-Robust Feature

Get the following Figure 5 in the main paper, VGG feature visualization code is in feature_release folder,

python test_VGGfeatures.py

Results in: ./results_VGGfeatures/shadow_VGGfeatures/layernumber/imagenumber/visual_featurenumber_RMSE.jpg

Link

Zhihu Analysis

Citation

If this work is useful for your research, please cite our paper.

@inproceedings{jin2021dc,
  title={DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network},
  author={Jin, Yeying and Sharma, Aashish and Tan, Robby T},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={5027--5036},
  year={2021}
}