PHDNet: Painterly Image Harmonization in Dual Domains

This is the official repository for the following paper:

Painterly Image Harmonization in Dual Domains [arXiv]

Junyan Cao, Yan Hong, Li Niu
Accepted by AAAI 2023.

Painterly image harmonization aims to adjust the foreground style of the painterly composite image to make it compatible with the background. A painterly composite image contains a photographic foreground object and a painterly background image.

Datesets

Paniterly image harmonization requires two types of images: photographic image and painterly image. We cut a certain object from a photographic image by the corresponding instance mask, and then paste it onto a painterly image, generating a composite image.

Photographic image

We apply images from COCO to produce the foreground objects. For each image, We select the object with foreground ratio in [0.05, 0.3] and generate the forefround mask. The selected foreground masks are provided in selected_masks (access code: ww1t). The training set can be downloaded from COCO_train (alternative: Baidu Cloud (access code: nfsh)) and the test set from COCO_test (alternative: Baidu Cloud (access code: nsvj)).

Painterly image

We apply images from WikiArt to be the backgrounds. The dataset can be downloaded from WikiArt (access code: sc0c). The training/test data list are provided in wikiart_split.

The example dataset dirs:

your_dir
│
└───MS-COCO
│   └───SegmentationClass_select
│   │   │   XXX.png
│   │   │  ...
│   │   
│   └───train2014
│   │   │   XXX.jpg
│   │   │  ...
│   │   
│   └───val2014
│       │   XXX.jpg
│       │  ...
│   
└───wikiart
    └───WikiArt_Split
    │   │   style_class.txt
    │   │   style_train.csv
    │   │   style_val.csv
    │       
    └───unzipped_subfolders

Prerequisites

Linux
Python 3
PyTorch 1.10
NVIDIA GPU + CUDA

Getting Started

Installation

Clone this repo:

git clone https://github.com/bcmi/PHDNet-Painterly-Image-Harmonization.git
# cd to this repo's root dir

Prepare the datasets.
Install PyTorch and dependencies from http://pytorch.org.
Install python requirements:

pip install -r requirements.txt

Download pre-trained VGG19 from Baidu Cloud (access code: pc9y).

PHDNet train/test

Train PHDNet:

cd PHDNet/scripts
bash train_phd.sh

The trained model would be saved under ./<checkpoint_dir>/<name>/.

If you want to load a model then continue to train it, add --continue_train and set the --epoch XX in train_phd.sh. It would load the model ./<checkpoint_dir>/<name>/net_G_<epoch>.pth.

Remember to modify the content_dir and style_dir to the corresponding path of each dataset in train_phd.sh.

Test PHDNet:

cd PHDNet/scripts
bash test_phd.sh

It would load the model ./<checkpoint_dir>/<name>/net_G_<epoch>.pth then save the visualization under ./<checkpoint_dir>/<name>/web/TestImages/

Our pre-trained model is available on Baidu Cloud (access code: po7q).

Note: <...> means modifiable parameters.

ironheads / PHDNet-Painterly-Image-Harmonization