Jireh-Jam / Inpainting-Evaluation-Metrics

The goal of this repo is to provide a common evaluation script for image inpainting tasks. It contains some commonly used image quality metrics for inpainting (e.g., L1, L2, SSIM, PSNR and LPIPS).

I am not maintaining this repo anymore. Please refer to this repo for updated version. Thank you.

Inpainting Evaluation Metrics (On-going)

The goal of this repo is to provide a common evaluation script for image inpainting tasks. It contains some commonly used image quality metrics for inpainting (e.g., L1, L2, SSIM, PSNR and LPIPS).

Pull requests and corrections/suggestions will be cordially appreciated.

Please Note

• Images are scaled to [0,1]. If you need to change the data range, please make sure to change the data range in SSIM and PSNR.
• Number of generated images and ground truth images have to be exactly same.
• I have resized the images to be (256,256). You can change the resolution based on your needs.
• Please make sure that all the images (generated and gt images) are in the corresponding folders. Currently,it can not calculate metrics if there are sub-folders. I will update the code to calculate for sub-folders as well.
• LPIPS is a bit slow. So, if you have lots of images, it might take a lot of time. (For ~1000 images, it took around ~15-20 minutes on my personal setup (1 TitanXP). Other metrics are fast and took around ~20 seconds to compute.)

Requirements

• PyTorch ( >= 1.0 )
• Python ( >=3.5 )
• PyTorch Image Quality (PIQ) ( $ pip install piq )

Usage

• Usable Arguments

  • --input_path - path to your generated images (required).
  • --gt_path - path to your ground truth images (required).
  • --batch_size - batch size you want to use (Default to 4).
  • --image_width - width of the image (both generated image and ground truth images will be resized to this width. Default to 256).
  • --image_height - width of the image (both generated image and ground truth images will be resized to this width. Default to 256).
  • --threads - threads to be used for multi-processing (Default to 4).

• Please provide paths of the folders (i.e., folder of generated images and folder of ground truth images).

  python main.py --input_path path/to/generated/images --gt_path path/to/ground/truth/images

• If you need to save it in a .txt file, then simply run

  python main.py --input_path path/to/generated/images --gt_path path/to/ground/truth/images >> results.txt

To-do

• L1
• L2
• SSIM
• PSNR
• LPIPS
• FID
• IS

Acknowledgement

Thanks to PhotoSynthesis Team for the wonderful implementation of the metrics. Please cite accordingly if you use PIQ for the evaluation.

Cheers!!