grishavak / PaCNet-denoiser

Patch Craft: Video Denoising by Deep Modeling and Patch Matching (ICCV 2021)

PaCNet

Arxiv | CVF (pdf) | CVF (suppl)

Official PyTorch implementation of the paper: "Patch Craft: Video Denoising by Deep Modeling and Patch Matching."

ICCV 2021

Overview

PaCNet (Patch-Craft Network) is a video and image denoiser.
The link to the presentation of the paper is here.

                  noisy with σ = 40                                 Filered with PaCNet                                       Original

Quantitative Results

Our S-CNN network works in a sliding window manner. In order to produce a reconstructed frame it loads seven noisy frames in the input – the corresponding noisy frame and three frames before and after it. In such a way, any frame enters the S-CNN network seven times, once as a processed frame and six more times as an auxiliary frame used for the nearest neighbor search. In simulations reported in the paper, we had a mistake in adding the noise. Instead of injecting noise to the whole video sequence before the processing, we added independent noise to each group of seven consecutive frames (after loading them from memory). In this scenario, each frame enters the network with seven different instantiations of noise, which makes the denoising process somewhat easier. We fixed this bug in the code, and that led to a small degradation of the results. The following tables contain both results: the one (too high) reported in the article and the more recent and correct one. In this repository, we publish both versions of the code: the new code after the bug fix and the old version that achieves a slightly higher PSNR.

Gaussian denoising results on DAVIS test set

Noise Level	V-BM4D 1	VNLB 2	VNLnet 3	DVDnet 4	FastDVDnet 5 6	PaCNet (reported in the paper)	PaCNet (correct result)
10	37.58	38.85	35.83	38.13	38.93	39.97	39.97
20	33.88	35.68	34.49	35.70	35.88	37.10	36.82
30	31.65	33.73	-7	34.08	34.12	35.07	34.79
40	30.05	32.32	32.32	32.86	32.87	33.57	33.34
50	28.80	31.13	31.43	31.85	31.90	32.39	32.20
Average	32.39	34.34	-	34.52	34.74	35.62	35.42

Best PSNR marked in bold.

Denoising with clipped Gaussian noise on DAVIS test set

Method	10	30	50	Average
ViDeNN 8	37.13	32.24	29.77	33.05
FastDVDnet5 6	38.65	33.59	31.28	34.51
PaCNet (reported in the paper)	40.13	34.92	32.15	35.73
PaCNet (correct result)	39.96	34.66	32.00	35.54

Best PSNR marked in bold.

Gaussian denoising results on Set8

Method	10	20	30	40	50	Average
DVDnet 4	36.08	33.49	31.79	30.55	29.56	32.29
FastDVDnet 6	36.44	33.43	31.68	30.46	29.53	32.31
PaCNet (ours)	37.06	33.94	32.05	30.70	29.66	32.68

Qualitative Results

          clean               noisy with σ = 40           VNLB                     VNLnet                FastDVDnet           PaCNet (ours)
                                                                PSNR = 28.66dB     PSNR = 29.03dB   PSNR = 29.27dB   PSNR = 27.67dB

          clean               noisy with σ = 40           VNLB                     VNLnet                FastDVDnet           PaCNet (ours)
                                                                PSNR = 27.92dB     PSNR = 27.95dB   PSNR = 28.23dB   PSNR = 28.88dB

PaCNet as Single Image Denoiser

PaCNet can be used as a single image denoiser. The following table reports the denoising performance of PaCNet, comparing it with the most similar image denoiser LIDIA ⁹.

Gaussian denoising results on BSD68 test set

Method	15	25	50	Average
LIDIA 9	34.03	31.31	27.99	31.11
PaCNet (ours)	33.95	31.22	27.93	31.03

Code

This code was tested with python 3.9, cuda 11.4 and pytorch 1.9 on GeForse GTX 1080 Ti and Quadro RTX 8000 (for video denoising with option gpu_usage=2).

Requirements

• numpy
• opencv
• matplotlib
• torch

Install dependencies:

python -m pip install -r code/requirements.txt

Video Denoising

Denoise a video sequence

python code/process_video_sequence.py --in_folder <in_folder> --file_ext <file_ext> --sigma <sigma> --clipped_noise <clipped_noise> [--save_jpg] --jpg_out_folder <jpg_out_folder> [--save_avi] --avi_out_folder <avi_out_folder> --gpu_usage <gpu_usage> [--plot]

Parameters

• in_folder - path to a test video sequence. Default: ./data_set/davis/horsejump-stick/
• file_ext - file extension: {jpg, png}. Default: jpg
• sigma - noise sigma.
  For AWGN: {10, 20, 30, 40, 50}.
  For clipped Gaussian noise: {10, 30, 50}
  Default: 20
• clipped_noise - noise type: {0 - AWGN, 1 - clipped Gaussian}
• [--save_jpg] (flag) - save the denoised video as JPG frames
• jpg_out_folder - path to folder for saving JPG frames. Default: ./output/videos/jpg_sequences/demo/
• [--save_avi] (flag) - save the denoised video as AVI file
• avi_out_folder - path to folder for saving AVI file. Default: ./output/videos/avi_files/demo/
• gpu_usage - GPU usage:
  0 - use CPU,
  1 - use GPU for nearest neighbor search,
  2 - use GPU for whole processing, requires large GPU memory (about 20-30GB))
• [--plot] (flag) - plot a frame from the processed video sequence

For demo run

python code/process_video_sequence.py --gpu_usage 1 --save_jpg --save_avi

For denoising a video sequence in your code call denoise_video_sequence function

denoise_video_sequence(<noisy_vid>, <vid_name>, <sigma>, [clipped_noise], [gpu_usage], [silent])

Parameters

• noisy_vid (required) - noisy video sequence
• vid_name (required) - video name
• sigma (required) - noise sigma.
  For AWGN: {10, 20, 30, 40, 50}
  For Clipped Gaussian noise: {10, 30, 50}
• clipped_noise (optional) - 0 - AWGN, 1 - clipped Gaussian noise
• gpu_usage (optional) - GPU usage:
  0 - use CPU,
  1 - use GPU for nearest neighbor search,
  2 - use GPU for whole processing, requires large GPU memory (about 20-30GB)

Outputs:

• denoised_vid - denoised video sequence
• denoising_time - denoising time

Denoise a video set

python code/process_video_sequence.py --in_folder <in_folder> --file_ext <file_ext> --sigma <sigma> --clipped_noise <clipped_noise> [--save_jpg] --jpg_out_folder <jpg_out_folder> [--save_avi] --avi_out_folder <avi_out_folder> --gpu_usage <gpu_usage> --max_frame_num <max_frame_num> [--silent]

Parameters

• in_folder - path to a video set. Default: ./data_set/davis/
• file_ext - file extension: {jpg, png}. Default: jpg
• sigma - noise sigma.
  For AWGN: {10, 20, 30, 40, 50}.
  For clipped Gaussian noise: {10, 30, 50}
  Default: 20
• clipped_noise - noise type: {0 - AWGN, 1 - clipped Gaussian}
• [--save_jpg] (flag) - save the denoised videos as JPG frames
• jpg_out_folder - path to folder for saving JPG frames. Default: ./output/videos/jpg_sequences/set/
• [--save_avi] (flag) - save the denoised videos as AVI files
• avi_out_folder - path to folder for saving AVI file. Default: ./output/videos/avi_files/set/
• gpu_usage - GPU usage:
  0 - use CPU,
  1 - use GPU for nearest neighbor search,
  2 - use GPU for whole processing, requires large GPU memory (about 20-30GB)
• max_frame_num - limit for maximum number of denoised frames in each video sequence in the set. Default: 85.
• [--silent] (flag) - don't print "done" every frame.

For denoising DAVIS test set run

python code/process_video_set.py --sigma <sigma> --gpu_usage 1 --save_jpg --save_avi

For denoising Set8 test set run

python code/process_video_set.py --sigma <sigma> --in_folder "./data_set/set8/" --file_ext png --gpu_usage 1 --save_jpg --save_avi

For reproducing the results with the bug during adding noise (higher PSNR that is reported in the paper), run

python code/process_video_set_with_bug.py --sigma <sigma> --gpu_usage 1 --save_jpg --save_avi

Image Denoising

Denoise an image

python code/process_image.py --in_folder <in_folder> --im_name <im_name> --out_folder <out_folder> --sigma <sigma> --gpu_usage <gpu_usage>  [--plot] [--save]

Parameters

• in_folder - input folder. Default: ./data_set/cbsd68/
• im_name - image name. Default: 119082.png
• sigma - noise sigma: {15, 25, 50}. Default: 25
• [--save] (flag) - save the denoised image
• out_folder - output foldeer. Default: ./output/images/demo/
• gpu_usage - GPU usage:
  0 - use CPU,
  1 - use GPU for nearest neighbor search,
  2 - use GPU for whole processing (requires about 10GB GPU memory)
• [--plot] (flag) - plot a frame from the processed video sequence

For demo run

python code/process_image.py --gpu_usage 1 --save

For denoising an image in your code call denoise_image function

denoise_image(<noisy_im>, <sigma>, [gpu_usage])

Parameters

• noisy_im (required) - noisy image
• sigma (required) - noise sigma: {15, 25, 50}
• gpu_usage (optional) - GPU usage:
  0 - use CPU,
  1 - use GPU for nearest neighbor search,
  2 - use GPU for whole processing (requires about 10GB GPU memory)

Outputs:

• denoised_im - denoised image
• denoising_time - denoising time

Denoise an image set

python code/process_image_set.py --in_folder <in_folder> --out_folder <out_folder> --sigma <sigma> --gpu_usage <gpu_usage> [--save]

Parameters

• in_folder - input folder. Default: ./data_set/cbsd68/
• sigma - noise sigma: {15, 25, 50}. Default: 25
• [--save] (flag) - save the denoised images
• out_folder - path to folder for saving denoised images. Default: ./output/images/set/
• gpu_usage - GPU usage:
  0 - use CPU,
  1 - use GPU for nearest neighbor search,
  2 - use GPU for whole processing (requires about 10GB GPU memory)

For denoising BSD68 test set run

python code/process_image_set.py --gpu_usage 1 --save

Citation

If you use this code for your research, please cite our paper:

@InProceedings{Vaksman_2021_ICCV,
    author    = {Vaksman, Gregory and Elad, Michael and Milanfar, Peyman},
    title     = {Patch Craft: Video Denoising by Deep Modeling and Patch Matching},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {2157-2166}
}

References and Footnotes

Footnotes

1. Matteo Maggioni, Giacomo Boracchi, Alessandro Foi, and Karen Egiazarian. Video denoising using separable 4-D nonlocal spatiotemporal transforms. ↩

2. Pablo Arias and Jean-Michel Morel. Video denoising via empirical bayesian estimation of space-time patches. ↩

3. Axel Davy, Thibaud Ehret, Jean-Michel Morel, Pablo Arias, and Gabriele Facciolo. A non-local cnn for video denoising. ↩ ↩² ↩³

4. Matias Tassano, Julie Delon, and Thomas Veit. Dvdnet: A fast network for deep video denoising. ↩ ↩²

5. FastDVDnet ⁶ PSNR values are obtained from the released code. The rest of the values reported in the tables are taken from ⁶. ↩ ↩²

6. Matias Tassano, Julie Delon, and Thomas Veit. Fastdvdnet: Towards real-time deep video denoising without flow estimation. ↩ ↩² ↩³ ↩⁴ ↩⁵

7. The PSNR value for VNLnet ³ with σ = 30 is missing as ³ did not provide a model for this noise level. ↩

8. Michele Claus and Jan van Gemert. Videnn: Deep blind video denoising. ↩

9. Gregory Vaksman, Michael Elad, and Peyman Milanfar. Lidia: Lightweight learned image denoising with instance adaptation. ↩ ↩²