Model 5 seems to downscale by 4
Memnarch opened this issue · comments
Hi,
While I got good results, they did not seem to be quite as sharp as I expected them. When I tried to process a 136x264 Video, I suddenly got the error:
Traceback (most recent call last):
File "E:\Git\RivenTools\Reproduce.py", line 26, in <module>
video.output(f, y4m=True)
File "src\cython\vapoursynth.pyx", line 1790, in vapoursynth.VideoNode.output
File "src\cython\vapoursynth.pyx", line 1655, in frames
File "D:\Program Files\Python39\lib\concurrent\futures\_base.py", line 445, in result
return self.__get_result()
File "D:\Program Files\Python39\lib\concurrent\futures\_base.py", line 390, in __get_result
raise self._exception
vapoursynth.Error: The height and width of low-res inputs must be at least 64, but got 66 and 34.
But my Video is larger. Looking into it, executing Model 5 downscales the video. Or at least its internal data seems to be downscaled. While the returned clip reports the correct size, the frames seem to be lower. A quarter to be precise.
This seems to be the code causing it:
vs-basicvsrpp/vsbasicvsrpp/basicvsr_pp.py
Line 291 in da06646
And this is the script to reproduce it (It's a trimmed down version, that's why model 5 and 1 are executed directly after one another. My pipeline has some steps inbetween):
Reproduce.zip
If you need that specific video from my script, I can share that with you but would prefer to do that outside of this report, as it's a game asset.
Yes, it's by design for model 3-5 as the code you linked. That's why input resolution must be at least 256 and mod-4 for model 3-5.
I was not able to find that in the original mmediting-code.
Their documentation says
"is_low_res_input (bool, optional): Whether the input is low-resolution
or not. If False, the output resolution is equal to the input"
However in this case, the output-resolution is smaller than the input resolution. And the Vapoursynth api reports the wrong resolution (Reporting 136x264 while it's actually lower)
EDIT: Ok found it.
https://github.com/open-mmlab/mmediting/blob/23213c839ff2d1907a80d6ea29f13c32a24bb8ef/mmedit/models/backbones/sr_backbones/basicvsr_pp.py#L318
However, isn't the upsample pass at the end supposed to bring it back to the original resolution?
My biggest problem here is, that if I add a manual upscale by 4 using resize.Bicubic
, Vapoursynth would return an even higher frame size, wouldn't it?
EDIT2: Or am i supposed to do an upscale by 4 before running model 5 on it?
EDIT3: Ok no I get that mixed up. The outputsize is correct it seems. The 64 size check is after the downscale. Sorry for confusing it. So I need to upscale it appropiately before going through model 5 and just downscale it back to the original afterwards.