Can't reproduce HM benchmark, lower than reported.
jiehuang165 opened this issue · comments
jiehuang165 commented
Bug
Try to reproduce the HM benchmark, but "psnr-rgb" is quite lower than reported.
To Reproduce
Steps to reproduce the behavior:
- git clone this repo
- dowload and compile HM, the version is 'HM-16.20+SCM-8.8' same as
CompressAI/examples/run-benchmarks.sh
Line 88 in a4ae2ee
- run benchmark script
./examples/run-benchmarks.sh ~/dataset/kodak hm
Expected behavior
Expect to reproduct kodak/hm.json
However, I get following results:
{
"name": "my-HM",
"description": "HM",
"results": {
"bpp": [
3.845564948187935,
2.4760615030924478,
1.559600830078125,
0.9503741794162326,
0.5295282999674479,
0.26666259765625,
0.12086995442708333,
0.05143822564019097
],
"encoding_time": [
14.452731589476267,
10.646501451730728,
14.093859483798346,
12.758554518222809,
11.680685818195343,
12.660903205474218,
8.8968399365743,
8.146473407745361
],
"decoding_time": [
0.7784863809744517,
0.3728034794330597,
0.3049040138721466,
0.38336031635602313,
0.4734913905461629,
0.19524875283241272,
0.3037620981534322,
0.24185975392659506
],
"psnr-rgb": [
42.922397886745436,
41.28368055742206,
39.05052577559535,
36.308070045031265,
33.27506657838453,
30.333590592627463,
27.68549982125182,
25.38004455197402
],
"ms-ssim-rgb": [
0.9981328224142393,
0.9964238107204437,
0.9932830135027567,
0.9872708593805631,
0.974710909028848,
0.949935294687748,
0.9075854445497195,
0.8457145740588506
]
}
}
After plotting bitrate-psnr curve of my reproduce and official results, the difference is obvious especially when bitrate is large.
Environment
- PyTorch / CompressAI Version (1.10.1+cu111 / 1.2.4.dev0):
- OS (e.g., Linux): Ubuntu 20.04.3
- How you installed PyTorch / CompressAI (`pip`, source): pip install -e .
- Build command you used (if compiling from source):
- Python version: 3.8
- CUDA/cuDNN version: 11.1