Python error using docker

Question

Python error using docker

AlverGant opened this issue 3 years ago · comments

Alvaro Lopez Antelo commented 3 years ago

First of all I want to congratulate for your outstanding results on IEEE CLIC Challenge! Impressive work using machine learning with video compression.

I tried to run the code using docker as per instructions on README but stumbled upon this error using the sanity script:

root@eb31c26fd94a:/home/ubuntu/AIVC/src# ./sanity_script.sh
********************************************************************************
Starting encoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
[INFO] | Seed | | 666
[W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
Traceback (most recent call last):
File "encode.py", line 103, in
model = load_model(prefix='0_', on_cpu=True).to(COMPUTE_PARAM.get('device')).eval()
File "/home/ubuntu/AIVC/src/model_mngt/model_management.py", line 363, in load_model
model = torch.load('./' + prefix + 'model.pt', map_location=map_loc)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 595, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 764, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
pickle.UnpicklingError: invalid load key, 'v'.
********************************************************************************
Starting decoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
Traceback (most recent call last):
File "decode.py", line 56, in
model = load_model(prefix='0', on_cpu=True).to(COMPUTE_PARAM.get('device')).eval()
File "/home/ubuntu/AIVC/src/model_mngt/model_management.py", line 363, in load_model
model = torch.load('./' + prefix + 'model.pt', map_location=map_loc)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 595, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 764, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
pickle.UnpicklingError: invalid load key, 'v'.
********************************************************************************
Starting evaluation
/home/ubuntu/AIVC/src/clic21/metrics.py:50: RuntimeWarning: invalid value encountered in double_scalars
results['PSNR'] = mse2psnr(np.sum(sqerror_values) / num_dims)
/home/ubuntu/AIVC/src/clic21/metrics.py:52: RuntimeWarning: invalid value encountered in double_scalars
results['MSSSIM'] = np.sum(msssim_values) / num_dims
PSNR [dB]: nan
MS-SSIM : nan
MS-SSIM [dB]: nan
[ERROR]: bitstream not found, can not evaluate its size!
Bistream path: ../bitstream.bin
Traceback (most recent call last):
File "evaluate.py", line 78, in
print('Size [bytes]: ' + '%.0f' % (size_bytes))
NameError: name 'size_bytes' is not defined
root@eb31c26fd94a:/home/ubuntu/AIVC/src# ./sanity_script.sh
********************************************************************************
Starting encoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
[INFO] | Seed | | 666
[W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
Traceback (most recent call last):
File "encode.py", line 103, in
model = load_model(prefix='0', on_cpu=True).to(COMPUTE_PARAM.get('device')).eval()
File "/home/ubuntu/AIVC/src/model_mngt/model_management.py", line 363, in load_model
model = torch.load('./' + prefix + 'model.pt', map_location=map_loc)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 595, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 764, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
pickle.UnpicklingError: invalid load key, 'v'.
********************************************************************************
Starting decoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
Traceback (most recent call last):
File "decode.py", line 56, in
model = load_model(prefix='0', on_cpu=True).to(COMPUTE_PARAM.get('device')).eval()
File "/home/ubuntu/AIVC/src/model_mngt/model_management.py", line 363, in load_model
model = torch.load('./' + prefix + 'model.pt', map_location=map_loc)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 595, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 764, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.
********************************************************************************
Starting evaluation
/home/ubuntu/AIVC/src/clic21/metrics.py:50: RuntimeWarning: invalid value encountered in double_scalars
results['PSNR'] = mse2psnr(np.sum(sqerror_values) / num_dims)
/home/ubuntu/AIVC/src/clic21/metrics.py:52: RuntimeWarning: invalid value encountered in double_scalars
results['MSSSIM'] = np.sum(msssim_values) / num_dims
PSNR [dB]: nan
MS-SSIM : nan
MS-SSIM [dB]: nan
[ERROR]: bitstream not found, can not evaluate its size!
Bistream path: ../bitstream.bin
Traceback (most recent call last):
File "evaluate.py", line 78, in
print('Size [bytes]: ' + '%.0f' % (size_bytes))
NameError: name 'size_bytes' is not defined

Alvaro Lopez Antelo · Answer 1 · Thu Dec 02 2021 02:08:16 GMT+0800 (China Standard Time)

OK, managed to correct this error, we need to install and initialize git lfs first otherwise git clone does not download the models correctly:
sudo apt-get install git-lfs
git lfs install

Alvaro Lopez Antelo · Answer 2 · Thu Dec 02 2021 02:09:01 GMT+0800 (China Standard Time)

Now stuck with the following error:

root@369ba43657ce:/home/ubuntu/AIVC/src# ./sanity_script.sh
********************************************************************************
Starting encoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/torchac_backend...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF torchac_backend.o.d -DTORCH_EXTENSION_NAME=torchac_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /opt/conda/lib/python3.8/site-packages/torch/include -isystem /opt/conda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.8/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.8/site-packages/torch/include/THC -isystem /opt/conda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /opt/conda/lib/python3.8/site-packages/torchac/backend/torchac_backend.cpp -o torchac_backend.o
[2/2] c++ torchac_backend.o -shared -L/opt/conda/lib/python3.8/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o torchac_backend.so
Loading extension module torchac_backend...
[INFO] | Seed | | 666
[W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py:434: UserWarning: Setting attributes on ParameterList is not supported.
warnings.warn("Setting attributes on ParameterList is not supported.")
Traceback (most recent call last):
File "encode.py", line 109, in
encode({
File "/home/ubuntu/AIVC/src/real_life/encode.py", line 77, in encode
max_index = max(png_idx)
ValueError: max() arg is an empty sequence
********************************************************************************
Starting decoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py:434: UserWarning: Setting attributes on ParameterList is not supported.
warnings.warn("Setting attributes on ParameterList is not supported.")
[INFO] | | | Found CUDA device
[INFO] | Seed | | 666
[W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
[INFO] | Bitstream path | | ../bitstream.bin
[INFO] | Bitstream processing dir. | | ../tmp/3697412327659087746_0/tmp_out_bitstream/
[INFO] | Decoded frames directory | | ../compressed/
Traceback (most recent call last):
File "decode.py", line 83, in
decode_one_video({
File "/home/ubuntu/AIVC/src/real_life/decode.py", line 89, in decode_one_video
uncat_one_video({'video_file': bitstream_path, 'out_dir': bitstream_dir})
File "/home/ubuntu/AIVC/src/real_life/decode.py", line 402, in uncat_one_video
with open(video_file, 'rb') as fin:
FileNotFoundError: [Errno 2] No such file or directory: '../bitstream.bin'
********************************************************************************
Starting evaluation
/home/ubuntu/AIVC/src/clic21/metrics.py:50: RuntimeWarning: invalid value encountered in double_scalars
results['PSNR'] = mse2psnr(np.sum(sqerror_values) / num_dims)
/home/ubuntu/AIVC/src/clic21/metrics.py:52: RuntimeWarning: invalid value encountered in double_scalars
results['MSSSIM'] = np.sum(msssim_values) / num_dims
PSNR [dB]: nan
MS-SSIM : nan
MS-SSIM [dB]: nan
[ERROR]: bitstream not found, can not evaluate its size!
Bistream path: ../bitstream.bin
Traceback (most recent call last):
File "evaluate.py", line 78, in
print('Size [bytes]: ' + '%.0f' % (size_bytes))
NameError: name 'size_bytes' is not defined

Alvaro Lopez Antelo · Answer 3 · Thu Dec 02 2021 02:48:39 GMT+0800 (China Standard Time)

OK managed to correct this one also, sanity_script.sh has an error:
Where it reads i ../raw_videos/BlowingBubbles_416x240_50_420
it must be i ../raw_videos/BlowingBubbles_416x240_50_420.yuv

Alvaro Lopez Antelo · Answer 4 · Thu Dec 02 2021 02:51:25 GMT+0800 (China Standard Time)

Now I am getting different and much lower readings from the metrics, please help!!!

Alvaro Lopez Antelo · Answer 5 · Sat Dec 04 2021 03:37:06 GMT+0800 (China Standard Time)

Decoded video appears to be with out of order gops of frames, something is wrong probrably with the encoder

Théo Ladune · Answer 6 · Wed Dec 22 2021 16:37:37 GMT+0800 (China Standard Time)

Hi, thanks for the feedback, and sorry for my late response, I've been out of office for a while...
I've corrected the first two errors you mention (missing git LFS and erroneous file path for the video).

I am currently investigating your issue about the very low metrics at the decoder-side. In your logs, we can see that everything went well at the encoder-side (just before the starting decoding line):

[RESULT] | Estimated PSNR | [dB] | 26.7213
[RESULT] | Estimated MS-SSIM | [dB] | 11.8853
[RESULT] | Estimated rate | [byte] | 23382.6
[RESULT] | Real rate | [byte] | 28432
[RESULT] | Estimated rate overhead | [%] | 21.59

I can replicate the exact same results on my side, so the issue seems to come from the decoder.

I would have a few questions to try to solve your issue:

Could you tell me which hardware the code is running on? (GPU model)
Do you run the code within the specified docker environment i.e. PyTorch1.7 with cuda11? Cuda11 seems mandatory to ensure replicability between the encoding and decoding process.
Would it be possible for you to send me the first few decoded frames?
There is a detailed log file available at logs/detailed.txt once the encoding-decoding is done, could you attach it to your reply?
Last resort idea, try running sanity_script.sh on CPU by appending the option --cpu at the end of the command line. (the running time will be quite long)

Thanks for your answers and once again, sorry for the delay of my response, I should be able to answer more quickly now :)

Alvaro Lopez Antelo · Answer 7 · Thu Dec 23 2021 06:51:44 GMT+0800 (China Standard Time)

Hi Dr. Theo,

Thank you for your reply.
Here are the answers:

1 - Quadro RTX 8000 Driver Version: 460.73.01, on a container inside an ubuntu 20.04 VM

2 - yes DOCKERFILE FROM pytorch/pytorch:1.7.0-cuda11.0-cudnn8-devel

3 - bitstream.bin https://www.dropbox.com/s/mzxo2g1dhp2o6ma/bitstream.bin?dl=0
3 - compressed.yuv https://www.dropbox.com/s/85t659ow90ea4s4/compressed.yuv?dl=0

4 - detailed.txt https://www.dropbox.com/s/1wy0uvyv7bj2u79/detailed.txt?dl=0

Théo Ladune · Answer 8 · Mon Dec 27 2021 20:45:56 GMT+0800 (China Standard Time)

Thanks for the additional information. I may have found the bug that made the decoder output out-of-order GOPs.

Could you pull the last commit of the main branch and run the sanity script ? In case it still does not work, please send me the output you get from the sanity script :)

Alvaro Lopez Antelo · Answer 9 · Tue Dec 28 2021 06:08:20 GMT+0800 (China Standard Time)

It appears that the problem is solved!

Here is the output:

root@40c9d37b3a02:/home/ubuntu/AIVC/src# ./sanity_script.sh
********************************************************************************
Starting encoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/torchac_backend...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF torchac_backend.o.d -DTORCH_EXTENSION_NAME=torchac_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /opt/conda/lib/python3.8/site-packages/torch/include -isystem /opt/conda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.8/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.8/site-packages/torch/include/THC -isystem /opt/conda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /opt/conda/lib/python3.8/site-packages/torchac/backend/torchac_backend.cpp -o torchac_backend.o
[2/2] c++ torchac_backend.o -shared -L/opt/conda/lib/python3.8/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o torchac_backend.so
Loading extension module torchac_backend...
[STATE] Start processing: yuv -> png
Input file: ../raw_videos/BlowingBubbles_416x240_50_420.yuv
Output directory: ../raw_videos/BlowingBubbles_416x240_50_420/
Number of frames in the video: 101
Frame: 101 / 101
[STATE] End processing: yuv -> png
[STATE] End processing
[INFO] | Seed | | 666
[W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py:434: UserWarning: Setting attributes on ParameterList is not supported.
warnings.warn("Setting attributes on ParameterList is not supported.")
rm: cannot remove '../tmp/3697412327659087746_4/tmp_bitstream_working_dir/': No such file or directory
rm: cannot remove '../tmp/3697412327659087746_4/tmp_bitstream_working_dir': No such file or directory
[INFO] | Start encoding | |
[INFO] | infer_one_sequence | sequence name | BlowingBubbles_416x240_50_420
[INFO] | infer_one_sequence | GOP_struct_name | 2_GOP_16
[INFO] | infer_one_sequence | idx_rate | 0
[DEBUG] | infer_one_sequence | nb_GOP | 4
[DEBUG] | cat_one_video | gop_file | ../tmp/3697412327659087746_4/tmp_bitstream_working_dir/0g
[DEBUG] | cat_one_video | gop_file | ../tmp/3697412327659087746_4/tmp_bitstream_working_dir/1g
[DEBUG] | cat_one_video | gop_file | ../tmp/3697412327659087746_4/tmp_bitstream_working_dir/2g
[DEBUG] | cat_one_video | gop_file | ../tmp/3697412327659087746_4/tmp_bitstream_working_dir/3g
[INFO] | Encoding done | |
[INFO] | Bitstream path | | ../bitstream.bin
[INFO] | Frame resolution | [H x W] | 240 x 416
[INFO] | First coded frame | [frame] | 0
[INFO] | Last coded frame | [frame] | 100
[INFO] | Number of frames to code | [frame] | 101
[INFO] | Number of frames coded | [frame] | 132
[INFO] | Intra-period | [frame] | 33
[RESULT] | Number of frames | [frame] | 101
[RESULT] | Encoding/decoding time | [s] | 51.4
[RESULT] | Encoding/decoding FPS | [frame/s] | 2.0
[RESULT] | Estimated PSNR | [dB] | 26.7213
[RESULT] | Estimated MS-SSIM | [dB] | 11.8853
[RESULT] | Estimated rate | [byte] | 23382.6
[RESULT] | Real rate | [byte] | 28432
[RESULT] | Estimated rate overhead | [%] | 21.59
rmdir: failed to remove '../tmp/3697412327659087746_4': Directory not empty
********************************************************************************
Starting decoding
Using /root/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/torchac_backend/build.ninja...
Building extension module torchac_backend...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module torchac_backend...
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py:434: UserWarning: Setting attributes on ParameterList is not supported.
warnings.warn("Setting attributes on ParameterList is not supported.")
[INFO] | | | Found CUDA device
[INFO] | Seed | | 666
[W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
[INFO] | Bitstream path | | ../bitstream.bin
[INFO] | Bitstream processing dir. | | ../tmp/3697412327659087746_4/tmp_out_bitstream/
[INFO] | Decoded frames directory | | ../compressed/
[INFO] | Number of GOPs in bitstream | | 4
[INFO] | Index first frame | | 0
[INFO] | Index last frame | | 100
[INFO] | Start decoding | |
[INFO] | Decoding done | |
[RESULT] | Number of frames | [frame] | 101
[RESULT] | Decoding time | [s] | 17.0
[RESULT] | Decoding FPS | [frame/s] | 5.9
[STATE] Starting png -> yuv conversion
rm: cannot remove '../compressed.yuv': No such file or directory
[INFO] | Final decoded video | | ../compressed.yuv
********************************************************************************
Starting evaluation
PSNR [dB]: 26.72130
MS-SSIM : 0.93531
MS-SSIM [dB]: 11.89145
Size [bytes]: 28432

Théo Ladune · Answer 10 · Tue Dec 28 2021 15:44:27 GMT+0800 (China Standard Time)

Great! The commit I pushed yesterday only partially solve the issue. I pushed a few commits this morning to definitely fix the bug. You should pull them from the main branch.

Thanks again for your help fixing this bug :).