UnicodeDecodeError when attempting to load checkpoint
rocanaan opened this issue · comments
Hello,
I have been trying to load checkpoints from a previous run of the Rainbow agent, and getting a UnicodeDecodeError.
Usage:
-
First, run a few rounds of training by calling, from /agents/rainbow :
python run train.py --base_dir ../../log
This successfully trains the agents for the number of iterations on run_experiments and creates a log directory at root. -
After the training finishes, attempt to run the same command again (in my understanding, this is the correct usage for loading checkpoints, as train.py calls initialize_checkpointer, which should either load a checkpoint if it exists or create a new one)
Error message:
/Users/rodrigocanaan/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
['/Users/rodrigocanaan/Dev/hanabi-learning-environment/agents/rainbow', '/Users/rodrigocanaan/Dev/hanabi-learning-environment', '/Users/rodrigocanaan/anaconda3/lib/python36.zip', '/Users/rodrigocanaan/anaconda3/lib/python3.6', '/Users/rodrigocanaan/anaconda3/lib/python3.6/lib-dynload', '/Users/rodrigocanaan/anaconda3/lib/python3.6/site-packages', '/Users/rodrigocanaan/anaconda3/lib/python3.6/site-packages/aeosa', '/Users/rodrigocanaan/Dev/spawningtool/src/spawningtool']
I0301 18:12:50.248378 4649448896 tf_logging.py:115] Creating RainbowAgent agent with the following parameters:
I0301 18:12:50.248818 4649448896 tf_logging.py:115] gamma: 0.990000
I0301 18:12:50.249113 4649448896 tf_logging.py:115] update_horizon: 1.000000
I0301 18:12:50.249352 4649448896 tf_logging.py:115] min_replay_history: 500
I0301 18:12:50.249456 4649448896 tf_logging.py:115] update_period: 4
I0301 18:12:50.249592 4649448896 tf_logging.py:115] target_update_period: 500
I0301 18:12:50.249689 4649448896 tf_logging.py:115] epsilon_train: 0.000000
I0301 18:12:50.249766 4649448896 tf_logging.py:115] epsilon_eval: 0.000000
I0301 18:12:50.249860 4649448896 tf_logging.py:115] epsilon_decay_period: 1000
I0301 18:12:50.249946 4649448896 tf_logging.py:115] tf_device: /gpu:*
I0301 18:12:50.250036 4649448896 tf_logging.py:115] use_staging: True
I0301 18:12:50.250484 4649448896 tf_logging.py:115] optimizer: <tensorflow.python.training.rmsprop.RMSPropOptimizer object at 0x182d615710>
W0301 18:12:50.647318 4649448896 tf_logging.py:125] From /Users/rodrigocanaan/Dev/hanabi-learning-environment/agents/rainbow/rainbow_agent.py:232: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.
See `tf.nn.softmax_cross_entropy_with_logits_v2`.
2019-03-01 18:12:50.953091: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
I0301 18:12:51.199140 4649448896 tf_logging.py:115] learning_rate: 0.000025
I0301 18:12:51.199301 4649448896 tf_logging.py:115] optimizer_epsilon: 0.000031
Traceback (most recent call last):
File "train.py", line 107, in <module>
app.run(main)
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train.py", line 104, in main
launch_experiment()
File "train.py", line 89, in launch_experiment
FLAGS.checkpoint_file_prefix))
File "/Users/rodrigocanaan/Dev/hanabi-learning-environment/agents/rainbow/run_experiment.py", line 220, in initialize_checkpointing
checkpoint_dir, latest_checkpoint_version, dqn_dictionary):
File "/Users/rodrigocanaan/Dev/hanabi-learning-environment/agents/rainbow/dqn_agent.py", line 519, in unbundle
self._replay.load(checkpoint_dir, iteration_number)
File "/Users/rodrigocanaan/Dev/hanabi-learning-environment/agents/rainbow/replay_memory.py", line 582, in load
self.memory.load(checkpoint_dir, suffix)
File "/Users/rodrigocanaan/Dev/hanabi-learning-environment/agents/rainbow/replay_memory.py", line 420, in load
self.__dict__[attr] = np.load(infile, allow_pickle=False)
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/site-packages/numpy/lib/npyio.py", line 423, in load
magic = fid.read(N)
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/gzip.py", line 276, in read
return self._buffer.read(size)
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/_compression.py", line 68, in readinto
data = self.read(len(byte_view))
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/gzip.py", line 463, in read
if not self._read_gzip_header():
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/gzip.py", line 406, in _read_gzip_header
magic = self._fp.read(2)
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/gzip.py", line 91, in read
self.file.read(size-self._length+read)
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 132, in read
pywrap_tensorflow.ReadFromStream(self._read_buf, length, status))
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 100, in _prepare_value
return compat.as_str_any(val)
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/compat.py", line 107, in as_str_any
return as_str(value)
File "/Users/rodrigocanaan/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/compat.py", line 80, in as_text
return bytes_or_text.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
Thank you,
Rodrigo Canaan
Hello,
Was anyone able to reproduce or fix the issue?
Thanks,
Rodrigo
@rocanaan I have the same issue here when I am running python 3.6. I don't seem to have the problem if I use 2.7 instead.