Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint.

Question

Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint.

caolusg opened this issue 5 years ago · comments

i try to use pre trained model and got erro below:

Traceback (most recent call last):
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Tensor name "d_bn1/moving_mean" not found in checkpoint files Data/Models/latest_model_flowers_temp.ckpt
[[{{node save/RestoreV2}} = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1546, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Tensor name "d_bn1/moving_mean" not found in checkpoint files Data/Models/latest_model_flowers_temp.ckpt
[[node save/RestoreV2 (defined at generate_images.py:67) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
File "generate_images.py", line 107, in
main()
File "generate_images.py", line 67, in main
saver = tf.train.Saver()
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1102, in init
self.build()
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1114, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1151, in _build
build_save=build_save, build_restore=build_restore)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 795, in _build_internal
restore_sequentially, reshape)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 406, in _AddRestoreOps
restore_sequentially)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 862, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1466, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Tensor name "d_bn1/moving_mean" not found in checkpoint files Data/Models/latest_model_flowers_temp.ckpt
[[node save/RestoreV2 (defined at generate_images.py:67) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1556, in restore
names_to_keys = object_graph_key_mapping(save_path)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1830, in object_graph_key_mapping
checkpointable.OBJECT_GRAPH_PROTO_KEY)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 371, in get_tensor
status)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "generate_images.py", line 107, in
main()
File "generate_images.py", line 68, in main
saver.restore(sess, args.model_path)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1562, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Tensor name "d_bn1/moving_mean" not found in checkpoint files Data/Models/latest_model_flowers_temp.ckpt
[[node save/RestoreV2 (defined at generate_images.py:67) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
File "generate_images.py", line 107, in
main()
File "generate_images.py", line 67, in main
saver = tf.train.Saver()
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1102, in init
self.build()
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1114, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1151, in _build
build_save=build_save, build_restore=build_restore)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 795, in _build_internal
restore_sequentially, reshape)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 406, in _AddRestoreOps
restore_sequentially)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 862, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1466, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Tensor name "d_bn1/moving_mean" not found in checkpoint files Data/Models/latest_model_flowers_temp.ckpt
[[node save/RestoreV2 (defined at generate_images.py:67) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

zhaojw · Answer 1 · Sat Jan 19 2019 13:50:48 GMT+0800 (China Standard Time)

I also encountered the same error message. Have you solved it?

我尝试使用预先训练过的模型，并在下面得到错误：

回溯（最近一次调用最后一次）：
文件“/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py”，第1334行，在_do_call中
返回fn（* args）
文件“/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py”，第1319行，在_run_fn
选项中，feed_dict，fetch_list，target_list，run_metadata）
文件“/ home / lucao /anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py“，第1407行，在_call_tf_sessionrun
run_metadata中）
tensorflow.python.framework.errors_impl.NotFoundError：未找到张量名称”d_bn1 / moving_mean“在检查点文件Data / Models / latest_model_flowers_temp.ckpt中
[[{{node save / RestoreV2}} = RestoreV2 [dtypes = [DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT，...，DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT]，_ device =“/ job：localhost / replica ：0 / task：0 / device：CPU：0“]（_ arg_save / Const_0_0，save / RestoreV2 / tensor_names，save / RestoreV2 / shape_and_slices）]]

在处理上述异常期间，发生了另一个异常：

Traceback（最近一次调用最后一次）：
文件“/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py”，第1546行，在restore
{self.saver_def.filename_tensor_name： save_path}）
文件“/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py”，第929行，运行
run_metadata_ptr）
文件“/ home / lucao / anaconda3 / lib /python3.6/site-packages/tensorflow/python/client/session.py“，第1152行，在_run
feed_dict_tensor，options，run_metadata中）
文件”/home/lucao/anaconda3/lib/python3.6/site-packages/ tensorflow / python / client / session.py“，第1328行，在_do_run
run_metadata中）
文件”/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session。py“，第1348行，在_do_call中
raise type（e）（node_def，op，message）
tensorflow.python.framework.errors_impl.NotFoundError：在检查点文件中找不到Tensor名称“d_bn1 / moving_mean”Data / Models / latest_model_flowers_temp.ckpt
[[node save / RestoreV2（定义于generate_images.py:67）= RestoreV2 [dtypes = [DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT，...，DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT]，_ device =“/ job：localhost / replica：0 / task ：0 / device：CPU：0“]（_ arg_save / Const_0_0，save / RestoreV2 / tensor_names，save / RestoreV2 / shape_and_slices）]]

Caused by op 'save/RestoreV2', defined at:
File "generate_images.py", line 107, in
main()
File "generate_images.py", line 67, in main
saver = tf.train.Saver()
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1102, in init
self.build()
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1114, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1151, in _build
build_save=build_save, build_restore=build_restore)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 795, in _build_internal
restore_sequentially, reshape)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 406, in _AddRestoreOps
restore_sequentially)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 862, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1466, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Tensor name "d_bn1/moving_mean" not found in checkpoint files Data/Models/latest_model_flowers_temp.ckpt
[[node save/RestoreV2 (defined at generate_images.py:67) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1556, in restore
names_to_keys = object_graph_key_mapping(save_path)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1830, in object_graph_key_mapping
checkpointable.OBJECT_GRAPH_PROTO_KEY)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 371, in get_tensor
status)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "generate_images.py", line 107, in
main()
File "generate_images.py", line 68, in main
saver.restore(sess, args.model_path)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1562, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Tensor name "d_bn1/moving_mean" not found in checkpoint files Data/Models/latest_model_flowers_temp.ckpt
[[node save/RestoreV2 (defined at generate_images.py:67) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
File "generate_images.py", line 107, in
main()
File "generate_images.py", line 67, in main
saver = tf.train.Saver()
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1102, in init
self.build()
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1114, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1151, in _build
build_save=build_save, build_restore=build_restore)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 795, in _build_internal
restore_sequentially, reshape)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 406, in _AddRestoreOps
restore_sequentially)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 862, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1466, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/lucao/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

NotFoundError（参见上面的回溯）：从检查点恢复失败。这很可能是由于检查点中缺少变量名称或其他图形键。请确保您没有根据检查点更改预期的图表。原始错误：

在检查点文件中找不到张量名“d_bn1 / moving_mean”Data / Models / latest_model_flowers_temp.ckpt
[[node save / RestoreV2（在generate_images.py:67中定义）= RestoreV2 [dtypes = [DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT， ...，DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT，DT_FLOAT]，_ device =“/ job：localhost / replica：0 / task：0 / device：CPU：0”]（_ arg_save / Const_0_0，save / RestoreV2 / tensor_names，save / RestoreV2 / shape_and_slices）]]

I also encountered the same error message. Have you solved it?

svstbrt · Answer 2 · Thu Apr 04 2019 03:12:34 GMT+0800 (China Standard Time)

Did you guys solve this? I could really use help here. I get the same error @coderzjw @caolusg

Pragyan Subedi · Answer 3 · Thu Apr 25 2019 22:55:29 GMT+0800 (China Standard Time)

Writing tf.reset_default_graph() before training and saving could have made the checkpoint restorable @coderzjw @caolusg

Rohan · Answer 4 · Sun Jan 12 2020 16:16:35 GMT+0800 (China Standard Time)

Writing tf.reset_default_graph() before training and saving could have made the checkpoint restorable @coderzjw @caolusg

Hey so where should I write it?

Rohan · Answer 5 · Sun Jan 12 2020 19:13:34 GMT+0800 (China Standard Time)

Writing tf.reset_default_graph() before training and saving could have made the checkpoint restorable @coderzjw @caolusg

: Tensor("moments/moments_1/mean:0", shape=(512,), dtype=float32) must be from the same graph as Tensor("g_bn0/g_bn0/moments/moments_1/mean/ExponentialMovingAverage:0", shape=(512,), dtype=float32_ref).

I get this error when I write the above line, help!