ValueError: train_pb2.load_all_detection_checkpoint_vars unsupported in TF2
PetriJF opened this issue · comments
Hello. I am having problems with training the NN. I am using Tensorflow 2.4 and I am running the training from the model_main_tf2.py file.
Everything seems to be working fine up until the end when the error pops up:
" ValueError: train_pb2.load_all_detection_checkpoint_vars unsupported in TF2 "
(The full output can be seen below)
I am not sure what causes it or how to fix it so could anyone please give me a hand?
Thank you in advance,
JFPetri
##################################################################
(tf) C:\tensorflow\models\research\object_detection>python model_main_tf2.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config
2021-04-12 16:26:50.859207: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-12 16:26:55.113685: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-04-12 16:26:55.115144: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-04-12 16:26:55.150371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.56GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 119.24GiB/s
2021-04-12 16:26:55.150533: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-12 16:26:55.162904: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-12 16:26:55.163052: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-12 16:26:55.168690: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-04-12 16:26:55.170433: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-04-12 16:26:55.181130: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-04-12 16:26:55.186708: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-04-12 16:26:55.188168: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-04-12 16:26:55.189852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-04-12 16:26:55.191358: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-12 16:26:55.194233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.56GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 119.24GiB/s
2021-04-12 16:26:55.194363: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-04-12 16:26:55.194919: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-04-12 16:26:55.195475: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-04-12 16:26:55.195958: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-04-12 16:26:55.196457: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-04-12 16:26:55.196939: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-04-12 16:26:55.197472: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-04-12 16:26:55.198066: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-04-12 16:26:55.198800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-04-12 16:26:55.849072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-12 16:26:55.849196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-04-12 16:26:55.851581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-04-12 16:26:55.855843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2903 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5)
2021-04-12 16:26:55.857376: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0412 16:26:55.859783 5564 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: None
I0412 16:26:55.864782 5564 config_util.py:552] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0412 16:26:55.865785 5564 config_util.py:552] Maybe overwriting use_bfloat16: False
Traceback (most recent call last):
File "model_main_tf2.py", line 113, in
tf.compat.v1.app.run()
File "C:\Users\james\anaconda3\envs\tf\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\james\anaconda3\envs\tf\lib\site-packages\absl\app.py", line 303, in run
_run_main(main, args)
File "C:\Users\james\anaconda3\envs\tf\lib\site-packages\absl\app.py", line 251, in _run_main
sys.exit(main(argv))
File "model_main_tf2.py", line 104, in main
model_lib_v2.train_loop(
File "C:\tensorflow\models\research\object_detection\model_lib_v2.py", line 513, in train_loop
raise ValueError('train_pb2.load_all_detection_checkpoint_vars '
ValueError: train_pb2.load_all_detection_checkpoint_vars unsupported in TF2
hello i have the same problem did you find any solution for this ?
hello i have the same problem did you find any solution for this ?