tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: masked_lm_positions. Can't parse serialized Example.
yubini0906 opened this issue · comments
When I run the run_pretraining.py file, the code runs well, but this error occurs.
I don't know what the problem is.
I set max_predictions_per_seq=20 and ran. In the create_pretraining_data.py file, max_predictions_per_seq is also 20.
set max_seq_length=128
NFO:tensorflow:global_step/sec: 2.5727
I0830 19:28:21.643072 140575351207744 tpu_estimator.py:2307] global_step/sec: 2.5727
INFO:tensorflow:examples/sec: 82.3263
I0830 19:28:21.644017 140575351207744 tpu_estimator.py:2308] examples/sec: 82.3263
2022-08-30 19:28:21.647980: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: masked_lm_positions. Can't parse serialized Example.
INFO:tensorflow:global_step/sec: 2.58677
I0830 19:28:22.029351 140575351207744 tpu_estimator.py:2307] global_step/sec: 2.58677
INFO:tensorflow:examples/sec: 82.7765
I0830 19:28:22.029989 140575351207744 tpu_estimator.py:2308] examples/sec: 82.7765
INFO:tensorflow:global_step/sec: 2.57967
I0830 19:28:22.416983 140575351207744 tpu_estimator.py:2307] global_step/sec: 2.57967
INFO:tensorflow:examples/sec: 82.5493
I0830 19:28:22.417581 140575351207744 tpu_estimator.py:2308] examples/sec: 82.5493
INFO:tensorflow:global_step/sec: 2.57398
I0830 19:28:22.805490 140575351207744 tpu_estimator.py:2307] global_step/sec: 2.57398
INFO:tensorflow:examples/sec: 82.3672
I0830 19:28:22.806136 140575351207744 tpu_estimator.py:2308] examples/sec: 82.3672
INFO:tensorflow:global_step/sec: 2.59985
I0830 19:28:23.189946 140575351207744 tpu_estimator.py:2307] global_step/sec: 2.59985
INFO:tensorflow:examples/sec: 83.1952
I0830 19:28:23.190213 140575351207744 tpu_estimator.py:2308] examples/sec: 83.1952
ERROR:tensorflow:Error recorded from training_loop: {{function_node _inference_tf_data_experimental_map_and_batch_83}} Key: masked_lm_positions. Can't parse serialized Example.
[[{{node ParseSingleExample/ParseSingleExample}}]]
[[IteratorGetNext]]
E0830 19:28:23.417265 140575351207744 error_handling.py:75] Error recorded from training_loop: {{function_node _inference_tf_data_experimental_map_and_batch_83}} Key: masked_lm_positions. Can't parse serialized Example.
[[{{node ParseSingleExample/ParseSingleExample}}]]
[[IteratorGetNext]]
INFO:tensorflow:training_loop marked as finished
I0830 19:28:23.417502 140575351207744 error_handling.py:101] training_loop marked as finished
WARNING:tensorflow:Reraising captured error
W0830 19:28:23.417612 140575351207744 error_handling.py:135] Reraising captured error
Traceback (most recent call last):
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node _inference_tf_data_experimental_map_and_batch_83}} Key: masked_lm_positions. Can't parse serialized Example.
[[{{node ParseSingleExample/ParseSingleExample}}]]
[[IteratorGetNext]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_pretraining.py", line 493, in
tf.app.run()
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "run_pretraining.py", line 466, in main
estimator.train(input_fn=train_input_fn, max_steps=FLAGS.num_train_steps)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3035, in train
rendezvous.raise_errors()
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 136, in raise_errors
six.reraise(typ, value, traceback)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/six.py", line 719, in reraise
raise value
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3030, in train
saving_listeners=saving_listeners)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
saving_listeners)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
run_metadata=run_metadata)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
raise six.reraise(*original_exc_info)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/six.py", line 719, in reraise
raise value
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
return self._sess.run(*args, **kwargs)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
run_metadata=run_metadata)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
return self._sess.run(*args, **kwargs)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/lyb/anaconda3/envs/sbert/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node _inference_tf_data_experimental_map_and_batch_83}} Key: masked_lm_positions. Can't parse serialized Example.
[[{{node ParseSingleExample/ParseSingleExample}}]]
[[IteratorGetNext]]
I got the same issue. Did you solve that?
Same here... Any ideas how to solve?