training issue.....
Longyanaddress opened this issue · comments
INFO:tensorflow:using moving average in training, with decay = 0.999900
INFO:tensorflow:Ignoring --checkpoint_path because a checkpoint already exists in /home/longyan/OCR/ocr-data/icdar2015/log/
2019-04-10 09:57:09.416707: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2019-04-10 09:57:09.416730: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2019-04-10 09:57:09.416754: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2019-04-10 09:57:09.416760: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2019-04-10 09:57:09.416786: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2019-04-10 09:57:09.488480: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-04-10 09:57:09.488817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1060 with Max-Q Design
major: 6 minor: 1 memoryClockRate (GHz) 1.3415
pciBusID 0000:01:00.0
Total memory: 5.94GiB
Free memory: 5.51GiB
2019-04-10 09:57:09.488851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0
2019-04-10 09:57:09.488858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0: Y
2019-04-10 09:57:09.488884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 with Max-Q Design, pci bus id: 0000:01:00.0)
INFO:tensorflow:Restoring parameters from /home/longyan/OCR/ocr-data/icdar2015/log/model.ckpt-0
2019-04-10 09:57:09.630815: I tensorflow/core/common_runtime/simple_placer.cc:697] Ignoring device specification /device:GPU:0 for node 'clone_0/fifo_queue_Dequeue' because the input edge from 'icdar2015_prefetch_queue/prefetch_queue/fifo_queue' is a reference connection and already has a device field set to /device:CPU:0
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path /home/longyan/OCR/ocr-data/icdar2015/log/model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
2019-04-10 09:57:11.990542: W tensorflow/core/kernels/draw_bounding_box_op.cc:122] Bounding box (156,-258,181,-211) is completely outside the image and will not be drawn.
2019-04-10 09:57:12.112359: W tensorflow/core/kernels/draw_bounding_box_op.cc:122] Bounding box (156,-258,181,-211) is completely outside the image and will not be drawn.
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Incompatible shapes: [10] vs. [0]
[[Node: ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/LogicalAnd = LogicalAnd[_device="/job:localhost/replica:0/task:0/cpu:0"](ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/Less, ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/Equal)]]
INFO:tensorflow:Caught OutOfRangeError. Stopping Training.
INFO:tensorflow:Finished training! Saving model to disk.
2019-04-10 09:57:13.203005: W tensorflow/core/framework/op_kernel.cc:1192] Unknown: exceptions.AssertionError:
Traceback (most recent call last):
File "train_pixel_link.py", line 293, in
tf.app.run()
File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train_pixel_link.py", line 289, in main
train(train_op)
File "train_pixel_link.py", line 278, in train
session_config = sess_config
File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 767, in train
sv.stop(threads, close_summary_writer=True)
File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 792, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
enqueue_callable()
File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1235, in _single_operation_run
target_list_as_strings, status, None)
File "/home/longyan/anaconda3/envs/py27/lib/python2.7/contextlib.py", line 24, in exit
self.gen.next()
File "/home/longyan/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [10] vs. [0]
[[Node: ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/LogicalAnd = LogicalAnd[_device="/job:localhost/replica:0/task:0/cpu:0"](ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/Less, ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter_overlap/Equal)]]
do you solve this problem i meet the same issue