training error with gpu
alpace opened this issue · comments
INFO:tensorflow:using moving average in training, with decay = 0.999900
2019-03-16 09:41:40.360501: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2019-03-16 09:41:40.360536: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2019-03-16 09:41:40.360544: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2019-03-16 09:41:40.360552: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2019-03-16 09:41:40.360560: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2019-03-16 09:41:40.475868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: Quadro M2000
major: 5 minor: 2 memoryClockRate (GHz) 1.1625
pciBusID 0000:04:00.0
Total memory: 3.95GiB
Free memory: 2.79GiB
2019-03-16 09:41:40.475916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0
2019-03-16 09:41:40.475930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y
2019-03-16 09:41:40.475946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Quadro M2000, pci bus id: 0000:04:00.0)
INFO:tensorflow:Restoring parameters from ./models/icdar2015/model.ckpt-0
2019-03-16 09:41:43.114186: I tensorflow/core/common_runtime/simple_placer.cc:669] Ignoring device specification /device:GPU:0 for node 'clone_0/fifo_queue_Dequeue' because the input edge from 'icdar2015_prefetch_queue/prefetch_queue/fifo_queue' is a reference connection and already has a device field set to /device:CPU:0
INFO:tensorflow:Starting Session.
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
2019-03-16 09:42:05.462899: W tensorflow/core/kernels/draw_bounding_box_op.cc:122] Bounding box (92,-87,128,-77) is completely outside the image and will not be drawn.
2019-03-16 09:42:05.462944: W tensorflow/core/kernels/draw_bounding_box_op.cc:122] Bounding box (-16,75,-4,90) is completely outside the image and will not be drawn.
2019-03-16 09:42:05.462956: W tensorflow/core/kernels/draw_bounding_box_op.cc:122] Bounding box (-27,-48,-15,-37) is completely outside the image and will not be drawn.
2019-03-16 09:42:05.474938: W tensorflow/core/kernels/draw_bounding_box_op.cc:122] Bounding box (92,-87,128,-77) is completely outside the image and will not be drawn.
2019-03-16 09:42:05.475122: W tensorflow/core/kernels/draw_bounding_box_op.cc:122] Bounding box (-16,75,-4,90) is completely outside the image and will not be drawn.
2019-03-16 09:42:05.475224: W tensorflow/core/kernels/draw_bounding_box_op.cc:122] Bounding box (-27,-48,-15,-37) is completely outside the image and will not be drawn.
2019-03-16 09:42:06.839645: E tensorflow/stream_executor/cuda/cuda_dnn.cc:359] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-03-16 09:42:06.839699: E tensorflow/stream_executor/cuda/cuda_dnn.cc:326] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2019-03-16 09:42:06.839718: F tensorflow/core/kernels/conv_ops.cc:659] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms)
./scripts/train.sh: 行 37: 136261 已放弃 (核心已转储)
I tensorflow/core/common_runtime/simple_placer.cc:669] Ignoring device specification /device:GPU:0 for node 'clone_0/fifo_queue_Dequeue' because the input edge from 'icdar2015_prefetch_queue/prefetch_queue/fifo_queue' is a reference connection and already has a device field set to /device:CPU:0
Does anybody have this problem?