InternalError (see above for traceback): Blas SGEMM launch failed : m=61440, n=2, k=256

[Hoàng Đức Long]

I want to run test_pixel_link_on_any_image.py
i use
win 10os
anaconda3
cuda9
tensorflow 1.5.0

python test_pixel_link_on_any_image.py --checkpoint_path C:\Users\Administrator\Desktop\LOG\pixel_link\pixel_link_vgg_4s\conv3_3 --dataset_dir C:\Users\Administrator\Desktop\LOG\pixel_link\data --eval_image_width=1280 --eval_image_height=768 --pixel_conf_threshold=0.5 --link_conf_threshold=0.5 --gpu_memory_fraction=-1

i have error :

WARNING:tensorflow:From test_pixel_link_on_any_image.py:76: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
2020-04-01 16:32:53.590041: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2020-04-01 16:32:54.080675: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 4.96GiB
2020-04-01 16:32:54.085798: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from C:/Users/Administrator/Desktop/LOG/pixel_link/pixel_link_vgg_4s/conv3_3/model.ckpt-38055
2020-04-01 16:34:52.911629: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-04-01 16:34:52.943963: W C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\stream.cc:1901] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "C:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_call
return fn(*args)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1329, in _run_fn
status, run_metadata)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=61440, n=2, k=256
[[Node: evaluation_768x1280/pixel_cls/score_from_conv3_3/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](evaluation_768x1280/conv3/conv3_3/Relu, pixel_cls/score_from_conv3_3/weights/read/_193)]]
[[Node: evaluation_768x1280/Reshape/_195 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_209_evaluation_768x1280/Reshape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test_pixel_link_on_any_image.py", line 157, in
tf.app.run()
File "C:\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 124, in run
_sys.exit(main(argv))
File "test_pixel_link_on_any_image.py", line 153, in main
test()
File "test_pixel_link_on_any_image.py", line 120, in test
feed_dict={image: image_data})
File "C:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
run_metadata_ptr)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1128, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1344, in _do_run
options, run_metadata)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1363, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=61440, n=2, k=256
[[Node: evaluation_768x1280/pixel_cls/score_from_conv3_3/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](evaluation_768x1280/conv3/conv3_3/Relu, pixel_cls/score_from_conv3_3/weights/read/_193)]]
[[Node: evaluation_768x1280/Reshape/_195 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_209_evaluation_768x1280/Reshape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'evaluation_768x1280/pixel_cls/score_from_conv3_3/Conv2D', defined at:
File "test_pixel_link_on_any_image.py", line 157, in
tf.app.run()
File "C:\Anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 124, in run
_sys.exit(main(argv))
File "test_pixel_link_on_any_image.py", line 153, in main
test()
File "test_pixel_link_on_any_image.py", line 88, in test
net = pixel_link_symbol.PixelLinkNet(b_image, is_training=False)
File "C:\Users\Administrator\Desktop\LOG\pixel_link\nets\pixel_link_symbol.py", line 18, in init
self._fuse_feat_layers()
File "C:\Users\Administrator\Desktop\LOG\pixel_link\nets\pixel_link_symbol.py", line 150, in _fuse_feat_layers
config.num_classes, scope = 'pixel_cls')
File "C:\Users\Administrator\Desktop\LOG\pixel_link\nets\pixel_link_symbol.py", line 137, in _fuse_by_cascade_conv1x1_upsample_sum
num_classes, current_layer_name)
File "C:\Users\Administrator\Desktop\LOG\pixel_link\nets\pixel_link_symbol.py", line 58, in _score_layer
normalizer_fn=None)
File "C:\Anaconda3\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
return func(*args, **current_args)
File "C:\Anaconda3\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1057, in convolution
outputs = layer.apply(inputs)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\layers\base.py", line 762, in apply
return self.call(inputs, *args, **kwargs)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\layers\base.py", line 652, in call
outputs = self.call(inputs, *args, **kwargs)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\layers\convolutional.py", line 167, in call
outputs = self._convolution_op(inputs, self.kernel)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 838, in call
return self.conv_op(inp, filter)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 502, in call
return self.call(inp, filter)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 190, in call
name=self.name)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 725, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3160, in create_op
op_def=op_def)
File "C:\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1625, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InternalError (see above for traceback): Blas SGEMM launch failed : m=61440, n=2, k=256
[[Node: evaluation_768x1280/pixel_cls/score_from_conv3_3/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](evaluation_768x1280/conv3/conv3_3/Relu, pixel_cls/score_from_conv3_3/weights/read/_193)]]
[[Node: evaluation_768x1280/Reshape/_195 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_209_evaluation_768x1280/Reshape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]