Giters

BichenWuUCB / squeezeDet

A tensorflow implementation for SqueezeDet, a convolutional neural network for object detection.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

train squeezedet on KITTI

ismet29 opened this issue · comments

commented

Hi, I encountered a problem when I trained the model on KITTI, I have the following errors:
OS : win7

I launched the follopwing command to start the training :
python ./src/train.py --dataset=KITTI --pretrained_model_path=./data/squeezenet_v1.0_SR_0.750.pkl --data_path=./data/KITTI --image_set=train --train_dir=./train12 --net=squeezeDet --summary_step=100 --checkpoint_step=500 --gpu 0

Thx for replying.

2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B46EC8500 of size 589824
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B46F58500 of size 131072
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B46F78500 of size 65536
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B46F88500 of size 589824
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B47018500 of size 196608
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B47048500 of size 147456
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B4706C500 of size 1327104
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B471B0500 of size 294912
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B471F8500 of size 147456
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B4721C500 of size 1558016
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B47398B00 of size 5875200
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B47933100 of size 663552000
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B6F203100 of size 663552000
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000B96AD3100 of size 1327104000
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000BE5C73100 of size 663552000
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000C0D543100 of size 663552000
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000C34E13100 of size 663552000
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000C5C6E3100 of size 1327104000
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CAB883100 of size 82944000
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Free at 0000000CB079D100 of size 250272768
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CBF64AD00 of size 17625600
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC0719F00 of size 5875200
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC0CB4500 of size 5875200
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC124EB00 of size 5875200
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC17E9100 of size 5875200
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC1D83700 of size 5875200
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC231DD00 of size 1990656
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC2503D00 of size 36864
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC250CD00 of size 36864
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC2515D00 of size 16384
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC2519D00 of size 16384
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC251DD00 of size 147456
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC2541D00 of size 32768
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC2549D00 of size 16384
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:665] Chunk at 0000000CC254DD00 of size 263613184
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:671] Summary of in-use Chunks by size:
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 183 Chunks of size 256 totalling 45.8KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 21 Chunks of size 512 totalling 10.5KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 12 Chunks of size 768 totalling 9.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 12 Chunks of size 1024 totalling 12.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.3KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 12 Chunks of size 1536 totalling 18.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 10 Chunks of size 4096 totalling 40.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 2 Chunks of size 6912 totalling 13.5KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 4 Chunks of size 8192 totalling 32.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 10 Chunks of size 16384 totalling 160.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 28672 totalling 28.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 4 Chunks of size 32768 totalling 128.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 12 Chunks of size 36864 totalling 432.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 40960 totalling 40.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 4 Chunks of size 49152 totalling 192.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 7 Chunks of size 65536 totalling 448.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 3 Chunks of size 73728 totalling 216.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 5 Chunks of size 98304 totalling 480.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 114688 totalling 112.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 3 Chunks of size 131072 totalling 384.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 13 Chunks of size 147456 totalling 1.83MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 6 Chunks of size 196608 totalling 1.13MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 293888 totalling 287.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 5 Chunks of size 294912 totalling 1.41MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 295936 totalling 289.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 8 Chunks of size 331776 totalling 2.53MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 5 Chunks of size 589824 totalling 2.81MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 995328 totalling 972.0KiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 1130496 totalling 1.08MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 5 Chunks of size 1327104 totalling 6.33MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 1558016 totalling 1.49MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 1843200 totalling 1.76MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 4 Chunks of size 1990656 totalling 7.59MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 12 Chunks of size 5875200 totalling 67.24MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 2 Chunks of size 17625600 totalling 33.62MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 23500800 totalling 22.41MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 82944000 totalling 79.10MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 165888000 totalling 158.20MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 263613184 totalling 251.40MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 331776000 totalling 316.41MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 1 Chunks of size 501350400 totalling 478.13MiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 5 Chunks of size 663552000 totalling 3.09GiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:674] 2 Chunks of size 1327104000 totalling 2.47GiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:678] Sum Total of in-use chunks: 6.97GiB
2018-05-18 12:11:09.236327: I T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:680] Stats:
Limit: 7730841600
InUse: 7480552704
MaxInUse: 7480552704
NumAllocs: 513
MaxAllocSize: 2654208000

2018-05-18 12:11:09.236327: W T:\src\github\tensorflow\tensorflow\core\common_ru
ntime\bfc_allocator.cc:279] ****************************************************
*****************************************__****x
2018-05-18 12:11:09.236327: W T:\src\github\tensorflow\tensorflow\core\framework
\op_kernel.cc:1318] OP_REQUIRES failed at conv_ops.cc:386 : Resource exhausted:
OOM when allocating tensor with shape[20,135,240,128] and type float on /job:loc
alhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\client\session.py", line 1322, in _do_call
return fn(*args)
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\client\session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\client\session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocat
ing tensor with shape[20,135,240,128] and type float on /job:localhost/replica:0
/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: fire4/expand1x1/convolution = Conv2D[T=DT_FLOAT, data_format="N
HWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on
_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fire4/squeeze
1x1/relu, fire4/expand1x1/kernels/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: bbox/trimming/activation_summary_2/Mean/_373 = _Recv[client_ter

minated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_
device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1
, tensor_name="edge_1481_bbox/trimming/activation_summary_2/Mean", tensor_type=D
T_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./src/train.py", line 401, in
tf.app.run()
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "./src/train.py", line 397, in main
train()
File "./src/train.py", line 340, in train
op_list, feed_dict=feed_dict)
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\client\session.py", line 900, in run
run_metadata_ptr)
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\client\session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\client\session.py", line 1316, in _do_run
run_metadata)
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\client\session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocat
ing tensor with shape[20,135,240,128] and type float on /job:localhost/replica:0
/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: fire4/expand1x1/convolution = Conv2D[T=DT_FLOAT, data_format="N
HWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on
_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fire4/squeeze
1x1/relu, fire4/expand1x1/kernels/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: bbox/trimming/activation_summary_2/Mean/_373 = _Recv[client_ter

minated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_
device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1
, tensor_name="edge_1481_bbox/trimming/activation_summary_2/Mean", tensor_type=D
T_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'fire4/expand1x1/convolution', defined at:
File "./src/train.py", line 401, in
tf.app.run()
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "./src/train.py", line 397, in main
train()
File "./src/train.py", line 130, in train
model = SqueezeDet(mc)
File "E:\ismet\squeezeDet\src\nets\squeezeDet.py", line 25, in init
self._add_forward_graph()
File "E:\ismet\squeezeDet\src\nets\squeezeDet.py", line 55, in _add_forward_gr
aph
'fire4', pool3, s1x1=32, e1x1=128, e3x3=128, freeze=False)
File "E:\ismet\squeezeDet\src\nets\squeezeDet.py", line 102, in _fire_layer
padding='SAME', stddev=stddev, freeze=freeze)
File "E:\ismet\squeezeDet\src\nn_skeleton.py", line 567, in _conv_layer
name='convolution')
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\ops\gen_nn_ops.py", line 1042, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\framework\ops.py", line 3392, in create_op
op_def=op_def)
File "D:\Application\anaconda\envs\tensorflow\lib\site-packages\tensorflow\pyt
hon\framework\ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-
access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor wit
h shape[20,135,240,128] and type float on /job:localhost/replica:0/task:0/device
:GPU:0 by allocator GPU_0_bfc
[[Node: fire4/expand1x1/convolution = Conv2D[T=DT_FLOAT, data_format="N
HWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on
_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fire4/squeeze
1x1/relu, fire4/expand1x1/kernels/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[Node: bbox/trimming/activation_summary_2/Mean/_373 = _Recv[client_ter

minated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_
device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1
, tensor_name="edge_1481_bbox/trimming/activation_summary_2/Mean", tensor_type=D
T_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.