InvalidArgumentError (see above for traceback): slice index 262144 of dimension 0 out of bounds.
fanchunpeng opened this issue · comments
train loss: 7.23991, miou: 0.22505: 87%|██████████████████████████████████████████████████████████████████████████▊ | 1294/1487 [12:57<02:17, 1.40it/s]2020-11-02 16:43:36.873786: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 262144 of dimension 0 out of bounds.
2020-11-02 16:43:36.873945: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 262144 of dimension 0 out of bounds.
2020-11-02 16:43:36.873967: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 262144 of dimension 0 out of bounds.
2020-11-02 16:43:36.874000: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 262144 of dimension 0 out of bounds.
2020-11-02 16:43:36.874034: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 262144 of dimension 0 out of bounds.
train loss: 7.23991, miou: 0.22505: 87%|██████████████████████████████████████████████████████████████████████████▊ | 1294/1487 [12:57<01:55, 1.66it/s]
Traceback (most recent call last):
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 262144 of dimension 0 out of bounds.
[[{{node tower_0/BiseNetV2/seg_head_segmentation_loss/strided_slice}}]]
[[{{node ConstantFoldingCtrl/miou/mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Switch_0}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tools/cityscapes/train_bisenetv2_cityscapes.py", line 42, in
train_model()
File "tools/cityscapes/train_bisenetv2_cityscapes.py", line 34, in train_model
worker.train()
File "/pxsj/fancp/ZLC/bisenetv2-tensorflow/trainner/cityscapes/cityscapes_bisenetv2_multi_gpu_trainner.py", line 393, in train
self._loss, self._global_step
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 262144 of dimension 0 out of bounds.
[[node tower_0/BiseNetV2/seg_head_segmentation_loss/strided_slice (defined at /pxsj/fancp/ZLC/bisenetv2-tensorflow/bisenet_model/bisenet_v2.py:905) ]]
[[{{node ConstantFoldingCtrl/miou/mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Switch_0}}]]
Caused by op 'tower_0/BiseNetV2/seg_head_segmentation_loss/strided_slice', defined at:
File "tools/cityscapes/train_bisenetv2_cityscapes.py", line 42, in
train_model()
File "tools/cityscapes/train_bisenetv2_cityscapes.py", line 29, in train_model
worker = multi_gpu_trainner.BiseNetV2CityScapesMultiTrainer()
File "/pxsj/fancp/ZLC/bisenetv2-tensorflow/trainner/cityscapes/cityscapes_bisenetv2_multi_gpu_trainner.py", line 153, in init
is_net_first_initialized=is_network_initialized
File "/pxsj/fancp/ZLC/bisenetv2-tensorflow/trainner/cityscapes/cityscapes_bisenetv2_multi_gpu_trainner.py", line 337, in _compute_net_gradients
reuse=is_net_first_initialized
File "/pxsj/fancp/ZLC/bisenetv2-tensorflow/bisenet_model/bisenet_v2.py", line 1112, in compute_loss
n_min=self._ohem_min_sample_nums
File "/pxsj/fancp/ZLC/bisenetv2-tensorflow/bisenet_model/bisenet_v2.py", line 905, in _compute_ohem_cross_entropy_loss
ohem_cond = tf.greater(loss[n_min], ohem_thresh)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 654, in _slice_helper
name=name)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 820, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 9356, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): slice index 262144 of dimension 0 out of bounds.
[[node tower_0/BiseNetV2/seg_head_segmentation_loss/strided_slice (defined at /pxsj/fancp/ZLC/bisenetv2-tensorflow/bisenet_model/bisenet_v2.py:905) ]]
[[{{node ConstantFoldingCtrl/miou/mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Switch_0}}]]
@fanchunpeng If your train your custom data make sure the MIN_SAMPLE_NUMS params is not larger than your input_tensor_height * input_tensor_width in the following cinfig file.
bisenetv2-tensorflow/config/cityscapes/cityscapes_bisenetv2.yaml
Lines 80 to 83 in 710db86
cityscapes data ,Can train, but there are mistakes in the training process
@fanchunpeng Check your local data if there is any empty data or invalid image which can not be read:)
I'll get the data again. If there are still problems, I may need to trouble you. Thank you
@fanchunpeng That problem won't happen if you train the model on cityscapes dataset. Just make sure the local images are invalid and can be correctly read:)
train loss: 10.97951, miou: 0.36470: Thank you. After reorganizing the data, everything is normal. How long did you train? I found that Miou grew slowly.
@fanchunpeng I do not remember the exact time consuming:)