untimeError: For 'GatherNd', the indices[0]: [const vector][0, 409600], does not index into input_shape: [const vector][1, 409600].
zzzzcl opened this issue · comments
I want to train dbnet++
but when run the :
python tools/train.py --config configs/det/dbnet/db++_r50_icdar15.yaml
I got the following error:
Start training... (The first epoch takes longer, please wait...)
Traceback (most recent call last):
File "tools/train.py", line 300, in
main(config)
File "tools/train.py", line 243, in main
model.train(
File "/home/orange/anaconda3/envs/metro/lib/python3.8/site-packages/mindspore/train/model.py", line 1044, in train
self._train(epoch,
File "/home/orange/anaconda3/envs/metro/lib/python3.8/site-packages/mindspore/train/model.py", line 100, in wrapper
func(self, *args, **kwargs)
File "/home/orange/anaconda3/envs/metro/lib/python3.8/site-packages/mindspore/train/model.py", line 597, in _train
self._train_dataset_sink_process(epoch, train_dataset, list_callback,
File "/home/orange/anaconda3/envs/metro/lib/python3.8/site-packages/mindspore/train/model.py", line 681, in _train_dataset_sink_process
outputs = train_network(*inputs)
File "/home/orange/anaconda3/envs/metro/lib/python3.8/site-packages/mindspore/nn/cell.py", line 620, in call
out = self.compile_and_run(*args, **kwargs)
File "/home/orange/anaconda3/envs/metro/lib/python3.8/site-packages/mindspore/nn/cell.py", line 942, in compile_and_run
return _cell_graph_executor(self, *new_args, phase=self.phase)
File "/home/orange/anaconda3/envs/metro/lib/python3.8/site-packages/mindspore/common/api.py", line 1439, in call
return self.run(obj, *args, phase=phase)
File "/home/orange/anaconda3/envs/metro/lib/python3.8/site-packages/mindspore/common/api.py", line 1478, in run
return self._exec_pip(obj, *args, phase=phase_real)
File "/home/orange/anaconda3/envs/metro/lib/python3.8/site-packages/mindspore/common/api.py", line 102, in wrapper
results = fn(*arg, **kwargs)
File "/home/orange/anaconda3/envs/metro/lib/python3.8/site-packages/mindspore/common/api.py", line 1458, in _exec_pip
return self._graph_executor(args, phase)
RuntimeError: For 'GatherNd', the indices[0]: [const vector][0, 409600], does not index into input_shape: [const vector][1, 409600].
- C++ Call Stack: (For framework developers)
mindspore/ccsrc/plugin/device/gpu/kernel/arrays/gathernd_gpu_kernel.h:59 Launch
Can you give me some suggestions?
Thanks @zzzzcl for your report. Can you tell which MindSpore version did you use?
Thanks @zzzzcl for your report. Can you tell which MindSpore version did you use?
I meet the same problem, used mindspore version 2.0.0, could you give some solutions
This issue is caused by "empty" (i.e. no text instances or all are labeled as "ignored") images during loss calculation. The issue is fixed in PR #546. @zzzzcl @ShaunLeeblob Let me know if this worked for you.
This issue is caused by "empty" (i.e. no text instances or all are labeled as "ignored") images during loss calculation. The issue is fixed in PR #546. @zzzzcl @ShaunLeeblob Let me know if this worked for you.
Well done! It works!
Thanks @hadipash @ShaunLeeblob , I've already redeployed mindspore on another computer and it works fine. Currently, I'm reappearing the DB/DB++ method.
The environments I used are:
Before: 2080Ti GPU, CUDA 11.1, python3.9, mindspore 2.0.0, openmpi 4.0.3,
and mindspore lite.
Now: 3090GPU, CUDA 11.1, python3.7, mindspore 2.1.0, openmpi 4.0.3, and mindspore lite.