RuntimeError: [enforce fail at pybind_state.h:425] . Exception encountered running PythonOp function: ValueError: min() arg is an empty sequence
yuyijie1995 opened this issue · comments
I download the pretrained model from "'https://dl.fbaipublicfiles.com/detectron'" and put it in the /tmp/detectron/ImageNetPretrained/25093814
BUT the error happend when I start to train the model. I do not understand this error meaning.
Traceback (most recent call last):
File "/home/wrc/CBNet/tools/train_net.py", line 132, in
main()
File "/home/wrc/CBNet/tools/train_net.py", line 114, in main
checkpoints = detectron.utils.train.train_model()
File "/home/wrc/CBNet/detectron/utils/train.py", line 67, in train_model
workspace.RunNet(model.net.Proto().name)
File "/home/wrc/anaconda3/envs/py27/lib/python2.7/site-packages/caffe2/python/workspace.py", line 250, in RunNet
StringifyNetName(name), num_iter, allow_fail,
File "/home/wrc/anaconda3/envs/py27/lib/python2.7/site-packages/caffe2/python/workspace.py", line 211, in CallWithExceptionIntercept
return func(*args, **kwargs)
RuntimeError: [enforce fail at pybind_state.h:425] . Exception encountered running PythonOp function: ValueError: min() arg is an empty sequence
At:
/home/wrc/CBNet/detectron/utils/segms.py(136): polys_to_boxes
/home/wrc/CBNet/detectron/roi_data/mask_rcnn.py(46): add_mask_rcnn_blobs
/home/wrc/CBNet/detectron/roi_data/cascade_rcnn.py(193): _sample_rois
/home/wrc/CBNet/detectron/roi_data/cascade_rcnn.py(105): add_cascade_rcnn_blobs
/home/wrc/CBNet/detectron/ops/distribute_cascade_proposals.py(61): forward
Error from operator:
input: "gpu_0/proposals_3" input: "gpu_0/roidb" input: "gpu_0/im_info" output: "gpu_0/rois_3" output: "gpu_0/labels_int32_3" output: "gpu_0/bbox_targets_3" output: "gpu_0/bbox_inside_weights_3" output: "gpu_0/bbox_outside_weights_3" output: "gpu_0/mapped_gt_boxes_3" output: "gpu_0/mask_rois" output: "gpu_0/roi_has_mask_int32" output: "gpu_0/masks_int32" output: "gpu_0/rois_3_fpn2" output: "gpu_0/rois_3_fpn3" output: "gpu_0/rois_3_fpn4" output: "gpu_0/rois_3_fpn5" output: "gpu_0/rois_3_idx_restore_int32" output: "gpu_0/mask_rois_fpn2" output: "gpu_0/mask_rois_fpn3" output: "gpu_0/mask_rois_fpn4" output: "gpu_0/mask_rois_fpn5" output: "gpu_0/mask_rois_idx_restore_int32" name: "DistributeCascadeProposalsOp:gpu_0/proposals_3,gpu_0/roidb,gpu_0/im_info" type: "Python" arg { name: "grad_input_indices" } arg { name: "token" s: "forward:11" } arg { name: "grad_output_indices" } device_option { device_type: 0 }Error from operator:
input: "gpu_0/proposals_3" input: "gpu_0/roidb" input: "gpu_0/im_info" output: "gpu_0/rois_3" output: "gpu_0/labels_int32_3" output: "gpu_0/bbox_targets_3" output: "gpu_0/bbox_inside_weights_3" output: "gpu_0/bbox_outside_weights_3" output: "gpu_0/mapped_gt_boxes_3" output: "gpu_0/mask_rois" output: "gpu_0/roi_has_mask_int32" output: "gpu_0/masks_int32" output: "gpu_0/rois_3_fpn2" output: "gpu_0/rois_3_fpn3" output: "gpu_0/rois_3_fpn4" output: "gpu_0/rois_3_fpn5" output: "gpu_0/rois_3_idx_restore_int32" output: "gpu_0/mask_rois_fpn2" output: "gpu_0/mask_rois_fpn3" output: "gpu_0/mask_rois_fpn4" output: "gpu_0/mask_rois_fpn5" output: "gpu_0/mask_rois_idx_restore_int32" name: "DistributeCascadeProposalsOp:gpu_0/proposals_3,gpu_0/roidb,gpu_0/im_info" type: "Python" arg { name: "grad_input_indices" } arg { name: "token" s: "forward:11" } arg { name: "grad_output_indices" } device_option { device_type: 1 device_id: 0 }frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x59 (0x7f2dc790c409 in /home/wrc/anaconda3/envs/py27/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libc10.so)
frame #1: + 0xa2b85 (0x7f2dc805ab85 in /home/wrc/anaconda3/envs/py27/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #2: + 0xa0fe7 (0x7f2dc8058fe7 in /home/wrc/anaconda3/envs/py27/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #3: + 0xea931 (0x7f2dc80a2931 in /home/wrc/anaconda3/envs/py27/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #4: + 0xe8ffd (0x7f2dc80a0ffd in /home/wrc/anaconda3/envs/py27/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #5: caffe2::AsyncNetBase::run(int, int) + 0x144 (0x7f2da8e92b94 in /home/wrc/anaconda3/envs/py27/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #6: + 0x168f009 (0x7f2da8e99009 in /home/wrc/anaconda3/envs/py27/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #7: c10::ThreadPool::main_loop(unsigned long) + 0x2a3 (0x7f2dc79062f3 in /home/wrc/anaconda3/envs/py27/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libc10.so)
frame #8: + 0xc8421 (0x7f2ddc0a1421 in /home/wrc/anaconda3/envs/py27/bin/../lib/libstdc++.so.6)
frame #9: + 0x76ba (0x7f2de87036ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #10: clone + 0x6d (0x7f2de7d2941d in /lib/x86_64-linux-gnu/libc.so.6)
if i remember correctly that error occurs when your dataset is incorrect. been a while since i solved this error for me though, so no guarantee. do you have masks in your dataset?
I added the mask and it works! thanks a lot.