Error while training the model "ValueError: num_samples should be a positive integer value, but got num_samples=0"
prabhata07 opened this issue · comments
Code we used for training:
!python /content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/train.py --cfg_file /content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/cfgs/kitti_models/CaDDN.yaml
Log we got:
2022-11-19 21:23:27,419 INFO Start logging
2022-11-19 21:23:27,420 INFO CUDA_VISIBLE_DEVICES=ALL
2022-11-19 21:23:27,420 INFO cfg_file /content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/cfgs/kitti_models/CaDDN.yaml
2022-11-19 21:23:27,420 INFO batch_size 4
2022-11-19 21:23:27,420 INFO epochs 80
2022-11-19 21:23:27,420 INFO workers 8
2022-11-19 21:23:27,420 INFO extra_tag default
2022-11-19 21:23:27,421 INFO ckpt None
2022-11-19 21:23:27,421 INFO pretrained_model None
2022-11-19 21:23:27,421 INFO launcher none
2022-11-19 21:23:27,421 INFO tcp_port 18888
2022-11-19 21:23:27,421 INFO sync_bn False
2022-11-19 21:23:27,421 INFO fix_random_seed False
2022-11-19 21:23:27,421 INFO ckpt_save_interval 1
2022-11-19 21:23:27,421 INFO local_rank 0
2022-11-19 21:23:27,421 INFO max_ckpt_save_num 30
2022-11-19 21:23:27,421 INFO merge_all_iters_to_one_epoch False
2022-11-19 21:23:27,421 INFO set_cfgs None
2022-11-19 21:23:27,421 INFO max_waiting_mins 0
2022-11-19 21:23:27,422 INFO start_epoch 0
2022-11-19 21:23:27,422 INFO save_to_file False
2022-11-19 21:23:27,422 INFO cfg.ROOT_DIR: /content/gdrive/MyDrive/IITH_VCA/CaDDN
2022-11-19 21:23:27,422 INFO cfg.LOCAL_RANK: 0
2022-11-19 21:23:27,422 INFO cfg.CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']
2022-11-19 21:23:27,422 INFO
cfg.DATA_CONFIG = edict()
2022-11-19 21:23:27,422 INFO cfg.DATA_CONFIG.DATASET: KittiDataset
2022-11-19 21:23:27,422 INFO cfg.DATA_CONFIG.DATA_PATH: /gdrive/MyDrive/3D_Obj/CaDDN/data/kitti
2022-11-19 21:23:27,422 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [2, -30.08, -3.0, 46.8, 30.08, 1.0]
2022-11-19 21:23:27,422 INFO
cfg.DATA_CONFIG.DATA_SPLIT = edict()
2022-11-19 21:23:27,422 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train
2022-11-19 21:23:27,423 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: test
2022-11-19 21:23:27,423 INFO
cfg.DATA_CONFIG.INFO_PATH = edict()
2022-11-19 21:23:27,423 INFO cfg.DATA_CONFIG.INFO_PATH.train: ['kitti_infos_trainval.pkl']
2022-11-19 21:23:27,423 INFO cfg.DATA_CONFIG.INFO_PATH.test: ['kitti_infos_test.pkl']
2022-11-19 21:23:27,423 INFO cfg.DATA_CONFIG.FOV_POINTS_ONLY: True
2022-11-19 21:23:27,423 INFO
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict()
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'random_image_flip', 'ALONG_AXIS_LIST': ['horizontal']}]
2022-11-19 21:23:27,424 INFO
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict()
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity']
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity']
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'calculate_grid_size', 'VOXEL_SIZE': [0.16, 0.16, 0.16]}]
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.BASE_CONFIG: cfgs/dataset_configs/kitti_dataset.yaml
2022-11-19 21:23:27,425 INFO
cfg.DATA_CONFIG.IMAGE = edict()
2022-11-19 21:23:27,425 INFO cfg.DATA_CONFIG.IMAGE.ENABLED: True
2022-11-19 21:23:27,425 INFO
cfg.DATA_CONFIG.IMAGE.COLLATE = edict()
2022-11-19 21:23:27,425 INFO cfg.DATA_CONFIG.IMAGE.COLLATE.MODE: Pad
2022-11-19 21:23:27,425 INFO cfg.DATA_CONFIG.IMAGE.COLLATE.CONSTANT_VALUES: [0.485, 0.456, 0.406]
2022-11-19 21:23:27,425 INFO
cfg.DATA_CONFIG.DEPTH_MAP = edict()
2022-11-19 21:23:27,425 INFO cfg.DATA_CONFIG.DEPTH_MAP.ENABLED: True
2022-11-19 21:23:27,425 INFO cfg.DATA_CONFIG.DEPTH_MAP.DOWNSAMPLE_FACTOR: 4
2022-11-19 21:23:27,426 INFO
cfg.DATA_CONFIG.CALIB = edict()
2022-11-19 21:23:27,426 INFO cfg.DATA_CONFIG.CALIB.ENABLED: True
2022-11-19 21:23:27,426 INFO
cfg.MODEL = edict()
2022-11-19 21:23:27,426 INFO cfg.MODEL.NAME: CaDDN
2022-11-19 21:23:27,426 INFO
cfg.MODEL.FFE = edict()
2022-11-19 21:23:27,426 INFO cfg.MODEL.FFE.NAME: DepthFFE
2022-11-19 21:23:27,426 INFO
cfg.MODEL.FFE.DDN = edict()
2022-11-19 21:23:27,426 INFO cfg.MODEL.FFE.DDN.NAME: DDNDeepLabV3
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.DDN.BACKBONE_NAME: ResNet101
2022-11-19 21:23:27,427 INFO
cfg.MODEL.FFE.DDN.ARGS = edict()
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.DDN.ARGS.feat_extract_layer: layer1
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.DDN.ARGS.pretrained_path: ../checkpoints/deeplabv3_resnet101_coco-586e9e4e.pth
2022-11-19 21:23:27,427 INFO
cfg.MODEL.FFE.CHANNEL_REDUCE = edict()
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.CHANNEL_REDUCE.in_channels: 256
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.CHANNEL_REDUCE.out_channels: 64
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.CHANNEL_REDUCE.kernel_size: 1
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.CHANNEL_REDUCE.stride: 1
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.CHANNEL_REDUCE.bias: False
2022-11-19 21:23:27,428 INFO
cfg.MODEL.FFE.DISCRETIZE = edict()
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DISCRETIZE.mode: LID
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DISCRETIZE.num_bins: 80
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DISCRETIZE.depth_min: 2.0
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DISCRETIZE.depth_max: 46.8
2022-11-19 21:23:27,428 INFO
cfg.MODEL.FFE.DDN_LOSS = edict()
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DDN_LOSS.weight: 3.0
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DDN_LOSS.alpha: 0.25
2022-11-19 21:23:27,429 INFO cfg.MODEL.FFE.DDN_LOSS.gamma: 2.0
2022-11-19 21:23:27,429 INFO cfg.MODEL.FFE.DDN_LOSS.fg_weight: 13
2022-11-19 21:23:27,429 INFO cfg.MODEL.FFE.DDN_LOSS.bg_weight: 1
2022-11-19 21:23:27,429 INFO
cfg.MODEL.F2V = edict()
2022-11-19 21:23:27,429 INFO cfg.MODEL.F2V.NAME: FrustumToVoxel
2022-11-19 21:23:27,429 INFO
cfg.MODEL.F2V.SAMPLER = edict()
2022-11-19 21:23:27,429 INFO cfg.MODEL.F2V.SAMPLER.mode: bilinear
2022-11-19 21:23:27,429 INFO cfg.MODEL.F2V.SAMPLER.padding_mode: zeros
2022-11-19 21:23:27,429 INFO
cfg.MODEL.MAP_TO_BEV = edict()
2022-11-19 21:23:27,430 INFO cfg.MODEL.MAP_TO_BEV.NAME: Conv2DCollapse
2022-11-19 21:23:27,430 INFO cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 64
2022-11-19 21:23:27,430 INFO
cfg.MODEL.MAP_TO_BEV.ARGS = edict()
2022-11-19 21:23:27,430 INFO cfg.MODEL.MAP_TO_BEV.ARGS.kernel_size: 1
2022-11-19 21:23:27,430 INFO cfg.MODEL.MAP_TO_BEV.ARGS.stride: 1
2022-11-19 21:23:27,430 INFO cfg.MODEL.MAP_TO_BEV.ARGS.bias: False
2022-11-19 21:23:27,430 INFO
cfg.MODEL.BACKBONE_2D = edict()
2022-11-19 21:23:27,430 INFO cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
2022-11-19 21:23:27,431 INFO cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [10, 10, 10]
2022-11-19 21:23:27,431 INFO cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [2, 2, 2]
2022-11-19 21:23:27,431 INFO cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [64, 128, 256]
2022-11-19 21:23:27,431 INFO cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2, 4]
2022-11-19 21:23:27,431 INFO cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [128, 128, 128]
2022-11-19 21:23:27,431 INFO
cfg.MODEL.DENSE_HEAD = edict()
2022-11-19 21:23:27,431 INFO cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle
2022-11-19 21:23:27,431 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2022-11-19 21:23:27,431 INFO cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Car', 'anchor_sizes': [[3.9, 1.6, 1.56]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.6, 'unmatched_threshold': 0.45}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.8, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.76, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}]
2022-11-19 21:23:27,432 INFO
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
2022-11-19 21:23:27,433 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict()
2022-11-19 21:23:27,433 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2022-11-19 21:23:27,434 INFO
cfg.MODEL.POST_PROCESSING = edict()
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
2022-11-19 21:23:27,434 INFO
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict()
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.01
2022-11-19 21:23:27,435 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2022-11-19 21:23:27,435 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2022-11-19 21:23:27,435 INFO
cfg.OPTIMIZATION = edict()
2022-11-19 21:23:27,435 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4
2022-11-19 21:23:27,435 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 80
2022-11-19 21:23:27,435 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2022-11-19 21:23:27,435 INFO cfg.OPTIMIZATION.LR: 0.001
2022-11-19 21:23:27,435 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.PCT_START: 0.4
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.LR_WARMUP: False
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2022-11-19 21:23:27,437 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2022-11-19 21:23:27,437 INFO cfg.TAG: CaDDN
2022-11-19 21:23:27,437 INFO cfg.EXP_GROUP_PATH: content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/cfgs/kitti_models
2022-11-19 21:23:28,525 INFO Loading KITTI dataset
2022-11-19 21:23:28,526 INFO Total samples for KITTI dataset: 0
Traceback (most recent call last):
File "/content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/train.py", line 198, in
main()
File "/content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/train.py", line 112, in main
total_epochs=args.epochs
File "/content/gdrive/MyDrive/IITH_VCA/CaDDN/pcdet/datasets/init.py", line 69, in build_dataloader
drop_last=False, sampler=sampler, timeout=0
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 213, in init
sampler = RandomSampler(dataset)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/sampler.py", line 94, in init
"value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0
Screen shots of the same:
Error screen shot:
2022-11-19 21:23:28,526 INFO Total samples for KITTI dataset: 0,
It seems you don't load the data.
I have encountered the same problem. Have you resolved it?