About voc_2012_val_instance.json

Question

About voc_2012_val_instance.json

scott870430 opened this issue 3 years ago · comments

Hi, I am interested in your project.
I ran into some problems:

following README.md

python3 projects/WSL/tools/proposal_convert.py voc_2012_val_instance datasets/proposals/MCG-Pascal-Segmentation_trainvaltest_2012-proposals datasets/proposals/mcg_voc_2012_val_instance_segmentation_d2

I get error FileNotFoundError: [Errno 2] No such file or directory: 'datasets/VOC_SBD/annotations/voc_2012_val_instance.json'
Also, there are only voc_2012_val_panoptic.json, voc_2012_train_panoptic.json, sbd_9118_panoptic.json, three files in datasets/VOC_SBD/annotations/ folder.
Am i missing something?

Is proposal_generator use in Panoptic Segmentation? Or use the precomputed_proposal (MCG?)
Is roi_heads for instance segmentation and sem_seg_head for semanctic segmentation?
If I want to perform instance segmentation, I just need to remove the sem_seg_head?

Thank you for your help!

沈云航 Yunhang Shen · Answer 1 · Wed Jun 30 2021 13:37:07 GMT+0800 (China Standard Time)

Thanks for you interesting.

It seems we missed a script file.
I have added it to our code in the new commit.
Please use the new script file to generate voc_2012_val_instance.json, voc_2012_train_instance.json, sbd_9118_instance.json:

python3 projects/WSL/tools/convert_voc2012_and_sbd_instance.py

We use the precomputed MCG proposal.
The roi_heads is for instance segmentation and sem_seg_head is for semanctic segmentation.
To perform instance segmentation, we need adjust the configure file by removing sem_seg_head and changing Dataset to TRAIN: ('voc_2012_train_instance', 'sbd_9118_instance') and TEST: ('voc_2012_val_instance',).
And it should also set PS_ON: False, which turns off panoptic segmentation.
We will plan to update new configure files for instance segmentation only.

Chung-Hsuan Cheng · Answer 2 · Wed Jun 30 2021 16:37:58 GMT+0800 (China Standard Time)

Thanks, I will try it.

Another question, what is SP_ON mean in config?

沈云航 Yunhang Shen · Answer 3 · Thu Jul 01 2021 00:37:23 GMT+0800 (China Standard Time)

The MCG segmentation proposals are based on superpixels.
Thus, we need to deal with superpixel by setting SP_ON: True.

Chung-Hsuan Cheng · Answer 4 · Tue Jul 06 2021 00:25:38 GMT+0800 (China Standard Time)

Thank you for the reply!

I traced your code, and found that SP_ON work in train_net.py

And I checked wsl.data build_detection_train_loader and detectron2.data build_detection_train_loader, I didn't find any different of them. And also parameter LOAD_PROPOSALS in config didn't update in add_wsl_config? So the model will not use proposal in training?
Sorry, I am new in detectron2 project framework, maybe I misunderstood some places...

Is superpixel means proposal in your project? As far as I know, the proposal provided by mcg is called superpixel. Or you did other operating for superpixel?
Will you provide pre-trained weight for panoptic segmentation / instance segmentation?

沈云航 Yunhang Shen · Answer 5 · Thu Jul 08 2021 15:40:05 GMT+0800 (China Standard Time)

Both build_detection_train_loader functions are the same in detectron2 and wsl. But load_proposals_into_datasets functions are different. And load_proposals_into_datasets is used by get_detection_dataset_dicts, and get_detection_dataset_dicts is called by _train_loader_from_config, and _train_loader_from_config is need by build_detection_train_loader.
We use proposals during training. LOAD_PROPOSALS is set to True in the parent configure file PascalVOC-Detection/oicr_WSR_18_DC5_1x.yaml.
Superpixel is used to generate proposals. We use the superpixel from mcg.
Yes, I will try to find some pre-trained weight in previous experiments.

Chung-Hsuan Cheng · Answer 6 · Thu Jul 08 2021 16:09:36 GMT+0800 (China Standard Time)

Thank you for the reply!

Why use superpixel instead of proposal from mcg?

Which GPU did you use for training? And how much time is usually spent on training?
I try to reproduce your work. I run the code on three 2080ti GPUs, and it looks like it will take nearly a week.
I also encountered some errors like Original error was: PyCapsule_Import could not import module "datetime", BlockingIOError: [Errno 11] Resource temporarily unavailable...
Is this caused by insufficient performance of my machine?

沈云航 Yunhang Shen · Answer 7 · Fri Jul 09 2021 17:17:15 GMT+0800 (China Standard Time)

Hi,

MCG uses superpixel to generate proposals, and we follow MCG to generate proposals during training with their superpixel.
Our training time is about 10 days in four 1080ti for 50,000 iterations.
Here is two logs for 100,000 iterations in MS COCO and 50,000 iterations in PASCAL VOC:
log_coco.txt
log_voc.txt
To reproduce the results of PASCAL VOC in paper, one should increase the number of iterations to 100,000 in configure file of PASCAL VOC.
For MS COCO, this repository has better results than that reported in the paper, as we fixed some bugs.

Chung-Hsuan Cheng · Answer 8 · Fri Jul 09 2021 20:03:37 GMT+0800 (China Standard Time)

Thank you for the reply!

It seems that I can have the same results as you when I reach 5000 iterations. However, I encountered ```runtimeerror: can't start new thread`` at 6000 iterations when testing... (The above errors also occurred during testing or start testing)

It seems that the thread of my server is not enough for testing? Or the program has too many threads and it crashes. I didn’t find any relevant information on detectron2’s github.

Is there any solution, or can these errors be avoided?

Thanks.

Chung-Hsuan Cheng · Answer 9 · Fri Jul 09 2021 21:30:18 GMT+0800 (China Standard Time)

About instance segmentation, what do you mean we need adjust the configure file by removing sem_seg_head?
In config remove the SEM_SEG_HEAD?

MODEL:
  META_ARCHITECTURE: "GeneralizedMCNNWSL"
  MASK_ON: True
  #SEM_SEG_HEAD:
  #  NAME: "TwoClassHead"
  #  IN_FEATURES: ["res5",]
  #  NUM_CLASSES: 2
  ROI_HEADS:
    NAME: "JTSMROIHeads"
  ROI_BOX_HEAD:
    POOLER_TYPE: "MOIPool"
  ROI_MASK_HEAD:
    NAME: "MaskRCNNConvUpsampleWSLHead"
    #NAME: "MaskRCNNConvUpsampleHead"
  BACKBONE:
    #NAME: "build_wsl_resnet_backbone"
    NAME: "build_wsl_resnet_v2_backbone"
    FREEZE_AT: 5

After removed SEM_SEG_HEAD, I got error

  File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/build.py", line 21, in build_model
    model = META_ARCH_REGISTRY.get(meta_arch)(cfg)
  File "/home/chcheng/research/JTSM/detectron2/config/config.py", line 181, in wrapped
    explicit_args = _get_args_from_config(from_config_func, *args, **kwargs)
  File "/home/chcheng/research/JTSM/detectron2/config/config.py", line 238, in _get_args_from_config
    ret = from_config_func(*args, **kwargs)
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/meta_arch/mcnn.py", line 99, in from_config
    "sem_seg_head": build_sem_seg_head(cfg, backbone.output_shape()),
  File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/semantic_seg.py", line 100, in build_sem_seg_head
    return SEM_SEG_HEADS_REGISTRY.get(name)(cfg, input_shape)
  File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/semantic_seg.py", line 130, in __init__
    1, int(np.log2(feature_strides[in_feature]) - np.log2(self.common_stride))
KeyError: 'p2'

Thanks.

沈云航 Yunhang Shen · Answer 10 · Mon Jul 12 2021 13:37:58 GMT+0800 (China Standard Time)

Thank you for the reply!

It seems that I can have the same results as you when I reach 5000 iterations. However, I encountered ```runtimeerror: can't start new thread`` at 6000 iterations when testing... (The above errors also occurred during testing or start testing)

It seems that the thread of my server is not enough for testing? Or the program has too many threads and it crashes. I didn’t find any relevant information on detectron2’s github.

Is there any solution, or can these errors be avoided?

Thanks.

I don't have this error, maybe you can try to reduce the number of threads.

沈云航 Yunhang Shen · Answer 11 · Mon Jul 12 2021 13:58:35 GMT+0800 (China Standard Time)

About instance segmentation, what do you mean we need adjust the configure file by removing sem_seg_head?
In config remove the SEM_SEG_HEAD?

MODEL:
  META_ARCHITECTURE: "GeneralizedMCNNWSL"
  MASK_ON: True
  #SEM_SEG_HEAD:
  #  NAME: "TwoClassHead"
  #  IN_FEATURES: ["res5",]
  #  NUM_CLASSES: 2
  ROI_HEADS:
    NAME: "JTSMROIHeads"
  ROI_BOX_HEAD:
    POOLER_TYPE: "MOIPool"
  ROI_MASK_HEAD:
    NAME: "MaskRCNNConvUpsampleWSLHead"
    #NAME: "MaskRCNNConvUpsampleHead"
  BACKBONE:
    #NAME: "build_wsl_resnet_backbone"
    NAME: "build_wsl_resnet_v2_backbone"
    FREEZE_AT: 5

After removed SEM_SEG_HEAD, I got error

  File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/build.py", line 21, in build_model
    model = META_ARCH_REGISTRY.get(meta_arch)(cfg)
  File "/home/chcheng/research/JTSM/detectron2/config/config.py", line 181, in wrapped
    explicit_args = _get_args_from_config(from_config_func, *args, **kwargs)
  File "/home/chcheng/research/JTSM/detectron2/config/config.py", line 238, in _get_args_from_config
    ret = from_config_func(*args, **kwargs)
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/meta_arch/mcnn.py", line 99, in from_config
    "sem_seg_head": build_sem_seg_head(cfg, backbone.output_shape()),
  File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/semantic_seg.py", line 100, in build_sem_seg_head
    return SEM_SEG_HEADS_REGISTRY.get(name)(cfg, input_shape)
  File "/home/chcheng/research/JTSM/detectron2/modeling/meta_arch/semantic_seg.py", line 130, in __init__
    1, int(np.log2(feature_strides[in_feature]) - np.log2(self.common_stride))
KeyError: 'p2'

Thanks.

Sorry, it is my wrong.
To only perform instance segmentation for PASCAL VOC, you can try to set PS_ON`` to False, and leave SEM_SEG_HEAD``` unchanged.
More mofidies are need for MS COCO.

Chung-Hsuan Cheng · Answer 12 · Mon Jul 12 2021 14:11:44 GMT+0800 (China Standard Time)

Thank you for the reply!

Is there any way to reduce the number of threads from the code or I need to add code for reduce threads?

Thanks, I will try instance segmentation for VOC.

Regarding instance segmentation on MS COCO, it need to remove SEM_SEG_HEAD in config?
Btw, will you upload config for instance segmentation recently?

沈云航 Yunhang Shen · Answer 13 · Mon Jul 12 2021 14:25:41 GMT+0800 (China Standard Time)

You should change the codes in this line.

For MS COCO, you also need to set SEM_SEG_HEAD as PASCAL VOC.
I will plan to upload new config.

Thank you for the reply!

Is there any way to reduce the number of threads from the code or I need to add code for reduce threads?

Thanks, I will try instance segmentation for VOC.

Regarding instance segmentation on MS COCO, it need to remove SEM_SEG_HEAD in config?
Btw, will you upload config for instance segmentation recently?

Chung-Hsuan Cheng · Answer 14 · Mon Jul 12 2021 14:32:05 GMT+0800 (China Standard Time)

Thanks, I will try it.

SEM_SEG_HEAD is not use in instance segmentation, but for MS COCO, I need to set SEM_SEG_HEAD as PASCAL VOC?
Am i misunderstanding something?

Thanks!

沈云航 Yunhang Shen · Answer 15 · Mon Jul 12 2021 17:46:39 GMT+0800 (China Standard Time)

Yes, another way is remove semantic segmentation part from mcnn.py.

Thanks, I will try it.

SEM_SEG_HEAD is not use in instance segmentation, but for MS COCO, I need to set SEM_SEG_HEAD as PASCAL VOC?
Am i misunderstanding something?

Thanks!

Chung-Hsuan Cheng · Answer 16 · Tue Jul 13 2021 14:36:04 GMT+0800 (China Standard Time)

Hi, following your instance segmentation setting, I got error with

  File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 799, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/meta_arch/mcnn.py", line 217, in forward
    images, features, proposals, gt_instances, gt_sem_seg, superpixels
  File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/roi_heads_jtsm.py", line 540, in forward
    losses = self._forward_box(features, proposals)
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/roi_heads_jtsm.py", line 657, in _forward_box
    losses = self.box_predictor.losses(predictions, proposals, self.gt_classes_img_oh)
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 692, in losses
    gt_classes_img_oh,
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 404, in losses
    return {"loss_cls": self.binary_cross_entropy_loss()}
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 361, in binary_cross_entropy_loss
    self.predict_probs_img(), self.gt_classes_img_oh, reduction="mean"
  File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/functional.py", line 2886, in binary_cross_entropy
    "Please ensure they have the same size.".format(target.size(), input.size())
ValueError: Using a target size (torch.Size([1, 20])) that is different to the input size (torch.Size([1, 21])) is deprecated. Please ensure they have the same size.

It seems that the model will still output 21 categories (including semantic?) but the ground truth only 20 categories (for instance segmentation?).

In here, the box prediction will add num_classes_stuff - 1, should modify

SEM_SEG_HEAD:
    NAME: "TwoClassHead"
    IN_FEATURES: ["res5",]
    NUM_CLASSES: 2

NUM_CLASSES to 1? After modified to one, the code will work, but I am not sure about that the modify is correct?

Some questions about the architecture, the panoptic mining branch in paper is implement in ROI_HEADS in code?
What is pgt and mist meaning in here?

For instance segmentation, can I remove this line for speed up training?

Chung-Hsuan Cheng · Answer 17 · Wed Jul 14 2021 13:42:15 GMT+0800 (China Standard Time)

Something about Panoptic Segmentation question, as I know, the stuff class in Panoptic Segmentation amorphous regions of similar texture or material such as grass, sky, road.

In your work, you only recognize background, not the grass, sky... etc. in pascal voc 2012 dataset?
Is it because of the lack of relevant annotations that lead you to set up like this?
In pascal voc 2012, are all pixels other than instance classified as background?

Thanks.

沈云航 Yunhang Shen · Answer 18 · Thu Jul 15 2021 11:17:17 GMT+0800 (China Standard Time)

Hi, following your instance segmentation setting, I got error with

  File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 799, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/meta_arch/mcnn.py", line 217, in forward
    images, features, proposals, gt_instances, gt_sem_seg, superpixels
  File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/roi_heads_jtsm.py", line 540, in forward
    losses = self._forward_box(features, proposals)
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/roi_heads_jtsm.py", line 657, in _forward_box
    losses = self.box_predictor.losses(predictions, proposals, self.gt_classes_img_oh)
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 692, in losses
    gt_classes_img_oh,
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 404, in losses
    return {"loss_cls": self.binary_cross_entropy_loss()}
  File "/home/chcheng/research/JTSM/projects/WSL/wsl/modeling/roi_heads/fast_rcnn_tsm.py", line 361, in binary_cross_entropy_loss
    self.predict_probs_img(), self.gt_classes_img_oh, reduction="mean"
  File "/home/chcheng/Env/Env_JTSM/lib/python3.6/site-packages/torch/nn/functional.py", line 2886, in binary_cross_entropy
    "Please ensure they have the same size.".format(target.size(), input.size())
ValueError: Using a target size (torch.Size([1, 20])) that is different to the input size (torch.Size([1, 21])) is deprecated. Please ensure they have the same size.

It seems that the model will still output 21 categories (including semantic?) but the ground truth only 20 categories (for instance segmentation?).

In here, the box prediction will add num_classes_stuff - 1, should modify

SEM_SEG_HEAD:
    NAME: "TwoClassHead"
    IN_FEATURES: ["res5",]
    NUM_CLASSES: 2

NUM_CLASSES to 1? After modified to one, the code will work, but I am not sure about that the modify is correct?

I think the modification is right. But you should carefull check whether stuff classes are involved during mining, as it will affect the mined results for thing classes.
We prefer to manually remove the segmantic part to performan instance segmentation.

Some questions about the architecture, the panoptic mining branch in paper is implement in ROI_HEADS in code?

Yes. ROI_HEADS has both panoptic panoptic mining branch and instance segmentation branch.
The default semantic segmentation branch is SemSegFPNHead.

What is pgt and mist meaning in here?

pgt is pseudo-ground-truth and mist is multiple instance self-training.

For instance segmentation, can I remove this line for speed up training?

This line can be removed.

沈云航 Yunhang Shen · Answer 19 · Thu Jul 15 2021 11:19:36 GMT+0800 (China Standard Time)

Something about Panoptic Segmentation question, as I know, the stuff class in Panoptic Segmentation amorphous regions of similar texture or material such as grass, sky, road.

In your work, you only recognize background, not the grass, sky... etc. in pascal voc 2012 dataset?

Yes.

Is it because of the lack of relevant annotations that lead you to set up like this?

Yes.

In pascal voc 2012, are all pixels other than instance classified as background?

Yes.

Thanks.

You are welcome

Chung-Hsuan Cheng · Answer 20 · Thu Jul 15 2021 12:09:36 GMT+0800 (China Standard Time)

Thank you for the reply!

We prefer to manually remove the segmantic part to performan instance segmentation.
If I don't remove the semantic part, it will still compute loss for semantic? Does the modification for instance segmentation just change the calculation method of evaluation? (Panoptic Segmentation -> instance segmentation?)

If I want to remove the semantic segmentation part, is there anything else that needs to be removed besides this line?

Thanks.

沈云航 Yunhang Shen · Answer 21 · Thu Jul 15 2021 13:49:06 GMT+0800 (China Standard Time)

I think removing semantic segmentation part in mcnn.py is enough.

Chung-Hsuan Cheng · Answer 22 · Tue Jul 20 2021 16:06:24 GMT+0800 (China Standard Time)

Hi, some question about architecture, I found mask_head and mask_refinery_0 in architecture, these two layers are also MaskRCNNConvUpsampleWSLHead, and their architecture is the same... (the config you provided also has these two layers)
Why we need these two layer?
Thanks.

沈云航 Yunhang Shen · Answer 23 · Tue Jul 20 2021 16:47:03 GMT+0800 (China Standard Time)

Hi, some question about architecture, I found mask_head and mask_refinery_0 in architecture, these two layers are also MaskRCNNConvUpsampleWSLHead, and their architecture is the same... (the config you provided also has these two layers)
Why we need these two layer?
Thanks.

Hi,
mask_refinery_0 is used to refine mask results, which takes mask predictions from mask_head as supervison.

Chung-Hsuan Cheng · Answer 24 · Tue Jul 20 2021 16:51:46 GMT+0800 (China Standard Time)

Is mask_refinery_0 the instance segmentation branch?
And where is mask_head in architecture?
Isn't the instance segmentation branch take output from grabcut as supervision?

Chung-Hsuan Cheng · Answer 25 · Tue Jul 20 2021 22:20:26 GMT+0800 (China Standard Time)

Hi, I found that the training process program will also be interrupted in the config you provided.

RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 1800000ms for recv operation to complete

Why is this happening?
Thanks.

沈云航 Yunhang Shen · Answer 26 · Sat Oct 16 2021 21:23:05 GMT+0800 (China Standard Time)

Is mask_refinery_0 the instance segmentation branch? And where is mask_head in architecture? Isn't the instance segmentation branch take output from grabcut as supervision?

Hi, mask_refinery_0 and mask_head are both in the instance segmentation branch.
And mask_refinery_0 takes the predictions of mask_head as supervision.
We can also add more refinement.

沈云航 Yunhang Shen · Answer 27 · Sat Oct 16 2021 21:25:00 GMT+0800 (China Standard Time)

Hi, I found that the training process program will also be interrupted in the config you provided.
RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 1800000ms for recv operation to complete
Why is this happening? Thanks.

I haven't met such issues.
It seems to be the problem in distributed training.