nightowlowl / OV_PARTS

[NeurIPS 2023] OV-PARTS: Towards Open-Vocabulary Part Segmentation

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

OV-PARTS: Towards Open-Vocabulary Part Segmentation

This codebase contains code for baselines used in the paper "OV-PARTS: Towards Open-Vocabulary Part Segmentation".

  • Enviroment

    torch==1.13.1
    torchvision==0.14.1
    detectron2==0.6 #Following https://detectron2.readthedocs.io/en/latest/tutorials/install.html to install it and some required packages
    mmcv==1.7.1

    FurtherMore, install the modified clip package.

    cd third_party/CLIP
    python -m pip install -e .
  • Data Preparation

    We provide the download links for the two benchmark datasets in OV-PARTS: the refined Pascal-Part-116 and ADE20K-Part-234 datasets.

    [Pascal-Part-116] [ADE20K-Part-234]

    After downloading the datasets, please extract the files by running the following command and place the extracted folder under the "Datasets" directory.

    tar -xzf PascalPart116.tar.gz
    tar -xzf ADE20KPart234.tar.gz

    The Datasets folder should follow this structure:

    Datasets/
    ├─Pascal-Part-116/
    │ ├─train_16shot.json
    │ ├─images/
    │ │ ├─train/
    │ │ └─val/
    │ ├─annotations_detectron2_obj/
    │ │ ├─train/
    │ │ └─val/
    │ └─annotations_detectron2_part/
    │   ├─train/
    │   └─val/
    └─ADE20K-Part-234/
      ├─images/
      │ ├─training/
      │ ├─validation/
      ├─train_16shot.json
      ├─ade20k_instance_train.json
      ├─ade20k_instance_val.json
      └─annotations_detectron2_part/
        ├─training/
        └─validation/
  • Training

    • Training the two-stage baseline ZSseg+.

      Please first download the clip model fintuned with CPTCoOp.

      Then run the training command:

      # For ZSSeg+.
      python train_net.py --num-gpus 8 --config-file configs/${SETTING}/zsseg+_R50_coop_${DATASET}.yaml
    • Training the one-stage baselines CLIPSeg and CATSeg.

      Please first download the pre-trained object models of CLIPSeg and CATSeg and place them under the "pretrain_weights" directory.

      Models Pre-trained checkpoint
      CLIPSeg download
      CATSeg download

      Then run the training command:

      # For CATseg.
      python train_net.py --num-gpus 8 --config-file configs/${SETTING}/catseg_${DATASET}.yaml
      
      # For CLIPseg.
      python train_net.py --num-gpus 8 --config-file configs/${SETTING}/clipseg_${DATASET}.yaml
  • Evaluation

    We provide the trained weights for the three baseline models reported in the paper.

    Models Setting Pascal-Part-116 checkpoint ADE20K-Part-234 checkpoint
    ZSSeg+ Zero-shot download download
    CLIPSeg Zero-shot download download
    CatSet Zero-shot download download
    CLIPSeg Few-shot download download
    CLIPSeg cross-dataset - download

    To evaluate the trained models, add --eval-only to the training command. For example:

      python train_net.py --num-gpus 8 --config-file configs/${SETTING}/catseg_${DATASET}.yaml --eval-only MODEL.WEIGHTS ${WEIGHT_PATH}
  • Acknowledgement

    We would like to express our gratitude to the open-source projects and their contributors, including ZSSeg, CATSeg and CLIPSeg. Their valuable work has greatly contributed to the development of our codebase.

About

[NeurIPS 2023] OV-PARTS: Towards Open-Vocabulary Part Segmentation


Languages

Language:Python 100.0%Language:Makefile 0.0%