"rectangle back" | "thin seat" | "two legs" | "opening arms" |
---|---|---|---|
![]() |
![]() |
![]() |
![]() |
This repo is a pytorch-lightning implementation of CVPR2022 paper (Oral) PartGlot.
PartGlot introduces a novel way to segment 3D parts from solely part referential language.
We reflect the human perception mechanism that the human attends to specific regions on objects when given information about that region. To exploit this fact, we train a neural listener on reference games, where the model has to differentiate a target shape among two distractors based on utterances describing the target.
We investigate how well the model can connect part names in the utterances to specific regions in the geometry of the 3D shapes by just solving classification tasks.
We achieved 79.4% instance mIoU on ShapeNet Chair class without any direct geometric supervision and fine-grained annotations.
Furthermore, we deomonstrate that the part information learned from language can be generalizable to Out-of-Distribution shape classes as well.
For instance, Table top is highly considered as Chair seat with 78.2% mIoU and Lamp base as Chair leg with 44.6% mIoU.
IoU(%)s on ShapeNet Chair dataset:
Method | Back | Seat | Leg | Arm | Average |
---|---|---|---|---|---|
Part-Name-Agnostic | 82.2 | 78.8 | 75.5 | 40.6 | 69.3 |
Part-Name-Aware | 84.9 | 83.6 | 78.9 | 70.4 | 79.4 |
For main requirements, we have tested the code with Python 3.8.0, CUDA 11.1 and Pytorch 1.9.0.
Clone the repo and install other libraries:
git clone https://github.com/63days/PartGlot
cd PartGlot
pip install -e .
We provide ready-to-use data and pre-trained weights here. Download data and put them in data
directory.
To test pre-trained PartGlots, download checkpoints from the link above and pass a path of the checkpoint:
python test.py model=pn_agnostic ckpt_path=checkpoints/pn_agnostic.ckpt save_dir=logs/pre_trained/pn_agnostic
python test.py model=pn_aware ckpt_path=checkpoints/pn_aware.ckpt save_dir=logs/pre_trained/pn_aware
It will measure IoUs per part and instance mIoU, and save predicted labels per shape in save_dir
.
You can train PartGlot in two ways: Part-Name-Agnostic and Part-Name-Aware.
python train.py model=pn_agnostic epochs=30 batch_size=64 trainer.gpus=[0]
To train a part-name-agnostic model, pass model=pn_agnostic
.
You can train with multi GPUs by passing a list of devie numbers, for instance trainer.gpus=[0,1,2]
.
python train.py model=pn_aware model.xnt_reg=true model.xnt_reg_weights=1e-2
To train a model with part-name-aware setup, change model=pn_aware
.
You can control hyper-parameters related to a cross-entropy regularization by changing model.xnt_reg
and model.xnt_reg_weights
.
If you find our work useful, please consider citing our paper:
@inproceedings{koo2022partglot,
title={PartGlot: Learning Shape Part Segmentation from Language Reference Games},
author={
Koo, Juil and
Huang, Ian and
Achlioptas, Panos and
Guibas, Leonidas J and
Sung, Minhyuk
},
booktitle={Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2022}
}