praeclarumjj3 / OLIE

Image editing using SOLOv2 instance maps. Project at SHI Lab.

Object Level Image Editor (OLIE)

This code is derived from the AdelaiDet Project

Contents

1. Overview
2. Setup Instructions
3. Repository Overview
4. Reproduction
5. Experiments

1. Overview

This repo contains code for a SOLOv2 based Object Level Image Editor.

OLIE was aimed to reconstruct the original image with the objects removed (editing) by incorporating the mask features (produced by SOLOv2) as input to the reconstructor.

After experimentation for a few months, we scrapped this pipeline and moved on to a new one so the current code doesn't work well.

Results

For the results corresponding to different settings, please refer to the corresponding commit.

Desciption	Commit
Removing objects	33d9451
Image Reconstruction	2d24143

2. Setup Instructions

• First install Detectron2 following the official guide: INSTALL.md. Please use Detectron2 with commit id 9eb4831 if you have any issues related to Detectron2.

• Then build the project with:

git clone https://github.com/praeclarumjj3/OLIE.git
cd OLIE
python setup.py build develop

• Download the MS-COCO dataset:

wget http://images.cocodataset.org/zips/train2017.zip
wget http://images.cocodataset.org/zips/val2017.zip
wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip

• Unzip the files according to the following structure

OLIE
├── datasets
│   ├── coco
│   │   ├── annotations
│   │   ├── train2017
│   │   ├── val2017

• Download Pretrained weights (based on Resnet-50):

wget https://cloudstor.aarnet.edu.au/plus/s/chF3VKQT4RDoEqC/download -O SOLOv2_R50_3X.pth

wget https://cloudstor.aarnet.edu.au/plus/s/9w7b3sjaXvqYQEQ -O SOLOv2_R101_3X.pth

Name	inf. time	train. time	Mem	box AP	mask AP	download
SOLOv2_R50_3x	47ms	~25h(36 epochs)	3.7GB	-	37.6	model
SOLOv2_R101_3x	61ms	~30h(36 epochs)	4.7GB	-	39.0	model

3. Repository Overview

The repository is structured as follows:

• adet - Base code from the AdelaiDet repo.
• configs - Configuration files for SOLOv2.
• datasets - Dataset related scripts.
• modules - Contains python scripts for different modules.
• scripts - Contains shell scripts for running code.
• baselines - Results for image editing using baselines.

4. Reproduction

• Run the following command to run a demo and see visualization results:

$ sh scripts/demo.sh

• Run the following command to train the OLIE model for 30 epochs:

$ sh scripts/train.sh

• Run the following command to evaluate the OLIE model from checkpoints:

$ sh scripts/evaluate.sh

If you encounter an errors, install the pretty-errors package to see the beautified errors.

$ python -m pip install pretty_errors

$ python -m pretty_errors

5. Experiments

• The input image is of the BGR format.

• The model was trained for 30 Epochs with lr=1e-3. All experiments were conducted on the coco-val dataset.

• [Experiments Catalog]