Online Attention Accumulation

Try our PyTorch code.

This repository contains the original code and the links for data and pretrained models. Please see our Project Home for more details. If you have any questions about our paper "Integral Object Mining via Online Attention Accumulation", please feel free to contact Me (pt.jiang AT mail DOT nankai.edu.cn).
The idea about online accumulation may be usful for other problems and questions. Hope our work will bring any usefulness into your project.

Video to observe attention evolution.

Video to observe attention accumulation.

Thanks to the contribution from Lin-Hao Han for this video.

Pre-computed results
Pytorch re-implementations
Installation
Implementation
Citation

Pre-computed Results

We provide the pre-trained models, pre-computed attention maps and saliency maps for:

The pre-trained integral attention model. [link].
The pre-computed attention maps for OAA and OAA+.
The saliency maps used for proxy labels. [link]
The code for generating proxy segmentation labels can be download from this link.
The pre-trained vgg16-based segmentation models for OAA and OAA+.
CRF parameters: bi_w = 3, bi_xy_std = 67, bi_rgb_std = 4, pos_w = 1, pos_xy_std = 3.

Installation

1. Prerequisites

ubuntu 16.04
python 2.7 or python 3.x (adjust print function in *.py)
caffe dependence

2. Compilie caffe

git clone https://github.com/PengtaoJiang/OAA.git
cd OAA/
make all -j4 && make pycaffe

3. Download

Dataset

Download the VOCdevkit.tar.gz file and extract the voc data into data/ folder.

Init models

Download this model for initializing the classfication network. Move it to examples/oaa.
Download this model for initializing the VGG-based DeepLab-LargeFOV network. Move it to examples/seg.
Download this model for initializing the ResNet-based DeepLab-LargeFOV network. Move it to examples/seg.

Implementation

1. Attention Generation

First, train the classification network for accumulating attention,

cd examples/oaa/
./train.sh exp1 0

After OAA is finished, you can resize the cumulative attention maps to the size of original images by

cd exp1/
python res.py

(optional)
After OAA, you can train an integral attention model.
You need to perform serveal steps:
First, rename the cumulative attention maps,

cd exp1/
python res1.py
python eval.py 30000 0

Second, train the integral attention model,

cd examples/oaa/
./train.sh exp2 0

Third, generate attention maps from the integral attention model,

cd examples/oaa/exp2/
python eval.py 30000 0

2. Segmentation

We provide two Deeplab-LargeFOV versions, VGG16(examples/seg/exp1) and ResNet101(examples/seg/exp2).
After generating proxy labels, put them into data/VOCdevkit/VOC2012/.
Adjust the training list train_ins.txt,

cd examples/seg/exp1/
vim train_ins.txt

Train

cd examples/seg/
./train.sh exp1 0

Test

python eval.py 15000 0 exp1

If you want to use crf to smooth the segmentation results, you can download the crf code from this link.
Move the code the examples/seg/, compile it. Then uncomment line 175 and 176 in examples/seg/eval.py.
The crf parameters are in examples/seg/utils.py.

Pytorch Re-implementations

The pytorch code is coming.

Citation

If you use these codes and models in your research, please cite:

@inproceedings{jiang2019integral,   
      title={Integral Object Mining via Online Attention Accumulation},   
      author={Jiang, Peng-Tao and Hou, Qibin and Cao, Yang and Cheng, Ming-Ming and Wei, Yunchao and Xiong, Hong-Kai},   
      booktitle={Proceedings of the IEEE International Conference on Computer Vision},   
      pages={2070--2079},   
      year={2019} 
}

GuoleiSun / OAA