PWA: Part-based weighting Aggregation of Deep Convolutional Features
for Image Retrieval
DESCRIPTIONS
PWA is an unsupervised method for image retrieval. The idea of PWA is that,
Each feature map is activated by different parts or patterns of objects. We
can use feature maps as part detectors to highlight the discriminative parts
of objects and suppress the noise of background.
The PWA weights are computed as follows. We first compute variances for
each dimension of the pooled descriptor among the whole dataset. Then we
select 25 channels (~17%) with the largest variances. We regard feature
maps corresponding to those channels as part detectors since the responses
of them are significantly different among the various objects. The PWA
weights are the l2-normalized and sqrt-normalized selected spatial maps.
We compute one weighted-pooled descriptor for each one of the PWA weights
in turn by weighted sum-pool the deep descriptors. Therefore, 150
D-dimensional weighted-pooled descriptors are selected. We concatenate
those into a long vector and perform l2-normalized, PCA whitening to
4096-dimensional, which is the final descriptor for retrieval.
We learn the PCA whitening parameters from a separate set of images. To be
comparable with related works, we learn the PCA whitening parameters on
Oxford when testing on Paris and vice versa, as accustomed.
We used cropped images for query, i.e., we regard the cropped region as
input for the CNN and extract featrures.
REFERENCE
J. Xu, C. Shi, C. Qi, C. Wang, and B. Xiao. Part-based weighting
aggregation of deep convolutional features for image retrieval. arXiv
preprint arXiv: 1705.01247.
PREREQUIREMENTS
Caffe with Python interface supported.
Python2.7 with Numpy, PIL, sklearn supported.
LAYOUT
Data
./data/oxford5k/
./data/oxford5k/channels.npy # Selected channels
./data/oxford5k/conv/ # Convolution feature maps
./data/oxford5k/conv/vgg16/all/
./data/oxford5k/conv/vgg16/crop/
./data/oxford5k/groundtruth/ # 55*4=220 query groundtruth
./data/oxford5k/image/ # Raw .jpg images
./data/oxford5k/image/all/ # 5063 database images
./data/oxford5k/image/crop/ # 55 crop query images
./data/paris6k/
# Similar to oxford5k, with 6392 database and 55 query images
Documentations
./doc/ # Automatically generated documents
./README # This file
Library
./lib/ # Third-party library files
Source Code
./src/get/ # Extract crop query and pool5 features
./src/pwa/ # Compute PWA descriptor and evaluate
USAGE
Step 1. Extract crop query images and extract pool5 features.
# Extract cropped query images.
$ ./src/get/get_query.py --dataset oxford5k
$ ./src/get/get_query.py --dataset paris6k
# Extract pool5 features.
$ ./src/get/get_conv.py --dataset oxford5k --model vgg16 --gpu 0
$ ./src/get/get_conv.py --dataset paris6k --model vgg16 --gpu 0
Step 2. Compute PWA descriptors and evaluate.
# Select channels.
$ ./src/pwa/select_channels.py --dataset oxford5k --model vgg16
--channels 25
$ ./src/pwa/select_channels.py --dataset paris6k --model vgg16
--channels 25
# Evaluate.
$ ./src/pwa/evaluate.py --test oxford5k --via paris6k --model vgg16
--dim 4096
$ ./src/pwa/evaluate.py --test paris6k --via oxford5k --model vgg16
--dim 4096
AUTHOR
Hao Zhang: zhangh0214@gmail.com
LICENSE
CC BY-SA 3.0
NOTE
20 images out of 6412 Paris images are broken. As a common practice,
we manually removed them:
paris_louvre_000136.jpg
paris_louvre_000146.jpg
paris_moulinrouge_000422.jpg
paris_museedorsay_001059.jpg
paris_notredame_000188.jpg
paris_pantheon_000284.jpg
paris_pantheon_000960.jpg
paris_pantheon_000974.jpg
paris_pompidou_000195.jpg
paris_pompidou_000196.jpg
paris_pompidou_000201.jpg
paris_pompidou_000467.jpg
paris_pompidou_000640.jpg
paris_sacrecoeur_000299.jpg
paris_sacrecoeur_000330.jpg
paris_sacrecoeur_000353.jpg
paris_triomphe_000662.jpg
paris_triomphe_000833.jpg
paris_triomphe_000863.jpg
paris_triomphe_000867.jpg