CBIR using VGG-RMAC feature
1.1 image retrieval and object localization for No. 1 - 5 query (top_k = 3)
The following table has 4 rows, 5 columns, each column is for one query. In each column, the first one is query image, followed with 3 retrieved ones.
You can click images to see full size image in
For more results (top_k=10), Please go to Demo.ipynb.
1.2 quantitative results on validation data
2. How to use
Step-by-step tutorial: Demo.ipynb
If you want to further develop based on this repository, you may refer to section 4 (not finished yet): design, implementation, features and discussion.
- Methodology introduction: [github] | [my blog] (recomended, for better view of mathematical notations and flowchart)
4. Design, implementation, discussion
- get db_feature_mat computed or loaded to memory.
- image-level content retrieval.
- object-level content retrieval.similar to image-level retrieval, but will preprocess image by masking * image with object bounding boxes and locate objects on top ranked images (if needed).
- index new images on the fly
- compute image-level feature embedding
- compute top ranked image to retreieve
- locate objects in retrieved image
- map image path string to corresponding cached feature path string
- read cached image feature if found, otherwise compute and then cache it
- read database feature matrix if found, otherwise compute and ten cache it
Currently, the implementation is slightly different from the design:
- Function names
- BOWFeatureExtractor is not implemented yet, coming soon. SIFT-BOW-CIBR
When doing object localization, we apply level supression, since I observed that otherwise the localization will prefer larger window. see line xx in xx.
The RMAC implementaion is minimal and more efficient than existing implementations. Moreover, I providde other option on pooling method ('RAAC') and regional aggregation (average, not tested).
Experiments show that Level 1 pooling is important for retrieval. If initial scale is 2, the performance get worse (see dev notebook)
- support index new image on the fly
- support automatic caching
- easy to extend framework
- add table of contents in README.md
- add dynamical indexing function
- unify SIFT-BOW/SIFT-TFIDF to the this framework.
- query expansion and reranking