CBIR using VGG-RMAC feature
1. Results
1.1 image retrieval and object localization for No. 1 - 5 query (top_k = 3)
The following table has 4 rows, 5 columns, each column is for one query. In each column, the first one is query image, followed with 3 retrieved ones.
You can click images to see full size image in result directory.
For more results (top_k=10), Please go to Demo.ipynb.
| Q1 | Q2 | Q3 | Q4 | Q5 |
|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
1.2 quantitative results on validation data
TBA.
2. How to use
Step-by-step tutorial: Demo.ipynb
Dataset(google drive): pg_data | supplementary
If you want to further develop based on this repository, you may refer to section 4 (not finished yet): design, implementation, features and discussion.
3. Methodology
- Methodology introduction: [github] | [my blog] (recomended, for better view of mathematical notations and flowchart)
4. Design, implementation, discussion
4.1 design
SearchEngine
-
build()
- get db_feature_mat computed or loaded to memory.
-
retrieve_img()
- image-level content retrieval.
-
retrieve_object()
- object-level content retrieval.similar to image-level retrieval, but will preprocess image by masking * image with object bounding boxes and locate objects on top ranked images (if needed).
-
index_new_img()[pending]
- index new images on the fly
FeatureExtractor
-
compute_im_feature()
- compute image-level feature embedding
-
compute_top_matches()
- compute top ranked image to retreieve
-
compute_bb_mat()
- locate objects in retrieved image
-
_map_im_path_to_cache_path()
- map image path string to corresponding cached feature path string
-
get_im_feature_by_img_path()
- read cached image feature if found, otherwise compute and then cache it
-
get_db_feature_matrix()
- read database feature matrix if found, otherwise compute and ten cache it
Currently, the implementation is slightly different from the design:
- Function names
- BOWFeatureExtractor is not implemented yet, coming soon. SIFT-BOW-CIBR
4.2 implementation
-
When doing object localization, we apply level supression, since I observed that otherwise the localization will prefer larger window. see line xx in xx.
-
The RMAC implementaion is minimal and more efficient than existing implementations. Moreover, I providde other option on pooling method ('RAAC') and regional aggregation (average, not tested).
-
Experiments show that Level 1 pooling is important for retrieval. If initial scale is 2, the performance get worse (see dev notebook)
4.3 features
- support index new image on the fly
- support automatic caching
- easy to extend framework
5. TODO
- add table of contents in README.md
- add dynamical indexing function
- unify SIFT-BOW/SIFT-TFIDF to the this framework.
- query expansion and reranking