This repository contains the PyTorch implementation of the paper DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval.
- Python 3
- PyTorch >= 1.1
- Torchvision >= 0.4
- Clone this repo:
git clone https://github.com/mever-team/distill-and-select
cd distill-and-select
- You can install all the dependencies by
pip install -r requirements.txt
or
conda install --file requirements.txt
-
We provide our extracted features for all datasets to facilitate reproducibility for future research.
-
Download the feature files of the dataset you want:
-
All feature files are in HDF5 format
We provide the code for training and evaluation of our student models.
- To train a fine-grained student, run the
train_student.py
givenfine-grained
as value to the--student_type
argument, as in the following command:
python train_student.py --student_type fine-grained --experiment_path experiments/DnS_students --trainset_hdf5 /path/to/dns_100k.hdf5
- You can train an attention or binarization fine-grained students by setting either the
--attention
or--binarization
flags totrue
, respectively:
python train_student.py --student_type fine-grained --binarization true --experiment_path /path/to/experiment/ --trainset_hdf5 /path/to/dns_100k.hdf5
- To train a coarse-grained students, provide
coarse-grained
to the--student_type
argument:
python train_student.py --student_type coarse-grained --experiment_path /path/to/experiment/ --trainset_hdf5 /path/to/dns_100k.hdf5
- Provide one of the
teacher
,fg_att_student_iter1
,fg_att_student_iter2
to the--teacher
argument in odrder to train a student with a different teacher:
python train_student.py --teacher fg_att_student_iter2 --experiment_path /path/to/experiment/ --trainset_hdf5 /path/to/dns_100k.hdf5
-
Choose one of the
FIVR-5K
,FIVR-200K
,CC_WEB_VIDEO
,SVD
, orEVVE
datasets to evaluate your models. -
For the evaluation of the students, run the
evaluation_student.py
script by providing the path to the.pth
model to the--student_path
argument, as in the following command:
python evaluation_student.py --student_path experiments/DnS_students/model_fg_att_student.pth --dataset FIVR-5K --dataset_hdf5 /path/to/fivr_200k.hdf5
- If you don't pass any value to the
--student_path
, a pretrained model will be selected:
python evaluation_student.py --student_type fine-grained --attention true --dataset FIVR-5K --dataset_hdf5 /path/to/fivr_200k.hdf5
We also provide the code for training of the selector network and the evaluation of our overall DnS framework.
- To train a selector network, run the
train_selector.py
as in the following command:
python train_selector.py --experiment_path experiments/DnS_students --trainset_hdf5 /path/to/dns_100k.hdf5
- Provide different values to
--threshold
argument to train the selector network with different label functions.
- For the evaluation of the DnS framework, run the
evaluation_dns.py
script by providing the path to the.pth
model to the corresponding network arguments, as in the following command:
python evaluation_dns.py --selector_network_path experiments/DnS_students/model_selector_network.pth --dataset FIVR-5K --dataset_hdf5 /path/to/fivr_200k.hdf5
- If you don't pass any value to the network path argument, then the pretrained model will be selected. E.g. to evalute DnS with the Fine-grained Attention Student:
python evaluation_dns.py --attention true --dataset FIVR-5K --dataset_hdf5 /path/to/fivr_200k.hdf5
- Provide different values to
--percentage
argument to sent different number of video pairs for reranking to the Fine-grained student. Given the valueall
, it runs evaluation for all dataset percentages.
- Load our pretrained models as follows:
from model.feature_extractor import FeatureExtractor
from model.students import FineGrainedStudent, CoarseGrainedStudent
from model.selector import SelectorNetwork
# The feature extraction network used in out experiments
feature_extractor = FeatureExtractor(dims=512)
# Our Fine-grained Students
fg_att_student = FineGrainedStudent(pretrained=True, attention=True)
fg_bin_student = FineGrainedStudent(pretrained=True, binarization=True)
# Our Coarse-grained Students
cg_student = CoarseGrainedStudent(pretrained=True)
# Our Selector Networks
selector_att = SelectorNetwork(pretrained=True, attention=True)
selector_bin = SelectorNetwork(pretrained=True, binarization=True)
- First, extract video features by providing a video tensor to feature extractor (similar as here)
video_features = feature_extractor(video_tensor)
- Use the
index_video()
function providing video features to extract video representations for the student and selector networks
fg_features = fg_att_student.index_video(video_features)
cg_features = cg_student.index_video(video_features)
sn_features = selector_att.index_video(video_features)
- Use the
calculate_video_similarity()
function providing query and target features to calculate similarity based on the student networks.
fine_similarity = fg_att_student.calculate_video_similarity(query_fg_features, target_fg_features)
coarse_similarity = cg_student.calculate_video_similarity(query_cg_features, target_cg_features)
- To calculate the selector's score for a video pair, call the selector network by providing the features extracted for each video and their coarse similarity
selector_features = torch.cat([query_sn_features, target_sn_features, coarse_similarity], 1)
selector_scores = selector_att(selector_features)
If you use this code for your research, please cite our paper.
@article{kordopatis2021dns,
title={DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval},
author={Kordopatis-Zilos, Giorgos and Tzelepis, Christos and Papadopoulos, Symeon and Kompatsiaris, Ioannis and Patras, Ioannis},
journal={arXiv preprint arXiv:2106.13266},
year={2021}
}
This work has been supported by the projects WeVerify and MediaVerse, partially funded by the European Commission under contract number 825297 and 957252, respectively, and DECSTER funded by EPSRC under contract number EP/R025290/1.
This project is licensed under the Apache License 2.0 - see the LICENSE file for details
Giorgos Kordopatis-Zilos (georgekordopatis@iti.gr)