This repository is our implementation of Symmetrical Bidirectional Knowledge Alignment (SBKA) for zero-shot sketch-based image retrieval. More detailed descriptions and experimental results could be found in the paper.
Abstract: This paper studies the problem of zero-shot sketch-based image retrieval (ZS-SBIR), which aims to use sketches from unseen categories as queries to match the images of the same category. Due to the large cross-modality discrepancy, ZS-SBIR is still a challenging task and mimics realistic zero-shot scenarios. The key is to leverage transferable knowledge from the pre-trained model to improve generalizability. Existing researchers often utilize the simple fine-tuning training strategy or knowledge distillation from a teacher model with fixed parameters, lacking efficient bidirectional knowledge alignment between student and teacher models simultaneously for better generalization. In this paper, we propose a novel Symmetrical Bidirectional Knowledge Alignment for zero-shot sketch-based image retrieval (SBKA). The symmetrical bidirectional knowledge alignment learning framework is designed to effectively learn mutual rich discriminative information between teacher and student models to achieve the goal of knowledge alignment. Instead of the former one-to-one cross-modality matching in the testing stage, a one-to-many cluster cross-modality matching method is proposed to leverage the inherent relationship of intra-class images to reduce the adverse effects of the existing modality gap. Experiments on several representative ZS-SBIR datasets (Sketchy Ext dataset, TU-Berlin Ext dataset and QuickDraw Ext dataset) prove the proposed algorithm can achieve superior performance compared with state-of-the-art methods.
The requirements of this repository are listed in the requirements.txt .
conda create -n sbka python=3.6
pip install -r requirements.txtPlease download the datasets Sketchy, TU-Berlin, and QuickDraw for ZS-SBIR from Google Drive. Please put these three in ./datasets.
The pre-trained ConvNeXtV2-Tiny model on ImageNet-1K is provided on Google Drive. Please put convnextv2_tiny_1k_224_ema.pt in ``./pretrained_model`.
Train model on Sketchy Ext:
python train_sketchy_ext.pyTrain model on TU-Berlin Ext:
python train_tuberlin_ext.pyTrain model on QuickDraw Ext:
python train_quickdraw_ext.pyThe trained model on Sketchy , Sketchy Split is provided on Google Drive.
For Sketchy, please download zeroshot1/features_zero.pickle to ./checkpoint/sketchy/zeroshot1. For Sketchy Split, please download zeroshot2/features_zero.pickle to ./checkpoint/sketchy/zeroshot2.
Test model on Sketchy:
python test_sketchy_ext_zeroshot.pyTest model on Sketchy Split:
python test_sketchy_ext_split_zeroshot.pyTest model on TU-Berlin:
python test_tuberlin_ext_zeroshot.pyTest model on QuickDraw:
python test_quickdraw_ext_zeroshot.py