Ranking Distillation

A PyTorch implementation of Ranking Distillation:

Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System, Jiaxi Tang and Ke Wang , KDD '18

Requirements

Python 2 or 3
PyTorch v0.4+
Numpy
SciPy

Usage

Training student models (Model-S)

Run python train_caser.py with d=50 to get the performance of student model.

When finished, we will have the model saved in folder checkpoints/

Training teacher models (Model-T)

Run python train_caser.py with d=100 to get the performance of teacher model.

When finished, we will have the model saved in folder checkpoints/

Training student models with ranking distillation (Model-RD)

Run python train_caser.py with d=100 to get the well-trained teacher model.

When finished, we will have the teacher model saved in folder checkpoints/

(you can also skip this step, as there is one in the checkpoint/gowalla-caser-dim=100.pth.tar)
Run python distill_caser.py with d=50 and teacher_model_path pointed to the teacher model.

Configurations

Model Args (in train_caser.py)

d is set to 50 for student model and 100 for teacher model.
All other the hyper-parameters (e.g., nh, nv, ac_conv, ac_fc) are set by grid-search.

Please check this repo for more information and definations of these hyper-parameters.

Model Args (in distill_caser.py)

teacher_model_path: path to teacher's model checkpoint.
teacher_topk_path: (optional) path to teacher's top-K ranking cache for each training query.
teach_alpha: hyperparameter for balancing ranking loss and distillation loss.
K: length of teacher's exemplary ranking.
lamda: hyperparameter for tuning the sharpness of position importance weight.
mu: hyperparameter for tuning the sharpness of ranking discrepancy weight.
dynamic_samples: number of samples used for estimating student's rank.
dynamic_start_epoch: number of iteration to start using hybrid of two different weights.

Citation

If you use this Caser in your paper, please cite the paper:

@inproceedings{tang2018ranking,
  title={Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System},
  author={Tang, Jiaxi and Wang, Ke},
  booktitle={ACM SIGKDD International Conference on Knowledge Discovery & Data Mining},
  year={2018}
}

Acknowledgment

This project (utils.py, interactions.py, etc.) is heavily built on Spotlight. Thanks Maciej Kula for his great work.

graytowne / rank_distill