2Mw / RS-Reproduce

Recommender system reproducing.

Recommender System

The implementation of various recommender system models (Tensorflow).

1. The architecture of the project

RS
├─cf                    # collaborative filtering based recommender system
│  ├─config             # store the configurations of models (one model may has many configurations)
│  ├─layers             # store the layers source code
│  ├─models             # store the models source code
│  ├─preprocess         # store the preprocessing of datasets.
│  ├─result             # store the result of training models, include configs, model summays, weights and accuracy (1 -> n)
│  ├─run                # store the files which run the training process.
│  ├─tune               # The multiple config of yaml files which used to fine tune.
│  └─utils              # store some common functions
├─data
│  ├─avazu
│  ├─criteo
│  └─movielens
├─gnn                   # gnn based recommender system
├─shell                 # some shell scripts
├─notebook              # some classical models in the form of notebooks
└─log                   # store the log file.

2. The list of reproducing models

• CTR models

  1. DeepFM 2017
  2. DCN 2017
  3. DCN-v2 2020
  4. AutoInt 2018
  5. InterHAt 2020
  6. GateNet 2020
  7. EDCN 2021
  8. MaskNet 2021

• Recall models

  1. YoutubeDNN
  2. YoutubeSBC
  3. MIND

• Other:

  1. CowClip 2022

Dataset

1. criteo
2. avazu
3. movielens-1m
4. User Behavior Data from Taobao
5. Ad Display/Click Data
6. Fliggy
7. Huawei Competition --- data_3.zip

Code Reference

1. DeepCTR
2. Cowclip
3. FuxiCTR
4. DeepMatch
5. Torch-rechub
6. Deep-Youtube-Recommendations 한국의 미녀

Appendix

Tips

1. Without MirroredStrategy in lower tf version, it will come with unknown shape warning, even sometimes it will affect the training speed. You can ignore the warning.
2. In the Callback(keras.callbacks.Callback) of lower version of tensorflow (<2.4.0), if set self.model.stop_training = True, the training process will stop in the end of epoch, while higher version of tf will stop in the end of this step.
3. If you wanna to modify the process of keras function fit(), you can inherit the class keras.model.Model, then override the function train_step()(Learnt from cowclip model).
4. Some imports such as from keras.callbacks import Callback may not work in lower version of tensorflow 2.X(<2.4.0), then you should replace it with from tensorflow.keras.callbacks import Callback.
5. If you want to debug training process gracefully, you should set the flag run_eagerly=True in the compile() function when construct the model.
6. Original cowclip source code will crash when gradient because there are mismatch between gradient and trainable variables(In the train_step function).
7. For cowclip model in lower version of tensorflow 2.X(<2.4.0), you should also implement the function test_step otherwise it will cause OOM error because of bad conversion of tensor.
8. For recommendation system, the numerical feature have large variance and hurt algorithms, so we should normalize them(AutoInt C5.1). If we consider training numerical feature with sparse feature in the attention based model, the performance may not well.
9. In EDCN, using BatchNormalization is required, other than it will occur gradient vanishing problems because small number matrix hadamard product in bridge module, and use residual shortcut optionally.
10. For ragged tensor, it occurs keepdims=True is not supported for RaggedTensors error in lower version of Tensorflow(<2.4.0), while it is okay in higher version.
11. In lower version of tf(<2.4.0), the name of variables must conform to regex "^[A-Za-z0-9.][A-Za-z0-9_.\\/>-]*$".
12. In lower version of tf(<2.4.0), there is no PReLU activation function, you need to write it manually(In package cf.metric.recall.PReLU).

Train method

1. Training with 10240, 4096, 1024, 10240 batches in order which can reach better performance for CTR models.