3rd place solution to the OTTO – Multi-Objective Recommender System Kaggle Competition - Theo's Part

Status :

Document code : Done ✅
Clean notebooks : Done ✅
Make ReadMe : Done ✅
Rerun full pipeline to make sure everything works : To do 📝

Introduction - Adapted from kaggle

The pipeline follows the classical candidates extraction & reranker scheme.

CV = 0.5917 - [0.5621, 0.4438, 0.6706] -> LB 0.6028

Clicks is single model, I blend a few XGBs for carts & orders but the boost is small. Blending with models from my teammates gave our Public 0.60437 / Private 0.60382 LB !

Candidates

I use the candidates from Chris (link), as well as a slightly modified version of the ones from his public kernel. This results in approx. 80 candidates per sessions.

Feature engineering

Most of my (744) features come from the following process :

Compute item-item scores (such as w2v similarities, matrix factorization similarity, Chris' covisitation matrices coefficients) between the candidate and items in the session
Compute a weight adding information about to the item position in the session, timestamp, and type
Aggregate !

Features are computed per batch on a 32Gb V100 using RAPIDS. It's fast :)

Overall pipeline

I tune an Optuna for each fold (which is not a good practice, but I had a really reliable CV setup), pipeline can be a bit long to run but actually, the bottleneck is reading huge parquet files. Heavy downsampling makes it possible to have everything in RAM, and to train on GPU using the tricks Chris shared publicly.

How to use the repository

Prerequisites

Clone the repository
Requirements :
- RAPIDS ! Using the latest stable version should work.
- pip install -r requirements.txt
- Bunch of stuff that doesn't really matter that much
Download the data :
- Put the competition data from Kaggle in the input folder

Run The pipeline

Most of the pipelines is handled in notebooks. The order in which they should be run is specified in the name. Pipeline should run fine in a machine with a 32GB.

Prepare the data using 1-Preparation.ipynb.
Create covisitation matrices using 2-Matrices_Chris.ipynb and 2-Matrices_Theo.ipynb. Notebooks have to be run with MODE="val" and MODE="test"
Create candidates matrices using 3-Candidates.ipynb. Notebooks have to be run with MODE="val", MODE="test" and MODE="extra".
Create embeddings matrices using 4-Matrix_Factorization.ipynb, 4-Seq2Seq_Giba.ipynb and 4-Word2Vec.ipynb. Notebooks have to be run with MODE="val" and MODE="test"
Create features using the fe_main.py script in the src folder. Use python fe_main.py --mode MODE with modes val, test and extra.
Train an XGBoost model using 6-XGB.ipynb. You need to train a models with the 3 targets, the main parameter to tweak is POS_RATIO.
Evaluate your ensembles and generate submission files using 7-Blend.ipynb

If you run into memory issues :

For matrix computation, increase the PIECES values.
For candidates, Chris' candidates use a lot of ram but you can refactor the code to work by chunk (not implemented).
For feature engineering, reduce CHUNK_SIZE.
For training, validation data can be downsampled. I already downsample it for carts and clicks in the utils/load/load_parquets_cudf_folds function but you can downsample more.

Code structure

If you wish to dive into the code, the repository naming should be straight-forward. Each function is documented. The structure is the following :

src
├── data
│   ├── candidates_chris.py         # Chris' candidates utils
│   ├── candidates.py               # Theo's candidates utils
│   ├── covisitation.py             # Theo's covistation matrices
│   ├── fe.py                       # Feature engineering
│   └── preparation.py              # Data preparation utils
├── inference           
│   ├── boosting.py                 # Main file
│   └── predict.py                  # Predict function
├── model_zoo 
│   ├── __init__.py
│   ├── lgbm.py                     # LGBM Ranker kept for legacy
│   └── xgb.py                      # XGBoost classifier
├── otto_src                        
│   ├── evaluate.py                 # From the competition repo
│   ├── labels.py                   # From the competition repo
│   ├── my_split.py                 # My custom splitting functions
│   └── testset.py                  # From the competition repo
├── training           
│   └── boosting.py                 # Trains a boosting model
├── utils          
│   ├── load.py                     # Data loading utils 
│   ├── logger.py                   # Logging utils
│   ├── metrics.py                  # Metrics for the competition
│   ├── plot.py                     # Plotting utils
│   └── torch.py                    # Torch utils
│
├── fe_main.py                      # Main for feature engineering
└── params.py                       # Main parameters

TheoViel / kaggle_otto_rs