pytorch-lifestream a library built upon PyTorch for building embeddings on discrete event sequences using self-supervision. It can process terabyte-size volumes of raw events like game history events, clickstream data, purchase history or card transactions.
It supports various methods of self-supervised training, adapted for event sequences:
- Contrastive Learning for Event Sequences (CoLES)
- Contrastive Predictive Coding (CPC)
- Replaced Token Detection (RTD) from ELECTRA
- Next Sequence Prediction (NSP) from BERT
- Sequences Order Prediction (SOP) from ALBERT
- Masked Language Model (MLM) from ROBERTA
It supports several types of encoders, including Transformer and RNN. It also supports many types of self-supervised losses.
The following variants of the contrastive losses are supported:
- Contrastive loss (paper)
- Triplet loss (paper)
- Binomial deviance loss (paper)
- Histogramm loss (paper)
- Margin loss (paper)
- VICReg loss (paper)
pip install pytorch-lifestream# Ubuntu 20.04
sudo apt install python3.8 python3-venv
pip3 install pipenv
pipenv sync --dev # install packages exactly as specified in Pipfile.lock
pipenv shell
pytest
- Supervised model training notebook
- Self-supervided training and embeddings for downstream task notebook
- Self-supervided embeddings in CatBoost notebook
- Self-supervided training and fine-tuning notebook
- Self-supervised TrxEncoder only training with Masked Language Model task and fine-tuning notebook
- Pandas data preprocessing options notebook
- PySpark and Parquet for data preprocessing notebook
- Fast inference on large dataset notebook
- Supervised multilabel classification notebook
Library description index
pytorch-lifestream usage experiments on several public event datasets are available in the separate repo