data process service
The data process pipeline of recommendation system research usually follows the fashion of load raw data, filter data, reindex data, split train/val/test datasets, save file and also negative sampling for training recommendation models. This repo serves as a general tool for those above data process operations.
data process of a sampled taobao CTR dataset including 1M user-item interactions.
cd examples
python taobao_ctr.py
numpy
pandas
scipy
absl-py
The best entry point to use the following components is a DataFrame with 'uid', 'iid' and 'ts' in its columns.
CsvLoader: load csv file
CooLoader: load coo file (sparse matrix in coordidate format)
JsonLoader: load json file
CFFilter: k-core filter
DuplicationFilter: filter duplicated records with the earliest record left
Reindexer: reindex uid and iid, start from 0
AbsoluteSplitter: split the dataset with test and validation sample number fixed
PercentageSplitter: split the dataset proportionally in chronological order
RandomSplitter: split the dataset randomly
SkewSplitter: split the dataset into biased and unbiased parts according to related literatures (PF, CausE, DICE).
CooGenerator: generate sparse matrix in coordinate format
LilGenerator: generate sparse matrix in lists in list format
DokGenerator: generate sparse matrix in dictionary of keys format
SparseTransformer: perform sparse matrix format transformation from coo to lil and dok
CsvSaver: save DataFrame to file
CooSaver: save coo matrix to file
JsonSaver: save dict to file
CsvReporter: report statistics of the data
PointSampler: negative sampling for pointwise optimization such as logloss
PairSampler: negtive sampling for pairwise optimization such as bprloss