This is the Torch implementation for our paper:
Multi-Modality is All You Need for Transferable Recommender Systems
In this paper, we unleash the boundaries of the ID-based paradigm and propose a Pure Multi-Modality based Recommender system (PMMRec), which relies solely on the multi-modal contents of the items (e.g., texts and images) and learns transition patterns general enough to transfer across domains and platforms. Specifically, we design a plug-and-play framework architecture consisting of multi- modal item encoders, a fusion module, and a user encoder. To align the cross-modal item representations, we propose a novel next-item enhanced cross-modal contrastive learning objective, which is equipped with both inter- and intra-modality negative samples and explicitly incorporates the transition patterns of user behaviors into the item encoders. To ensure the robustness of user representations, we propose a novel noised item detection objec- tive and a robustness-aware contrastive learning objective, which work together to denoise user sequences in a self-supervised manner. PMMRec is designed to be loosely coupled, so after being pre-trained on the source data, each component can be trans- ferred alone, or in conjunction with other components, allowing PMMRec to achieve versatility under both multi-modality and single-modality transfer learning settings.
-
Dataset:
-
We provide one processed bili_food dataset example . You first need to download bili_food, and put it in current file.
-
Get the lmdb of the images of the items
python lmdb_build.py
-
-
Pre-trained PMMRec
- We provide Pre-trained PMMRec . You fist need to download Pmmrec_pt, and put it in current file.
-
Download text encoders : xlm-roberta-base , and put it in "TextEncoders"file.
-
Download text encoders : clip-vit-base-patch32 , and put it in "CVEncoders"file.
After processing the datasets, you can test the transfer learning of PMMrec on Industrial dataset by:
cd ./src
python setup.py You also can set different parameters to train this model.