This repository contains the implementation code for paper:
Mixup Your Own Pairs
Yilei Wu*, Zijian Dong*, Chongyao Chen, Wangchunshu Zhou, Juan Helen Zhou (* equal contribution)
In representation learning, regression has traditionally received less attention than classification. Directly applying representation learning techniques designed for classification to regression often results in fragmented representations in the latent space, yielding sub-optimal performance. In this paper, we argue that the potential of contrastive learning for regression has been overshadowed due to the neglect of two crucial aspects: ordinality-awareness and hardness. To address these challenges, we advocate "mixup your own contrastive pairs for supervised contrastive regression", instead of relying solely on real/augmented samples. Specifically, we propose Supervised Contrastive Learning for Regression with Mixup (SupReMix). It takes anchor-inclusive mixtures (mixup of the anchor and a distinct negative sample) as hard negative pairs and anchor-exclusive mixtures (mixup of two distinct negative samples) as hard positive pairs at the embedding level. This strategy formulates harder contrastive pairs by integrating richer ordinal information. Through extensive experiments on six regression datasets including 2D images, volumetric images, text, tabular data, and time-series signals, coupled with theoretical analysis, we demonstrate that SupReMix pre-training fosters continuous ordered representations of regression data, resulting in significant improvement in regression performance. Furthermore, SupReMix is superior to other approaches in a range of regression challenges including transfer learning, imbalanced training data, and scenarios with fewer training samples.
We separate the codebase for different datasets into different subfolders. Please go into the subfolders for more information (e.g., installation, dataset preparation, training, evaluation & models).
Coming Soon!
- [09/2023] arXiv version posted. Please stay tuned for updates.
If you find this code or idea useful, please cite our work:
@misc{wu2023mixup,
title={Mixup Your Own Pairs},
author={Yilei Wu and Zijian Dong and Chongyao Chen and Wangchunshu Zhou and Juan Helen Zhou},
year={2023},
eprint={2309.16633},
archivePrefix={arXiv},
primaryClass={cs.LG}
}If you have any questions, feel free to contact us through email (yilei.wu@u.nus.edu & zijian.dong@u.nus.edu) or GitHub issues. Enjoy!
The code is based on Yang et al., Delving into Deep Imbalanced Regression, ICML 2021.