ZeroCL: instance and feature correlation for negative-free symmetric contrastive learning (ICLR 2022)
Code of ICLR 22 paper "ZeroCL: instance and feature correlation for negative-free symmetric contrastive learning"
This paper presents a negative-free contrastive learning method with symmetric architecture. Zero-CL is composed of Zero-FCL (feature dimension) and Zero-ICL (instance dimension) depending on the alignment dimension.
To pre-train the encoder on CIFAR-10 and CIFAR-100, run:
python main.py --epochs 1000 --dataset cifar10 (cifar100)
The config --whiten is used for comparing directly alignment (collapses) and align the feature after the whitening opration.
For ImageNet-100, train the Zero-CL in one node with several GPUs. Run:
python main.py --data /imagenet100/ --epochs 400 --mode ins (fea)
You should implement the Dataset class by yourself.
The results are:
| Method | CIFAR-10 | CIFAR-100 | ImageNet-100 |
|---|---|---|---|
| VICReg | 90.07 (99.71) | 68.54 (90.83) | 79.22 (95.06) |
| SwAV | 89.17 (99.68) | 64.67 (88.52) | 74.28 (92.84) |
| W-MSE | 88.18 (99.61) | 61.29 (87.11) | 69.06 (91.22) |
| SimCLR | 90.74 (99.75) | 65.39 (88.58) | 77.48 (93.42) |
| Barlow Twins | 89.57 (99.73) | 69.18 (91.19) | 78.62 (94.72) |
| Zero-CL | 90.81 (99.77) | 70.33 (92.05) | 79.26 (94.98) |
For ImageNet pretraining, we implement on several nodes (distributed training). Run:
python main.py --data /imagenet/ --epochs 400 --mode fea
If you use Zero-CL as baseline, please cite our paper:
@inproceedings{
zhang2022zerocl,
title={Zero-{CL}: Instance and Feature decorrelation for negative-free symmetric contrastive learning},
author={Shaofeng Zhang and Feng Zhu and Junchi Yan and Rui Zhao and Xiaokang Yang},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=RAW9tCdVxLj}
}