Giters

yanyanSann / Long-Tailed-Classification-Leaderboard

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Long-Tailed-Classification-Leaderboard

date: 2021/3/3（Updated 2021/3/3）
auther: YW YSZ

1. Introduction

List of abbreviations:

Abbreviations ReW TrL MeL DeL Aug SeSu OtM
Full names Re-weighting Transfer Learning Meta Learning Decoupling Learning Data Augmentation Self-Supervised Learning Other methods

2. Benchmark datasets

Dataset Year Images(Triain/Val/Test) Classes Max images Min images Imbalance factor Reported by
CIFAR-LT-10 2019 50000–11203/--/10000 10 5,000 500–25 1.0–200.0 Source
CIFAR-LT-100 2019 50000–9502/--/10000 100 500 500–2 1.0–200.0 Source
ImageNet-LT 2019 115846/20000/50000 1000 1280 5 256.0 Source
Places-LT 2019 62500/7300/36500 365 4980 5 996.0 Source
iNat 2017 2017 579184/95986/-- 5089 3919 9 435.4 Source
iNat 2018 2018 437513/24426/-- 8142 1000 2 500.0 Source

3. Leaderboard

3.1 CIFAR-10-LT

Evaluation metric: classification error rate.

IF represents Imbalance factor.

Backbone: ResNet-32
Method Venue Year Backbone Type IF=10 IF=50 IF=100 Code Reported by
Class-Balanced Loss CVPR 2019 ResNet-32 ReW 12.51 20.73 25.43 ---- Source
LDAM-DRW NeurIPS 2019 ResNet-32 ReW 11.84 ----- 22.97 ---- Source
MW-Net NeurIPS 2019 ResNet-32 ReW/MeL 12.16 19.94 24.79 ---- Source
LDAM-M2m CVPR 2020 ResNet-32 TrL 12.5 ----- 20.9 ----- Source
BBN CVPR 2020 ResNet-32 DeL 11.68 17.82 20.18 ---- Source
CBasDA-LDAM CVPR 2020 ResNet-32 ReW 12.6 17.77 22.77 ---- Source
Balanced Softmax ECCV-Workshop 2020 ResNet-32 OtM 9.1 ----- 16.9 ---- Source
De-confound-TDE NeurIPS 2020 ResNet-32 OtM 11.5 16.4 19.4 ---- Source
BALMS NeurIPS 2020 ResNet-32 ReW 8.7 ----- 15.1 ---- Source
LDAM-DRW + SSP NeurIPS 2020 ResNet-32 SeSu 11.47 17.87 22.17 ---- Source
Baseline + tricks AAAI 2021 ResNet-32 OtM ----- 16.41 19.97 ---- Source
Remix-DRW ECCV-Workshop 2020 ResNet-32 Aug 10.98 ----- 20.24 ---- Source
Backbone: ResNet-18
Method Venue Year Backbone Type IF=10 IF=50 IF=100 Code Reported by
FSA ECCV 2020 ResNet-18 Aug 8.25 15.29 19.43 ---- Source
Backbone: ResNet-34
Method Venue Year Backbone Type IF=10 IF=50 IF=100 Code Reported by
FSA ECCV 2020 ResNet-34 Aug 8.8 15.51 17.94 ---- Source

3.2 CIFAR-100- LT

Evaluation metric: classification error rate.

Backbone: ResNet-32
Method Venue Year Backbone Type IF=10 IF=50 IF=100 Code Reported by
Class-Balanced Loss CVPR 2019 ResNet-32 ReW 42.01 54.68 60.40 ---- Source
LDAM-DRW NeurIPS 2019 ResNet-32 ReW 41.29 ----- 57.96 ---- Source
MW-Net NeurIPS 2019 ResNet-32 ReW/MeL 41.54 53.26 57.91 ---- Source
LDAM-M2m CVPR 2020 ResNet-32 TrL 42.4 ----- 56.5 ----- Source
BBN CVPR 2020 ResNet-32 DeL 40.88 52.98 57.44 ---- Source
CBasDA-LDAM CVPR 2020 ResNet-32 ReW 42.0 50.84 60.47 ---- Source
LFME+LDAM ECCV 2020 ResNet-32 TrL ----- ----- 56.2 ---- Source
Balanced Softmax ECCV-Workshop 2020 ResNet-32 OtM 36.9 ----- 49.7 ---- Source
De-confound-TDE NeurIPS 2020 ResNet-32 OtM 40.4 49.7 55.9 ---- Source
BALMS NeurIPS 2020 ResNet-32 ReW 37.0 ----- 49.2 ---- Source
LDAM-DRW + SSP NeurIPS 2020 ResNet-32 SeSu 41.09 52.89 56.57 ---- Source
Baseline + tricks AAAI 2021 ResNet-32 OtM ----- 48.31 52.17 ---- Source
Remix-DRW ECCV-Workshop 2020 ResNet-32 Aug 38.77 ----- 53.23 ---- Source
Backbone: ResNet-18
Method Venue Year Backbone Type IF=10 IF=50 IF=100 Code Reported by
FSA ECCV 2020 ResNet-18 Aug 34.92 48.1 53.43 ---- Source
Backbone: ResNet-34
Method Venue Year Backbone Type IF=10 IF=50 IF=100 Code Reported by
FSA ECCV 2020 ResNet-34 Aug 34.71 47.83 51.49 ---- Source

3.3 ImageNet-LT

Evaluation metric: closed-set setting/Top-1 classification accuracy.

Backbone: ResNet-10
Method Venue Year Backbone Type Many-Shot Medium-Shot Few-Shot ALL Code Reported by
OLTR CVPR 2019 ResNet-10 TrL 43.2 35.1 18.5 35.6 ---- Source
LWS ICLR 2020 ResNet-10 DeL ----- ----- ---- 41.4 ---- Source
IEM CVPR 2020 ResNet-10 OtM 48.9 44.0 24.4 43.2 ---- Source
LFME+OLTR ECCV 2020 ResNet-10 TrL 47.0 37.9 19.2 38.8 ---- Source
FSA ECCV 2020 ResNet-10 Aug 47.3 31.6 14.7 35.2 ---- Source
BALMS NeurIPS 2020 ResNet-10 ReW 50.3 39.5 25.3 41.8 ---- Source
cRT + SSP NeurIPS 2020 ResNet-10 SeSu ----- ----- ---- 43.2 ---- Source
Baseline + tricks AAAI 2021 ResNet-10 OtM ----- ----- ---- 43.31 ---- Source
Backbone: ResNeXt-50
Method Venue Year Backbone Type Many-Shot Medium-Shot Few-Shot ALL Code Reported by
LWS ICLR 2020 ResNeXt-50 DeL 60.2 47.2 30.3 49.9 ---- Source
Backbone: ResNeXt-152
Method Venue Year Backbone Type Many-Shot Medium-Shot Few-Shot ALL Code Reported by
LWS ICLR 2020 ResNeXt-152 DeL 63.5 50.4 34.2 53.3 ---- Source

3.4 Places-LT

Evaluation metric: closed-set setting/Top-1 classification accuracy.

Backbone: ResNet-152
Method Venue Year Backbone Type Many-Shot Medium-Shot Few-Shot ALL Code Reported by
OLTR CVPR 2019 ResNet-152 TrL 44.7 37 25.3 35.9 ---- Source
LWS ICLR 2020 ResNet-152 DeL 40.6 39.1 28.6 37.6 ---- Source
τ -normalized ICLR 2020 ResNet-152 DeL 37.8 40.7 31.8 37.9 ---- Source
IEM CVPR 2020 ResNet-152 OtM 46.8 39.2 28.0 39.7 ---- Source
LFME+OLTR ECCV 2020 ResNet-152 TrL 39.3 39.6 24.2 36.2 ---- Source
FSA ECCV 2020 ResNet-152 Aug 42.8 37.5 22.7 36.4 ---- Source
Backbone: ResNet-10
Method Venue Year Backbone Type Many-Shot Medium-Shot Few-Shot ALL Code Reported by
BALMS NeurIPS 2020 ResNet-10 ReW 41.2 39.8 31.6 38.7 ----

3.5 iNaturalist

Evaluation metric: Top-1 classification accuracy

Backbone: ResNet-50
Method Venue Year Backbone Type iNat-2017(Top1) iNat-2018(Top1) Code Reported by
CB Focal CVPR 2019 ResNet-50 ReW 58.08 61.12 ---- Source
LWS ICLR 2020 ResNet-50 DeL ----- 65.9/69.5 (90/200) ---- Source
IEM CVPR 2020 ResNet-50 OtM ----- 70.2 ---- Source
BBN CVPR 2020 ResNet-50 DeL 63.39 66.29 ---- Source
BBN(2×) CVPR 2020 ResNet-50 DeL 65.75 69.62 ---- Source
CBasDA-CE CVPR 2020 ResNet-50 ReW 59.38 67.55 ---- Source
FSA ECCV 2020 ResNet-50 Aug 61.96 65.91 ---- Source
cRT + SSP NeurIPS 2020 ResNet-50 SeSu ----- 68.1 ---- Source
Baseline + tricks AAAI 2021 ResNet-50 OtM ----- 70.87 ---- Source
Remix-DRS ECCV-Workshop 2020 ResNet-50 Aug ----- 70.74 ---- Source
Backbone: ResNet-152
Method Venue Year Backbone Type iNat-2017(Top1) iNat-2018(Top1) Code Reported by
CB Focal CVPR 2019 ResNet-152 ReW 61.84 64.16 ---- Source
LWS ICLR 2020 ResNet-152 DeL ----- 69.1/72.1 (90/200) ---- Source
FSA ECCV 2020 ResNet-152 Aug 66.58 69.08 ---- Source

4. Contact

Yan Wang : yanwang@smail.nju.edu.cn

Yongshun Zhang: zhangys@lamda.nju.edu.cn

5. References

2019

  • Shu et.al., Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting, NeurIPS 2019.
  • Cao et.al., Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019.
  • Cui et.al., Class-Balanced Loss Based on Effective Number of Samples, CVPR 2019.

2020

  • Tang et.al., Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect, NeurIPS 2020.
  • Yang et.al., Rethinking the Value of Labels for Improving Class-Imbalanced Learning, NeurIPS 2020.
  • Ren et.al., Balanced Meta-Softmax for Long-Tailed Visual Recognition, NeurIPS 2020.
  • Kang et.al., Decoupling Representation and Classifier for Long-Tailed Recognition, ICLR 2020.
  • Kim et.al., M2m: Imbalanced Classification via Major-to-minor Translation, CVPR 2020.
  • Zhou et.al., BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition, CVPR 2020.
  • Jamal et.al., Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective, CVPR 2020.
  • Zhu et.al., Inflated Episodic Memory with Region Self-Attention for Long-Tailed Visual Recognition, CVPR 2020.
  • Liu et.al., Large-Scale Long-Tailed Recognition in an Open World, CVPR 2019.
  • Xiang et.al., Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification, ECCV 2020.
  • Chu et.al., Feature Space Augmentation for Long-Tailed Data, ECCV 2020.
  • Ren et.al., Balanced Activation for Long-tailed Visual Recognition, ECCV 2020.
  • Chou et.al., Remix: Rebalanced Mixup, ECCV'2020 Workshop.

2021

  • Zhang et.al., Bag of Tricks for Long-Tailed Visual Recognition with Deep Convolutional Neural Networks, AAAI 2021.

About

License:MIT License