Mammoth - An Extendible (General) Continual Learning Framework for Pytorch

Run DAE: Dynamic Architecture and Knowledge Ensemble

Step 1: Training models: Knowledge Base Task Specific (KBTS) and Expanded Task Specific (ETS):

KBTS: select from obtained knowledge to solve the new task.
ETS: expand and learn new knowledge for the new task.
BA: using Batch Augmentation when evaluation
Run experiments on CIFAR100 - 5 tasks: python main.py --model dae --dataset seq-cifar100 --ablation op_cal --total_tasks 5 --num_aug 32 --norm_type bn_track_affine --lr 0.1 --lr_score 0.5 --temperature 0.1 --lamb 0.05_0.1 --buffer_size 2000 --dropout 0.2 --sparsity 0.8 --seed 0 --batch_size 32 --val_batch_size 256 --verbose

Step 2: Evaluate models:

python main.py --model dae --eval --dataset seq-cifar100 --ablation op_cal --total_tasks 5 --num_aug 32 --norm_type bn_track_affine --lr 0.1 --lr_score 0.5 --temperature 0.1 --lamb 0.05_0.1 --buffer_size 2000 --dropout 0.2 --sparsity 0.8 --seed 0 --batch_size 32 --val_batch_size 256 --verbose
Displayed results: mode (kbts, ets, kbts_ets, kbts_ets_ba), setting (cil, til), avg acc, acc each tasks

Step 3: Calibrate models:

python main.py --model dae --cal --dataset seq-cifar100 --ablation op_cal --total_tasks 5 --num_aug 32 --norm_type bn_track_affine --lr 0.1 --lr_score 0.5 --temperature 0.1 --lamb 0.05_0.1 --buffer_size 2000 --dropout 0.2 --sparsity 0.8 --seed 0 --batch_size 32 --val_batch_size 256 --verbose

Official repository of Class-Incremental Continual Learning into the eXtended DER-verse and Dark Experience for General Continual Learning: a Strong, Simple Baseline

Setup

Use ./utils/main.py to run experiments.
Use argument --load_best_args to use the best hyperparameters from the paper.
New models can be added to the models/ folder.
New datasets can be added to the datasets/ folder.

Models

eXtended-DER (X-DER)
Dark Experience Replay (DER)
Dark Experience Replay++ (DER++)
Learning a Unified Classifier Incrementally via Rebalancing (LUCIR)
Greedy Sampler and Dumb Learner (GDumb)
Bias Correction (BiC)
Regular Polytope Classifier (RPC)
Gradient Episodic Memory (GEM)
A-GEM
A-GEM with Reservoir (A-GEM-R)
Experience Replay (ER)
Meta-Experience Replay (MER)
Function Distance Regularization (FDR)
Greedy gradient-based Sample Selection (GSS)
Hindsight Anchor Learning (HAL)
Incremental Classifier and Representation Learning (iCaRL)
online Elastic Weight Consolidation (oEWC)
Synaptic Intelligence (SI)
Learning without Forgetting (LwF)
Progressive Neural Networks (PNN)

Datasets

Sequential MNIST (Class-Il / Task-IL)
Sequential CIFAR-10 (Class-Il / Task-IL)
Sequential Tiny ImageNet (Class-Il / Task-IL)
Sequential CIFAR-100 (Class-Il / Task-IL)
Permuted MNIST (Domain-IL)
Rotated MNIST (Domain-IL)
MNIST-360 (General Continual Learning)

Citing these works

@article{boschini2022class,
  title={Class-Incremental Continual Learning into the eXtended DER-verse},
  author={Boschini, Matteo and Bonicelli, Lorenzo and Buzzega, Pietro and Porrello, Angelo and Calderara, Simone},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2022},
  publisher={IEEE}
}

@inproceedings{buzzega2020dark,
 author = {Buzzega, Pietro and Boschini, Matteo and Porrello, Angelo and Abati, Davide and Calderara, Simone},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
 pages = {15920--15930},
 publisher = {Curran Associates, Inc.},
 title = {Dark Experience for General Continual Learning: a Strong, Simple Baseline},
 volume = {33},
 year = {2020}
}

Awesome Papers using Mammoth

Our Papers

Dark Experience for General Continual Learning: a Strong, Simple Baseline (NeurIPS 2020) [paper]
Rethinking Experience Replay: a Bag of Tricks for Continual Learning (ICPR 2020) [paper] [code]
Class-Incremental Continual Learning into the eXtended DER-verse (TPAMI 2022) [paper]
Effects of Auxiliary Knowledge on Continual Learning (ICPR 2022) [paper]
Transfer without Forgetting (ECCV 2022) [paper][code]
Continual semi-supervised learning through contrastive interpolation consistency (PRL 2022) [paper][code]
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (NeurIPS 2022) [paper] [code]

Other Awesome CL works using Mammoth

New Insights on Reducing Abrupt Representation Change in Online Continual Learning (ICLR2022) [paper] [code]
Learning fast, learning slow: A general continual learning method based on complementary learning system (ICLR2022) [paper] [code]
Self-supervised models are continual learners (CVPR2022) [paper] [code]
Representational continuity for unsupervised continual learning (ICLR2022) [paper] [code]
Continual Learning by Modeling Intra-Class Variation (Tech Report) [paper]
Consistency is the key to further Mitigating Catastrophic Forgetting in Continual Learning (CoLLAs2022) [paper] [code]
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (ICLR2022) [paper] [code]
NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks (ICML2022) [paper]
Learning from Students: Online Contrastive Distillation Network for General Continual Learning (IJCAI2022) [paper] [code]

Update Roadmap

In the near future, we plan to incorporate the following improvements into this master repository:

ER+Tricks (Rethinking Experience Replay: a Bag of Tricks for Continual Learning)
TwF & Pretraining Baselines (Transfer without Forgetting)
CCIC & CSSL Baselines (Continual semi-supervised learning through contrastive interpolation consistency)
LiDER (On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning)
Additional X-DER datasets (Class-Incremental Continual Learning into the eXtended DER-verse)

Pull requests welcome! Get in touch

Previous versions

If you're interested in a version of this repo that only includes the code for Dark Experience for General Continual Learning: a Strong, Simple Baseline, please use our neurips2020 tag.

khanhkhanhlele / ADVERSARIAL