chester-w-xie / continual-learning-baselines

Continual learning baselines and strategies from popular papers, using Avalanche. We include EWC, SI, GEM, AGEM, LwF, iCarl, GDumb, and other strategies.

Continual Learning Baselines

Avalanche Website | Avalanche Repository

This project provides a set of examples with popular continual learning strategies and baselines. You can easily run experiments to reproduce results from original paper or tweak the hyperparameters to get your own results. Sky is the limit!

To guarantee fair implementations, we rely on the Avalanche library, developed and maintained by ContinualAI. Feel free to check it out and support the project!

Experiments

The table below describes all the experiments currently implemented in the experiments folder.

Strategy	Benchmarks
Synaptic Intelligence (SI)	Split MNIST, Permuted Mnist
CoPE	Split MNIST
Deep Streaming LDA (DSLDA)	CORe50
Elastic Weight Consolidation (EWC)	Permuted MNIST
Average GEM (AGEM)	Permuted MNIST, Split CIFAR-100
GEM	Permuted MNIST, Split CIFAR-100
GSS-greedy	Split MNIST
Learning without Forgetting (LwF)	Split MNIST, Permuted MNIST, Split Tiny ImageNet
GSS	Split MNIST
iCaRL	Split CIFAR-100
GDumb	Split MNIST
Memory Aware Synapses	Split Tiny ImageNet

Python dependencies for experiments

Outside Python standard library, the main packages required to run the experiments are PyTorch, Avalanche and Pandas.

• Avalanche: The latest version of this repo requires the latest Avalanche version (from master branch): pip install git+https://github.com/ContinualAI/avalanche.git. The CL baselines repo is tagged with the supported Avalanche version (you can browse the tags to check out all the versions). You can install the corresponding Avalanche versions with pip install avalanche-lib==[version number], where [version number] is of the form 0.1.0.
  For more details on how to install Avalanche, please check out the complete guide on how to install Avalanche here.
• PyTorch: we recommend to follow the official guide.
• Pandas: pip install pandas. Official guide.

Run experiments with Python

Place yourself into the project root folder.

Experiments can be run with a python script by simply importing the function from the experiments folder and executing it.
By default, experiments will run on GPU, when available.

The input argument to each experiment is an optional dictionary of parameters to be used in the experiments. If None, default parameters (taken from original paper) will be used.

from experiments.split_mnist import synaptic_intelligence_smnist  # select the experiment

 # can be None to use default parameters
custom_hyperparameters = {'si_lambda': 0.01, 'cuda': -1, 'seed': 3}

# run the experiment
result = synaptic_intelligence_smnist(custom_hyperparameters)

# dictionary of avalanche metrics
print(result)  

Command line experiments

Place yourself into the project root folder.
You should add the project root folder to your PYTHONPATH.

For example, on Linux you can set it up globally:

export PYTHONPATH=${PYTHONPATH}:/path/to/continual-learning-baselines

or just for the current command:

PYTHONPATH=${PYTHONPATH}:/path/to/continual-learning-baselines command to be executed

You can run experiments directly through console with the default parameters.
Open the console and run the python file you want by specifying its path.

For example, to run Synaptic Intelligence on Split MNIST:

python experiments/split_mnist/synaptic_intelligence.py

To execute experiment with custom parameters, please refer to the previous section.

Run tests

Place yourself into the project root folder.

You can run all tests with

python -m unittest

or you can specify a test by providing the test name in the format tests.strategy_class_name.test_benchmarkname.

For example to run Synaptic Intelligence on Split MNIST you can run:

python -m unittest tests.SynapticIntelligence.test_smnist

Contribute to the project

We are always looking for new contributors willing to help us in the challenging mission of providing robust experiments to the community. Would you like to join us? The steps are easy!

1. Take a look at the opened issues and find yours
2. Fork this repo and write an experiment (see next section)
3. Submit a PR and receive support from the maintainers
4. Merge the PR, your contribution is now included in the project!

Write an experiment

1. Create the appropriate script into experiments/benchmark_folder. If the benchmark is not present, you can add one.
2. Fill the experiment.py file with your code, following the style of the other experiments. The script should return the metrics used by the related test.
3. Add to tests/target_results.csv the expected result for your experiment. You can add a number or a list of numbers.
4. Write the unit test in tests/strategy_folder/experiment.py. Follow the very simple structure of existing tests.
5. Update table in README.md.

Find the avalanche commit which produced a regression

1. Place yourself into the avalanche folder and make sure you are using the avalanche version from that repository in your python environment (it is usually enough to add /path/to/avalanche to your PYTHONPATH).
2. Use the gitbisect_test.sh (provided in this repository) in combination with git bisect to retrieve the avalanche commit introducing the regression.
   git bisect start HEAD v0.1.0 -- # HEAD (current version) is bad, v0.1.0 is good
git bisect run /path/to/gitbisect_test.sh /path/to/continual-learning-baselines optional_test_name
git bisect reset
3. The gitbisect_test.sh script requires a mandatory parameter pointing to the continual-learning-baselines directory and an optional parameter specifying the path to a particular unittest (e.g., tests.EWC.test_pmnist). If the second parameter is not given, all the unit tests will be run.
4. The terminal output will tell you which commit introduced the bug
5. You can change the HEAD and v0.1.0 ref to any avalanche commit.