automenta / ActivationRelaxation

Code to reproduce experiments for "Activation Relaxation: A Local, Dynamical Approximation to Backprop in the Brain"

ActivationRelaxation

Code for the paper "Activation Relaxation: A Local Dynamical Approximation to Backprop in the Brain" -- https://arxiv.org/pdf/2009.05359.pdf.

The paper demonstrates a novel algorithm which can converge to the backpropagation of error algorithm using only local learning rules. We demonstrate numerical convergence to autodiff gradients as well as strong performance in training MLPs on MNIST and Fashion-MNIST. Moreover, we propose additional simplifications of the algorithm which remove further biological implausibilities and which maintain strong learning performance.

Installation and Usage

Simply git clone the repository to your home computer. The numerical_test.py file will recreate the numerical results in Figure 1. The main.py file contains code to reproduce all the other experiments.

Requirements

The code is written in [Python 3.x] and uses the following packages:

• [NumPY]
• [PyTorch] version 1.3.1
• [matplotlib] for plotting figures

If you find this code useful, please reference in your paper:

@article{millidge2020activation,
  title={Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain},
  author={Millidge, Beren and Tschantz, Alexander and Buckley, Christopher L and Seth, Anil},
  journal={arXiv preprint arXiv:2009.05359},
  year={2020}
}