IDSIA / neuraldiffeq-fwp

Official repository for the paper "Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules" (NeurIPS 2022)

Neural Differential Equations for Learning to Program Neural Nets (Continuous Fast Weight Programmers)

This is the official repository containing code for the paper:

Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules (NeurIPS 2022)

Contents

• speech_and_physionet directory contains the code used for the Speech Commands and PhysioNet Sepsis experiments (Table 1). Originally forked from patrick-kidger/NeuralCDE.
• eigenworms directory contains the code used for the EigenWorms experiment (Table 2). Originally forked from jambo6/neuralRDEs.
• appendix_mujoco directory for the extra reinforcement learning experiments presented in the appendix. Originally forked from dtak/mbrl-smdp-ode.

Separate license files can be found in each of these directories.

Links

• Models implemented here are the continuous-time counterparts of Fast Weight Programmers. For the discrete-time models, see our previous works:
  • Linear Transformers are Secretly Fast Weight Programmers (ICML 2021)
  • Going Beyond Linear Transformers with Recurrent Fast Weight Programmers (NeurIPS 2021)
  • A Modern Self-Referential Weight Matrix That Learns to Modify Itself (ICML 2022)
• Jürgen Schmidhuber's AI blog post on Fast Weight Programmers (March 26, 2021).

BibTex

@inproceedings{irie2022neural,
  title={Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules},
  author={Irie, Kazuki and Faccio, Francesco and Schmidhuber, J{\"u}rgen},
  booktitle={Proc. Advances in Neural Information Processing Systems (NeurIPS)},
  address = {New Orleans, {LA}, {USA}},
  month = dec,
  year={2022}
}