NTM-Lasagne

NTM-Lasagne is a library to create Neural Turing Machines (NTMs) in Theano using the Lasagne library. If you want to learn more about NTMs, check out our blog post.

This library features:

A Neural Turing Machine layer NTMLayer, where all its components (controller, heads, memory) are fully customizable.
Two types of controllers: a feed-forward DenseController and a "vanilla" recurrent RecurrentController.
A dashboard to visualize the inner mechanism of the NTM.
Generators to sample examples from algorithmic tasks.

Getting started

To avoid any conflict with your existing Python setup, and to keep this project self-contained, it is suggested to work in a virtual environment with virtualenv. To install virtualenv:

sudo pip install --upgrade virtualenv

Create a virtual environment called venv, activate it and install the requirements given by requirements.txt. NTM-Lasagne requires the bleeding-edge version, check the Lasagne installation instructions for details. The latest version of Lasagne is included in the requirements.txt.

virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
pip install .

Example

Here is minimal example to define a NTMLayer

# Neural Turing Machine Layer
memory = Memory((128, 20), memory_init=lasagne.init.Constant(1e-6),
    learn_init=False, name='memory')
controller = DenseController(l_input, memory_shape=(128, 20),
    num_units=100, num_reads=1,
    nonlinearity=lasagne.nonlinearities.rectify,
    name='controller')
heads = [
    WriteHead(controller, num_shifts=3, memory_shape=(128, 20),
        nonlinearity_key=lasagne.nonlinearities.rectify,
        nonlinearity_add=lasagne.nonlinearities.rectify,
        learn_init=False, name='write'),
    ReadHead(controller, num_shifts=3, memory_shape=(128, 20),
        nonlinearity_key=lasagne.nonlinearities.rectify,
        learn_init=False, name='read')
]
l_ntm = NTMLayer(l_input, memory=memory, controller=controller, heads=heads)

For more detailed examples, check the examples folder. If you would like to train a Neural Turing Machine on one of these examples, simply run the corresponding script, like

PYTHONPATH=. python examples/copy-task.py

Tests

This projects has a few basic tests. To run these tests, you can run the py.test on the project folder

venv/bin/py.test ntm -vv

Known issues

Graph optimization is computationally intensive. If you are encountering suspiciously long compilation times (more than a few minutes), you may need to increase the amount of memory allocated (if you run it on a Virtual Machine). Alternatively, turning off the swap may help for debugging (with swapoff/swapon).

Note: Unlucky initialisation of the parameters might lead to a diverging solution (NaN scores).

Paper

Alex Graves, Greg Wayne, Ivo Danihelka, Neural Turing Machines, [arXiv]

Contributing

Please see the Contribution Guidelines.

Copyright

This library is provided by Snips as Open Source software. See LICENSE for more information.

snipsco / ntm-lasagne