Meta-Learning with Memory-Augmented Neural Networks in Tensorflow
A concise alternative Tensorflow Implementation of Papar Santoro, Adam, et al."Meta-learning with memory-augmented neural networks."International conference on machine learning. 2016. And the model are encapsulated into class MANNCell which can be used as BasicRNNCell. The code is inspired by the excellent implementations of tristandeleu and snowkylin.
Memory-Augmented Neural Networks
As shown in reference paper, MANNs(Memory-Augmented Neural Networks) refer to the class of external memory equipped networkds such as NTMs(Neural Turing Machines).
Dependencies
- Python 3.6
- Tensorflow==1.14
- numpy==1.16.4
- PIL==7.1.1
Usage
Omniglot DataSet
Download images_background.zip (964 classes) and images_evaluation.zip (679 classes), and place them in the ./omniglot folder.
Running
python run_mann.py
python run_mann.py --mode test
python run_mann.py --model LSTM
python run_mann.py --model LSTM --mode test
Class MANNCell()
from mann.mann_cell import MANNCell
cell = MANNCell(
lstm_size = 200,
memory_size = 128,
memory_dim = 40,
nb_reads = 4,
gamma = 0.95
)
state = cell.zero_state(batch_size, tf.float32)
output, state = tf.scan(lambda init, elem: cell(elem, init[1]), elems=tf.transpose(input, perm=[1, 0, 2]), initializer=(tf.zeros(shape=(batch_size, lstm_size+nb_reads*memory_dim)), state))
output = tf.transpose(output, perm=[1, 0, 2])Performance
Omniglot Classfication:
 |
 |
|---|
Test-set classfication accuracies on the Omniglot dataset, using one-hot encodings of labels and five classes presented per episode.
| Model | 1st | 2nd | 3rd | 4th | 5th | 10th |
|---|---|---|---|---|---|---|
| LSTMref | 24.4% | 49.5% | 55.3% | 61.0% | 63.6% | 62.5% |
| LSTMrepo | 30.4% | 77.9% | 85.3% | 87.5% | 88.8% | 91.6% |
| MANNref | 36.4% | 82.8% | 91.0% | 92.6% | 94.9% | 98.1% |
| MANNrepo | 35.4% | 89.2% | 95.2% | 96.3% | 96.9% | 97.8% |