This repository contains the implementation to reproduce the numerical experiments of the paper Long Expressive Memory for Sequence Modeling
To setup the environment, install the requirements using python 3.7.10 with
pip install -r requirements.txt
This repository contains the codes to reproduce the results of the following experiments for the proposed LEM:
- Adding Task
- EigenWorms
- FitzHugh-Nagumo
- Google12 (V2)
- Heart rate prediction
- noisy CIFAR-10
- Permuted Sequential MNIST
- PennTree Bank (word- and char-level)
- Sequential MNIST
All datasets get either synthetically generated or downloaded automatically,
providing a full data pipline for each experiment.
Note that the PTB experiments are based on the open-source tensorflow project
LAnguage Modelling Benchmarks (lamb)
for tuning and testing Tensorflow language models.
In order to reproduce these experiments,
please follow the steps explained in the README of the PTB directory.
The results of LEM for each of the experiments are:
| Experiment | Result |
| EigenWorms | 92.3% mean test accuracy |
| FitzHugh-Nagumo | 0.002 test L^2 loss |
| Google12 (V2) | 95.7% test accuracy |
| Heart rate prediction | 0.85 test L^2 loss |
| noisy CIFAR-10 | 60.5% test accuracy |
| Permuted Sequential MNIST | 96.6% test accuracy |
| PennTree Bank char-level (single layer) | 1.25 test bits-per-character |
| PennTree Bank word-level (single layer) | 72.8 test perplexity |
| Sequential MNIST | 99.5% test accuracy |