Lookahead Optimizer: k steps forward, 1step back
It is a MXNet implementation of Lookahead Optimizer.
The link of the paper: https://arxiv.org/abs/1907.08610
Usage
Import optimizer.py, then add the prefix Lookahead before the name of arbitrary optimizer.
import optimizer
optimizer.LookaheadSGD(k=5, alpha=0.5, learning_rate=1e-3)Example
python mnist.py --optimizer sgd --seed 42
python mnist.py --optimizer lookaheadsgd --seed 42