Chainer-COCOB
COCOB-Backprop (https://arxiv.org/abs/1705.07795) implementation for Chainer.
Results on MNIST
Chainer's examples/mnist/train_mnist.py is used.
COCOB-Backprop:
GPU: 0
# unit: 1000
# Minibatch-size: 100
# epoch: 20
epoch main/loss validation/main/loss main/accuracy validation/main/accuracy elapsed_time
1 0.249614 0.099938 0.925634 0.9674 2.7832
2 0.0755173 0.0888865 0.976983 0.973 5.41667
3 0.0440907 0.0701315 0.985948 0.9787 8.06651
4 0.0292496 0.0640009 0.990565 0.9824 10.7521
5 0.0172558 0.0621488 0.994799 0.9816 13.375
6 0.0114029 0.056497 0.996732 0.9837 16.0696
7 0.00611094 0.0603573 0.998366 0.985 18.7638
8 0.0025165 0.0627047 0.999483 0.9847 21.4385
9 0.00150869 0.0651257 0.9997 0.9846 24.1239
10 0.000642935 0.0664863 0.999917 0.9858 26.8137
11 0.000272793 0.0676209 1 0.9856 29.4332
12 0.000166279 0.068667 1 0.9854 32.1014
13 0.000126843 0.0702716 1 0.9855 34.7998
14 0.000106101 0.0708647 1 0.9855 37.5045
15 9.10139e-05 0.0720333 1 0.9854 40.169
16 8.00931e-05 0.0725282 1 0.9853 42.8428
17 7.12147e-05 0.0732533 1 0.9852 45.5039
18 6.42114e-05 0.07402 1 0.9854 48.2198
19 5.8484e-05 0.0746067 1 0.9853 50.9703
20 5.35766e-05 0.0749895 1 0.9854 53.6555
Adam (default):
GPU: 0
# unit: 1000
# Minibatch-size: 100
# epoch: 20
epoch main/loss validation/main/loss main/accuracy validation/main/accuracy elapsed_time
1 0.1905 0.0864317 0.942617 0.9727 2.75064
2 0.0722297 0.0810042 0.977516 0.9746 5.40206
3 0.0481114 0.0688474 0.984433 0.9802 8.10238
4 0.0355592 0.0715027 0.988365 0.98 10.8169
5 0.0317207 0.0704834 0.989632 0.9805 13.4787
6 0.0218142 0.0931152 0.992949 0.9761 16.1899
7 0.0207527 0.0977832 0.992999 0.9786 18.8689
8 0.0181046 0.0842731 0.994298 0.9793 21.5223
9 0.0174475 0.0728308 0.994066 0.9828 24.1926
10 0.0134345 0.0794548 0.995449 0.9823 26.8899
11 0.0172595 0.0846022 0.994682 0.982 29.5463
12 0.0119427 0.0910769 0.996665 0.9815 32.1977
13 0.00899746 0.0828989 0.997049 0.9816 34.9126
14 0.0129418 0.0927937 0.996299 0.9795 37.6036
15 0.0104701 0.10654 0.996632 0.9815 40.2741
16 0.0107644 0.0874664 0.996499 0.9845 43.1266
17 0.00734351 0.0980337 0.997699 0.9851 47.5643
18 0.0141138 0.115777 0.996082 0.9807 51.049
19 0.0109274 0.0923189 0.996749 0.9826 53.7219
20 0.00728701 0.0946888 0.998049 0.9812 56.3729