Adam with Decay

[Shubham Agrawal]

I just want to ask that if I want to use weight decay with Adam. How should I proceed with it?
As there is no parameter for weight decay in Adam optimizer.

[Marcus Edel]

At this point you would have to use the SGD class as basis:

CustomDecay decay;
SGD<AdamUpdate, MyDecay> optimizer(0.01, 32, 100000, 1e-5, true, decay);

and the CustomDecay class has to implement the following interface: https://github.com/mlpack/ensmallen/blob/master/include/ensmallen_bits/sgd/decay_policies/no_decay.hpp

Ideally, the adam optimizer can use the decay policy directly, instead of going over the SGD base class.

[mlpack-bot]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions! 👍