darsnack / ParameterSchedulers.jl

Common hyperparameter scheduling for ML

ParameterSchedulers

ParameterSchedulers.jl provides common machine learning (ML) schedulers for hyper-parameters. Though this package is framework agnostic, a convenient interface for pairing schedules with Flux.jl optimizers is available. Using this package with Flux is as simple as:

using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler

opt = Scheduler(Momentum, Exp(start = 1e-2, decay = 0.8))

Available Schedules

This is a table of the common schedules implemented, but ParameterSchedulers provides utilities for creating more exotic schedules as well. The higher order schedules should make it so that you will rarely need to write a schedule from scratch.

You can read this paper for more information on the schedules below.

Schedule	Description	Type	Example
Step(; start, decay, step_sizes)	Exponential decay by decay every step in step_sizes	Decay	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = Step(start = 1.0, decay = 0.8, step_sizes = [2, 3, 2]) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
Exp(start, decay)	Exponential decay by decay every iteration	Decay	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = Exp(start = 1.0, decay = 0.5) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
CosAnneal(;l0, l1, period)	Cosine annealing	Cyclic	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = CosAnneal(l0 = 0.0, l1 = 1.0, period = 4) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
OneCycle(nsteps, maxval)	One cycle cosine	Complex	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = OneCycle(10, 1.0) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
Triangle(l0, l1, period)	Triangle wave function	Cyclic	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = Triangle(l0 = 0.0, l1 = 1.0, period = 2) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
TriangleDecay2(l0, l1, period)	Triangle wave function with half the amplitude every period	Cyclic	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = TriangleDecay2(l0 = 0.0, l1 = 1.0, period = 2) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
TriangleExp(l0, l1, period, decay)	Triangle wave function with exponential amplitude decay at rate decay	Cyclic	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = TriangleExp(l0 = 0.0, l1 = 1.0, period = 2, decay = 0.8) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
Poly(start, degree, max_iter)	Polynomial decay at degree degree.	Decay	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = Poly(start = 1.0, degree = 2, max_iter = t[end]) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
Inv(start, decay, degree)	Inverse decay at rate (1 + t * decay)^degree	Decay	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = Inv(start = 1.0, degree = 2, decay = 0.8) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
Sin(;l0, l1, period)	Sine function	Cyclic	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = Sin(l0 = 0.0, l1 = 1.0, period = 2) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
SinDecay2(l0, l1, period)	Sine function with half the amplitude every period	Cyclic	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = SinDecay2(l0 = 0.0, l1 = 1.0, period = 2) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
SinExp(l0, l1, period)	Sine function with exponential amplitude decay at rate decay	Cyclic	using UnicodePlots, ParameterSchedulers # hide t = 1:10 |> collect # hide s = SinExp(l0 = 0.0, l1 = 1.0, period = 2, decay = 0.8) # hide lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide