For my master's project.
Gradient-based hyperparameter optimization algorithms have the potential to scale to numbers of individual hyperparameters proportional to the number of elementary parameters, unlike other current approaches. Some candidate completions of DrMAD, one such algorithm that updates the hyperparameters after fully training the parameters of the model, are explored, with experiments tuning per-parameter L2 regularization coefficients on CIFAR10 with the DenseNet architecture. Experiments with DenseNets on CIFAR10 are also conducted with an adaptive method, which updates the hyperparameters during the training of elementary parameters, tuning per-parameter learning rates and L2 regularization. The experiments do not establish the utility of either method, but the adaptive method shows some promise, with further experiments required.
DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks (code)
Gradient-based Hyperparameter Optimization through Reversible Learning (code)
Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters (code)
Online Learning Rate Adaptation with Hypergradient Descent
Parameter Adaptation in Stochastic Optimization
Densely Connected Convolutional Networks
The code here is edited from https://github.com/bigaidream-projects/drmad (and https://github.com/HIPS/hypergrad), and https://github.com/Lasagne/Recipes/tree/master/papers/densenet.
The CPU version (cpu-ver) is written in Python and uses https://github.com/HIPS/autograd, while the GPU version (gpu-ver) is written in Theano.