adityaramesh / nninit

A collection of initialization schemes for neural networks.

Overview

This library offers a collection of initialization schemes for neural networks based on recent developments. Some parts of this library are based on this other one with the same name.

Supported Initialization Schemes

• A generalization of the variance-normalizing initialization schemes from 1, 2, and 3.
• The orthogonal initialization scheme, described in 4.
• Input preserving initialization schemes that are useful for pixel-labeling architectures:
  • Identity initialization for convolutional layers, described in 6.
  • Downsampling initialization for strided convolutional layers.
  • Bilinear upsamping initialization for transpose convolutional layers (known as SpatialFullConvolution in Torch -- a horrible name), described in 5.