Going Deeper: Infinite Deep Neural Networks
This repository contains the code for the experiments of the following paper-like document: doc/going_deeper.pdf
Summary
The document describes a meta-layer for infinite deep neural networks. It basically wraps a few other layers in a special way that allows the neural network to decide how many sub-layers in the meta-layer should be used. Each sub-layer has its own weights, so the network also decides how many weights should be used. The complete training process may be done with gradient descent-based methods.
Please read doc/going_deeper.pdf for more details.
Library
The repository contains a small library that allows it to use the described meta-layer. The library is based on Keras. The library is very minimal, so not all network architectures may be created with it. A basic model (the model of the first experiment), may be created like this:
# Create the model
n_input_units = 8
n_internal_units = 24
model = TDModel()
model += Input((n_input_units,))
model += Dense(n_internal_units, activation='relu', trainable=False)
# The described meta-layer
model += GInftlyLayer(
# The name
'd0',
# f_i(x)
f_layer=[
lambda reg: Dense(n_internal_units),
lambda reg: GammaRegularizedBatchNorm(reg=reg, max_free_gamma=0.),
lambda reg: Dropout(0.1),
],
# h(x)
h_step=[
lambda reg: Activation('relu')
],
# Regularizers
w_regularizer=(c_l2, w_reg),
f_regularizer=(c_l2, f_reg)#1e-2)
)
model += Dense(1, activation='sigmoid', trainable=False)
# Build the model
model.init(
optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy']
)
You may try to test the first experiment to get a better feeling for the interface.
Experiments
All experiments are more detailed documented in doc/going_deeper.pdf.
The first experiment uses 8 binary inputs and calculates the XOR-result of them. The used netwok contains only trainable weights in a GInftyLayer-layer. Tests are done with 0-8 active inputs for the XOR-calculation. Inactive inputs are not used for the XOR-calculation and just get random input values.
It can be assumed that a network with more active inputs for the XOR-computation is more complex and, therefore, requires more sub-layers in the GInftyLayer-layer. Exactly this can be shown with the given experiment. The w-value, which basically contains the amount of sub-layers is higher for more active inputs:
The second experiment is conducted on the MNIST-dataset. The used network architecture contains two convolutional GInftyLayer-layers and one fully connected GInftyLayer-layer. The test accuracy is up to 99.5 % and it can be seen that the second convolutional GInftyLayer-layer is the deepest layer. The first convolutional and the fully connected GInftyLayer-layer have a very low activation. The second convolutional layer has a depth of 2. The weights are visualized on the following plot:
