Simple Deep Learning
This is a Python implementation of Deep Learning models and algorithms with a minimum use of external library. Simple Deep Learning aims to learn the basic concepts of deep learning by creating a library from scratch.
Installation
Activation Functions
The following activation functions are defined in activation.py as class that has forward and backward methods.
ReLU (Rectified Linear Unit)
LReLU (Leaky Rectified Linear Unit)
PReLU (Parameteric Rectified Linear Unit)
ELU (Exponential Linear Unit)
SELU (Scaled Exponential Linear Unit)
Sigmoid (Logistic Function)
SoftPlus
Tanh
Arctan
SoftSign
Layers
The following layers are defined in layers.py as class that has forward and backward methods (someof them have predict method)
Convolution Layer (3D)
This layer is compatible with minibatch and deals with a 3D tensor consists of (channel, hight, width). The input data will have a shape of (batch number, channel, hight, width).
Pooling Layer
Two options, max pooling and average pooling, are avalable for this layer.
Affine Layer
This layer is compatible with tensor expression so that you can directly connect 3D layer and fully-conected (2D) layer.
Maxout Layer
This layer can only be used in fully-conected (2D) layer.
Batch Normalization Layer
Dropout Layer
Loss Function
MAE (Mean Absolute Error)
MSE (Mean Square Error)
RMSE (Root Mean Square Error)
Reference
Following links are used as reference:
https://en.wikipedia.org/wiki/Activation_function
http://www.deeplearningbook.org/contents/optimization.html