Implementations and experiments with Neural Networks. All of this code will be ported to Python 3 shortly. The main file of importance is neuralnetwork.py. It contains a from-scratch implementation of a neural network with a single hidden layer, without the use of any external libraries other than numpy.
The network achieves about 96.5% accuracy on MNIST if the hyperparameters are tuned.
The implementation has a few of the bells and whistles that help neural networks learn better:
- An option for using stochastic gradient descent with minibatch learning
- Decaying the learning rate during training
- Implementation of the momentum method for better convergence & less oscillations
- Nesterov momentum
- L2 regularization to promote smaller weights
- The dropout method that discards hidden layer activations to prevent overfitting.
Another interesting file is utils/utils.py. It contains several of my machine learning utilities for one-hot encoding, k-fold cross-validation, splitting datasets, and hyperparameter tuning.