DIY-neural-network

The goal of this project is to build a neural network in order to understand how it works.

The objective of the neural network if to classify handwritten digits so it can tell you what digit it is.

Digits data source: https://www.kaggle.com/competitions/digit-recognizer/data Downloaded in train.csv

The Math

Images:

Image size 28x28 pixels (784 pixels total)
Each pixel value [0 ... 255] (0: black, 255: white)

We can then put this in an matrix:

Columns are exemples in other words training samples
Rows are the pixels (784 long)

Transpose this matrix:

We now have a matrix pixels by training samples

Neural network:

What will be able to give a good prediction of the digit is our neural network. This simple neural network only has 2 layers.

0th layer AKA input layer: 780 nodes (every pixel maps to a node)
1st layer AKA hidden layer: 10 units corresponding to each digit (0, 1, 2, ...9)
2nd layer AKA output layer:

Training:

Input layer

Input layer a[0] will have 784 units corresponding to the 784 pixels in each 28x28 input image

Hidden layer

A hidden layer a[1] will have 10 units with ReLU activation (in order for it to not just be a fancy linear regression with the input layer, we need to apply an activation function).

Activation function exemples:

Hyperbolic tangent function (Tanh) and sigmoid functions.
Rectified linear unit (RelU): RelU(x) is defined as if x>0 then x=x and if x<=0 then x=0

It adds a layer of complexity rather than just a linear model. It makes it more powerfull.

Output layer

Finally output layer a[2] will have 10 units corresponding to the ten digit classes with softmax activation.

1. Forward propogation:

Take the image and run it through the network and compute what your output is going to be.

2. Backward propagation

Evaluates the weights and biasses of the predictions to optimize these weights and biasses. We will mesure how much the prediction deviated for the actual result. Calculate how much these weights and biasses contributed to the the prediction.