The educational project to build an entire training process of a dense artificial neural network from scratch (not using any deep learning frameworks). I'm planning to expand this project by visualizing training process on the website.
Until this moment I've implemented training simulation with Numpy. My motivation was to fully understand backpropagation algorithm behind a dense ANN.
python test.py
- Set number of epochs and batch size
NUM_EPOCHS = 10
BATCH_SIZE = 2
- Define inputs as 2darray where columns are training examples and rows are features for ANN
inputs = np.array([[1, 2], [2, 3], [3, 4], [4, 5]], dtype=T_PRECISION) # 2 examples
- Define targets as 2darray where columns are targets and rows are output neurons for each example( ex. for binary problem it will be 1 output neuron )
targets = np.array([[1, 0]]) # 2 targets
- Define a network structure as a list where its lenght is number of layers( along with input layer ) and each number within the list indicate amount of neurons in each layer, 2 higher-order functions which initialize respectively weights and biases ( here weights are initialized with 0.01 and biases with 1 )
# 3 hidden layers
dense_nn = NN_Dense([inputs.shape[0], 3, 2, 1], init.with_value(0.01), init.with_value(1))
- Define an activation function which will be applied in each hidden layer( here is used sigmoid )
output = dense_nn.forward(inputs[:, id_batch * BATCH_SIZE : upper_bound], sigmoid)
- Define a loss function( here cross entropy )
loss = cross_entropy(output, targets[:, id_batch * BATCH_SIZE : upper_bound])
- Define an optimazation higher-order function( here simple gradient descent with learning rate set to 0.02 )
dense_nn.step(gradient_descent(lr=0.02))
- Python 3.7.7
- Numpy 1.18.1
To install all dependencies, run the command pip install -r requirements.txt
Ready features:
- training simulation
To-do list( from the most important ):
- support for input data in csv file format
- training visualization on the website
Project is: in progress
Forward propagation and backpropagation based on this video
My email: alglowa1@gmail.com
Linkedin: Alex Głowacki