Neurons that fire together, wire together.
mnist genetic algorithm
Example usage:
python mnist.py --mutation-rate 0.01 --perturbation-rate 0.1 --perturbation-strength 0.1 --population 100Base MNIST example from official PyTorch examples.
- how to tackle dropout in a genetic algo? It was preventing monotonic fitness improvement
- eliminating dropout makes it overfit quite bad
- actually implement firewire
- Activations of layers of best 2 models don't seem to correlate - each network is learning different things. (TODO a backprop training step first, that probably makes it more homogeneous. Done, and true, but that's probably because there is an optimal answer so you'd expect them to converge on it.)
- conv layers are much more correlated than fc layers
- It learns, strictly monotonic fitness improvement - on the train set!
- Optimizing directly for accuracy, not for loss