This repository is an implementation of the Direct Kolen Pollack(DKP) method, as well as Direct Feedback Alignment(DFA), using PyTorch. DKP is a combination of two alternative credit assignment learning algorithms: DFA and the Kolen Pollack(KP) method as adapted by Akrout et al. The network used for testing each method on CIFAR10 consists of two convolutional layers followed by two fully connected layers. For testing on CIFAR100 we use AlexNet with batch normalization after each convolutional layer.
main.py usage example:
python main.py --train-mode DKP --batch-size 50 --epochs 100
alexnet/main.py usage example:
python main.py -a alexnet --dist-url 'tcp://127.0.0.1:8080' --dist-backend 'nccl' --multiprocessing-distributed --world-size 1 --rank 0 /home/user01/datasets/ILSVRC2012 --dataset cifar100 --train-mode DFA
Code for alexnet/main.py and alexnet/models/alexnet.py borrows heavily from the repositories below:
- https://github.com/pytorch/examples/tree/master/imagenet
- https://github.com/pytorch/vision/tree/master/torchvision/models
Both DFA and DKP make use of direct connections from the output of the network to each layer in the backward path. This attribute allows for the parallelization of the backwards pass, also refered to as backwards unlocking, meaning that each layer's gradient can be calculated and updated in parallel with another.
In DFA, is a fixed random weight matrix that projects the gradient at the output of a network to the output of layer
. Just as it is with backpropagation, the learning signal at the output of the network
, where
is the derivative of the activation function, would be calculated in the following way.
Then, the learning signal at some layer , as prescribed by DFA(and DKP), would be calculated in the following way.
As for DKP, the rules above remain the same, however is no longer a fixed matrix. We will adjust the backward matrices using the following update rule.