Stacked DNC
aryopg opened this issue · comments
Is it possible to create a stacked DNC (add DNC on top of a DNC)? i'm struggling to understand the graph flow in the DNC implementation.
If you mean to stack DNCs as you would LSTMs or other Neural Network neurons/blocks (ie. have one DNC output to another DNC, and repeat as desired, potentially making a deep implementation), then yes, you certainly can. Each DNC would have its own external memory in this case.
You could probably also make that deep DNC be the controller to an over-arching DNC, but I am uncertain on how much of the model would need modified, if at all. I also doubt that it would be beneficial for such an implementation. I am still new to this myself, so take what I say with a grain of salt.
A paper you may want to look at that attempts to dissect and explain the DNC better is "Implementation and Optimization of Differentiable Neural Computers" by Carol Hsin. I haven't read it but saved it to see if it offers anything enlightening outside of the original paper. Make sure you read the original DNC paper fully though.
Some notable competition to the DNC is EntNet, a "Recurrent Entity Network" by Mikael Henaff et al. Its an interesting model and is available in Torch and Tensorflow on github, just look around. I mention this just in case it better fits your needs because EntNet has memory for individual neurons, and depending how extreme you go with multiple DNC's, it may be better suited for your needs.
Also implementing this DNC in TensorFlow and then using TensorBoard would detail the graph you are having trouble with, I believe.
Great suggestions @prijatelj !!! I will certainly read all of the papers you gave. I think the biggest obstacle that i got is to understand the tensorflow graph construction. I kind of confused on how to reuse a layer class (in this case the DNC). It would be awesome if you have any tutorial on how to make a custom block that can be reused :)
I'm new to Tensor Flow in general, so as I learn and if I create such a "custom block", I will let you know.
You could try using the DeepRNN class in sonnet to stack DNC modules:
https://github.com/deepmind/sonnet/blob/master/sonnet/python/modules/basic_rnn.py#L179