range of key_strengths
jingweiz opened this issue · comments
Hey,
For the key strengths vectors (beta's), in NTM it is >0 ("a positive key strength" from section 3.3.1), while in DNC it is >=1 (output from a oneplus), is it a change by intention?
Also in the code in addressing.py, the read/write_strengths are by passing the controller_output through a Linear then through a softplus, which is just the log part of the oneplus, I could not locate where are they added by one additional 1?
Thanks in advance!
The short answer to the whole question is that once the network begins to learn it always increases the strength above the value of 1 and so it doesn't really matter if you clip it to be > 0 or >= 1.
Great, thanks a lot!