Softmax Node has useless input operand (selected)?
et-nivard opened this issue · comments
Softmax Node has a second input named "selected" https://github.com/pytorch/glow/blob/master/tools/ClassGen/NodeGen.cpp#L374 which doesn't seem to be used at all. Actually, Softmax instruction doesn't have this extra input https://github.com/pytorch/glow/blob/master/tools/ClassGen/InstrGen.cpp#L300 and there is no specific lowering between them. Only in the case of gradient version it seems to have a usage.
Could we remove this apparently useless input and cleanup the corresponding OperatorTest? Or maybe I'm missing something?
I believe this is used when doing training. E.g. see IRGen for SoftMaxGrad
Lines 222 to 239 in d546718
Yes, I know. But why does SoftMax Node also contain this input operand? Is it required due to some relationship or flow between SoftMax and SoftMaxGrad? If not, why not removing it? It seems like redundant code to me.
Sorry, I'm just trying to understand it better.
Yes, after differentiation it is also passed as an input into SoftMaxGradNode
, and is used there. See screenshot below of an example dot, from MLTest's classifyPlayerSport
:
We include it in the SoftMaxNode
so that when we autogenerate the SoftMaxGradNode
it's there for use by the SoftMaxGrad kernel.
glow/lib/Backends/Interpreter/InterpreterNodes.cpp
Lines 2253 to 2269 in 4c1b787
OK, got it. Thanks.