Implementation of Incremental Decoding
barry-jin opened this issue · comments
Description
The implementation of incremental decoding in gluon-nlp is somewhat different from fairseq. In fairseq, the keys/values both before and after linear projection are memorialized, but in gluon-nlp, only the keys/values before the linear projection is memorialized. This difference leads to different execution number of FC operators (In fairseq, keys/values are directly pulled from prev_keys/prev_values; In gluon-nlp, two more linear projections are needed to get the projectioned keys/values). We may need to correct the gluon-nlp's implementation of incremental decoding.