Not really correct to include <|endoftext|> token in scoring?
rsimmons opened this issue Ā· comments
I noticed that the probability of the final <|endoftext|> token is included in scoring. For the purposes of scoring sentences, it seems to me that it would be more correct (for most use cases) to omit that one, because it doesn't really matter whether or not more text follows a given sentence. Doesn't the probability of an <|endoftext|> token following a sentence depend on the (somewhat arbitrary) details of how text was broken up for training?