How to identify positive samples and negative samples?
MAxx8371 opened this issue · comments
In the paper, it mentions that the score is defined as follow. Is that calculated by summing the logprob of each token of the ground true y conditioned on (e,x)? If that were right, the score would not be between 0 and 1 as shown in the pic. In that case, what is the threshold used for identifying pos&neg samples? Appreciate it!
In the paper, it mentions that the score is defined as follow. Is that calculated by summing the logprob of each token of the ground true y conditioned on (e,x)?
Yes, you are correct.
If that were right, the score would not be between 0 and 1 as shown in the pic. I
Yes. I found it to be more intuitive to explain with probability instead of logprobs.... It means the same thing.
In that case, what is the threshold used for identifying pos&neg samples?
We just take the top-5 as positive and the bottom-5 as negative.
Since all the candidates are from a BM25, the bottom-5 are good negatives