How does the reported 90.93 reported?

[Jie Yang]

Thanks for sharing this nice work!

I am wondering how is the 90.93 reported? Is it the mean value of the system performance under different random seeds or the best result among several random seeds?

Thank you very much!

[Guillaume Lample]

Hi,

Yes, this is an average over several experiments using different random seeds. There was a bit of variance. In particular, the best results on the validation set were often very close, but the score on the test set associated to these best scores on the validation set were sometimes quite different (I guess the validation set was a bit small). Results on the test were oscillating between 90.4 and 91.3 or something like that.

[Jie Yang]

Thank you very much!