data split and experiment result
Wangpeiyi9979 opened this issue · comments
hi， thanks for your nice work. I have a question for data split, in FewRel, s your partition method the same as [1] partition?
Is the report result of your paper reproduced by yourself? In my own practice, I divide the FewRel dataset according to the method in [1], with ProtoCNN in 5way 1shot, the accuracy rate can reach 76%, I'm really confused. Can you help me, tthx.
[1] Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings ofthe 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4803–4809, 2018.
Hello! Thank you for your question. FewRel was split using a "hard split," in which train classes are distant to val and test to simulate word-level distributional shift. Please see A.4 of our paper (page 15) for more details. In addition, our data splits are provided in the download.
Do you mean that you have re-divided the data set and hope that the training set, verification set, and test set will have different word-level distribution? Then have you conducted your experiment on the original data set division? How is the comparison effect with your ProtoCNN
All results were reported on our splits, including baseline models, which we implemented ourselves. The details behind the exact data splitting algorithms may be found in Appendix A.4.
thx