Human novelty evaluation
kudkudak opened this issue · comments
Categorize triplets into various triviality categories, or state no triviality is seen. Based on 5 closest neighbours in OMCS embedding distance. Then see if:
- how many arguably trivial triplets each dataset has
- there is statistically significant difference in top50% vs bottom50%
Datasets:
- test from CN
- wiki 100 out of 10k (so it is not extremely biased)
- random 100 from train