Giters

malllabiisc / RESIDE

EMNLP 2018: RESIDE: Improving Distantly-Supervised Neural Relation Extraction using Side Information

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Question about the accuracy on test split

speedcell4 opened this issue · comments

commented

to my understanding, self.input_y is the hot representation of relations in each bag,
https://github.com/malllabiisc/RESIDE/blob/master/reside.py#L352
in the training stage, the number of relationships is ensured to be one, then this is turn to be a one-hot representation
but in the testing stage, there are multiple relations, so this is actually a multi-hot representation?

and then you use self.input_y to calculate y_actual by using argmax, this means you only need the relation with min index in this bag?
https://github.com/malllabiisc/RESIDE/blob/master/reside.py#L634

finally, this min index relation is used to calculate the accuracy?
https://github.com/malllabiisc/RESIDE/blob/master/reside.py#L635

so for the testing stage, I can not figure out the meaning of accuracy, could you please explain that?

commented

Hi @speedcell4,
For evaluation, I have just followed the procedure which has been used by the prior works (reference). Yes, during training only one label is kept for each bag but while testing a bag can have more than one label. Actually, accuracy has not been used as an evaluation metric in the paper, we report precision and AUC of PR curve, where we do not perform argmax over logits.