amie.eval.rules.Evaluator is buggy for rules that predict constants
falcaopetri opened this issue · comments
Description
amie.eval.rules.Evaluator accepts a training KB, a target KB, and a list of pairs <rule, prediction> (generated by amie.eval.rules.Predictor), and classifies the prediction in 4 cases:
- 0, if the prediction is in the target dataset.
- 1, if it contradicts a functional constraint in the training dataset.
- 2, if it contradicts a functional constraint in the target dataset.
- 3, otherwise (i.e., the prediction require manual inspection).
Consider a rule that predicts a const, i.e., => r(?a, CONST), and generates predictions such as r(A, CONST). Evaluator will fail to classify the prediction as cases 1 or 2, and will always classify it as case 0 or 3, i.e., it will unnecessarily throw predictions to manual inspection.
Reproducing
Train KB:
$ cat train_kb.txt
<s1> <relation> <value1>
<s2> <relation> <value3>
<s3> <relation> <value4>Target KB:
$ cat target_kb.txt
<s1> <relation> <value1>
<s1> <relation> <value2>
<s2> <relation> <value3>
<s3> <relation> <value4>
Hand-crafted "mined" rules:
$ cat crafted_rules.txt
?a <relation> <value1> => ?a <relation> <value2>
?a <relation> <value1> => ?a <relation> <value3>Executing Predictor generates:
$ java -cp amie3.jar amie.rules.eval.Predictor train_kb.txt 5 1 crafted_rules.txt > preds.txt
$ cat preds.txt
Loading files...
Starting train_kb.txt
Finished train_kb.txt, still running: 0
Loaded 1 facts in 12 ms using 0 MB
?a <relation> <value1> => ?a <relation> <value2> <s1> <relation> <value2>
?a <relation> <value1> => ?a <relation> <value3> <s1> <relation> <value3>And then using Evaluator:
$ java -cp amie3.jar amie.rules.eval.Evaluator preds.txt train_kb.txt target_kb.txt manual.out auto.out
$ cat manual.out
?a <relation> <value1> => ?a <relation> <value3> <s1> <relation> <value3> ManualEvaluation
$ cat auto.out
?a <relation> <value1> => ?a <relation> <value2> <s1> <relation> <value2> TargetSource TrueAs far as I understand, this is wrong, since the prediction <s1> <relation> <value3> contradicts the functional constraint in the training dataset. Indeed, it should have being identified as case 1 here:
https://github.com/lajus/amie/blob/75323c6ed76f7576066ffff41d8bdb9e3ac1c112/rules/src/main/java/amie/rules/eval/Evaluator.java#L89-L102
The bug
- creates the
headvariable equal to the rule's head, - bounds the functional variable and;
- queries training and test for this triple.
If the rule's head is r(?a, ?b), we still have a variable after bounding. But if the rule's head is r(?a, CONST), we will try to query r(CONST1, CONST), which will return 0 if the predicted fact is false.
Fix
public static int evaluate(Rule rule,
int[] triple, KB training, KB target){
- // TODO Auto-generated method stub
- int[] head = rule.getHead();
- int boundVariable = 0;
+ int[] head = new int[3];
+ head[0] = KB.map("?s");
+ head[1] = triple[1];
+ head[2] = KB.map("?o");
+
int returnVal = 3;
boolean relationIsFunctional =
(rule.getFunctionalVariablePosition() == 0 && training.functionality(rule.getHead()[1]) >= 0.9) ||
(rule.getFunctionalVariablePosition() == 2 && training.inverseFunctionality(rule.getHead()[1]) >= 0.84);
int boundVarPos = rule.getFunctionalVariablePosition();
//If we know something else about the triple, PCA says it is false
if(target.count(triple) > 0){
//Bingo!
returnVal = 0;
}else{
- boundVariable = head[boundVarPos];
head[boundVarPos] = triple[boundVarPos];
//Here apply PCA on the most functional variable
if(training.count(head) > 0 && relationIsFunctional)
returnVal = 1;
else if(target.count(head) > 0 && relationIsFunctional)
returnVal = 2;
else
returnVal = 3;
-
- //Restore the head
- head[boundVarPos] = boundVariable;
}The following also seems to be a fix:
public static int evaluate(Rule rule,
int[] triple, KB training, KB target){
+ return evaluate(triple, training, target);i.e., we could get rid of
Note
I also noticed that the buggy evaluate() uses rule.getFunctionalVariablePosition(), which, as far as I debugged, will always be set to 0. The same seems true for when using AMIEParser. Is it true that a rule read from file will always have 0 as the functional variable position? Can this be a problem?
Hi,
Thank you very much for your bug report.
The problem is that AMIEParser only reads the atoms of a rule, whereas its functional variable is context-dependent (being the context a knowledge graph). I agree with you that for evaluating AMIE, we only need the method:
public static int evaluate(int[] triple, KB training, KB target)
However, one may want to test the precision of a rule for both positions of the functional variable. Since we do not need the entire rule for this purpose, I would propose to have a version of the method where we send the functional variable as argument.
public static int evaluate(int[] triple, KB training, KB target, boolean relationIsFunctional) {
...
}
public static int evaluate(int[] triple, KB training, KB target) {
boolean relationIsFunctional = training.functionality(triple[1]) >= 0.9;
return evaluate(triple, training, target, relationIsFunctional)
}
If you agree, do not hesitate to send a pull request. You can even pull request what you are proposing now and I can take care of the wrapping.
Thanks!
Luis