Question on evaluation time

[Guangyao Li]

Hi, recently I noticed this project and the corresponding paper. And I also found that the evaluation time is a big advantage of this project, I want to know why torchkge can evaluate faster than OpenKE or AmpliGraph. I have read the corresponding files about evaluation, e.g., torchkge/evaluation/link_prediction.py, but I still don't know the key element of this success. Can you help me?
The reason for my interest in evaluation time is that I find a relatively complex model will spend much time evaluating on some large KGs. So I hope to obtain some insights from this project to accelerate the evaluation stage.
I will appreciate your reply. Thanks!

[Armand Boschin]

Hello, thanks for your interest in TorchKGE.

As explained in the corresponding article, the key element to fast evaluation is the definition for each model (required by the torchkge.models.interfaces.Model interface) of the lp_scoring_function method.

• The scoring_function method takes as input 1D tensors containing the indexes of head, tail and relation entities of the current batch.
• The lp_scoring_function method is called by the link-prediction evaluation module and can compute the scores of all possible candidate entities for each fact of the current batch in a vectorized way. It takes as input 2D and 3D tensors containing the embedding vectors of either head entities (2D), relations (2D) and all possible entities (3D) (candidates to complete (h, r, _)) or all possible entities (3D) (candidates to completer (_, r, t)) and relations (2D) and tail entities (2D). The three dimensions being batch size, number of candidates, embedding dimension.

Obviously there is a lot of redundancy in the data especially because all 3d slices of the candidate tensor are replications (simply all the embeddings of all possible entities). Attention is paid to do this efficiently.

OpenKE and Ampligraph were doing this computation sequentially on the facts of the evaluation set: for each fact, the scores are computed for all candidates (only grouping the candidates in a batch when TorchKGE groups the candidates and the various facts in a batch.

I hope it's clearer for you.

[Guangyao Li]

I see. Thanks for your patience!