When training "alignment model", the result is poor

Question

When training "alignment model", the result is poor

jiazhaojun opened this issue 2 years ago · comments

Hello,
thank you for opening this code, but I found in the process of training the model that the verification loss of "alignment model" is very high and the result is very poor from the first epoch. The situation of the 0 epoch is very different. Do you know the specific reason。The specific results are as follows:
I am a graduate student at school. I look forward to your reply. Thank you

2022-09-12 05:52:41,090 INFO 2022-09-12 05:52:44,719 INFO 2022-09-12 05:52:50,049 INFO 2022-09-12 05:52:55,915 INFO 2022-09-12 05:53:00,731 INFO 2022-09-12 05:53:05,533 INFO 2022-09-12 05:53:11,439 INFO 2022-09-12 05:53:17,333 INFO 2022-09-12 05:53:23,425 INFO 2022-09-12 05:53:29,524 INFO 2022-09-12 05:53:30,719 INFO 2022-09-12 05:53:31,885 INFO 2022-09-12 05:53:36,116 INFO 2022-09-12 05:53:40,436 INFO 2022-09-12 05:53:42,626 INFO 2022-09-12 05:53:44,750 INFO 2022-09-12 05:53:46,474 INFO 2022-09-12 05:53:48,185 INFO 2022-09-12 05:53:50,166 INFO 2022-09-12 05:53:52,150 INFO 2022-09-12 05:54:08,436 INFO 2022-09-12 05:54:24,647 INFO 2022-09-12 05:54:40,986 INFO 2022-09-12 05:54:51,248 INFO 2022-09-12 05:55:01,442 INFO 2022-09-12 05:57:17,300 INFO 2022-09-12 05:59:33,198 INFO 2022-09-12 06:01:16,357 INFO 2022-09-12 06:02:59,333 INFO 2022-09-12 06:04:39,537 INFO 2022-09-12 06:06:19,625 INFO 2022-09-12 06:06:19,626 INFO 2022-09-12 06:06:19,626 INFO 2022-09-12 06:06:22,872 INFO 2022-09-12 06:06:22,872 INFO 2022-09-12 06:06:22,874 INFO 2022-09-12 06:06:24,395 INFO 2022-09-12 06:06:24,395 INFO 2022-09-12 06:06:24,396 INFO 2022-09-12 06:06:24,396 INFO 2022-09-12 06:06:25,716 INFO 2022-09-12 06:06:25,717 INFO 2022-09-12 06:06:29,338 INFO 2022-09-12 06:06:32,889 INFO 2022-09-12 06:06:38,230 INFO 2022-09-12 06:06:43,511 INFO 2022-09-12 06:06:48,272 INFO 2022-09-12 06:06:53,006 INFO 2022-09-12 06:06:58,839 INFO 2022-09-12 06:07:04,671 INFO 2022-09-12 06:07:10,778 INFO 2022-09-12 06:07:16,835 INFO 2022-09-12 06:07:18,000 INFO 2022-09-12 06:07:19,153 INFO 2022-09-12 06:07:23,323 INFO 2022-09-12 06:07:27,476 INFO 2022-09-12 06:07:29,557 INFO 2022-09-12 06:07:31,627 INFO 2022-09-12 06:07:33,286 INFO 2022-09-12 06:07:34,944 INFO 2022-09-12 06:07:36,959 INFO 2022-09-12 06:07:38,917 INFO 2022-09-12 06:07:55,090 INFO 2022-09-12 06:08:11,212 INFO 2022-09-12 06:08:27,317 INFO 2022-09-12 06:08:37,599 INFO 2022-09-12 06:08:47,770 INFO 2022-09-12 06:11:02,964 INFO 2022-09-12 06:13:21,543 INFO 2022-09-12 06:15:04,463 INFO 2022-09-12 06:16:47,372 INFO 2022-09-12 06:18:27,494 INFO 2022-09-12 06:20:07,768 INFO 2022-09-12 06:20:07,769 INFO 2022-09-12 06:20:07,769 INFO 2022-09-12 06:20:11,016 INFO 2022-09-12 06:20:11,016 INFO 2022-09-12 06:20:11,019 INFO 2022-09-12 06:20:12,544 INFO 2022-09-12 06:20:12,544 INFO 2022-09-12 06:20:12,544 INFO 2022-09-12 06:20:12,545 INFO 2022-09-12 06:20:13,887 INFO 2022-09-12 06:20:13,888 INFO 2022-09-12 06:20:17,512 INFO 2022-09-12 06:20:21,065 INFO 2022-09-12 06:20:26,339 INFO 2022-09-12 06:20:31,574 INFO 2022-09-12 06:20:36,383 INFO 2022-09-12 06:20:41,173 INFO 2022-09-12 06:20:47,038 INFO 2022-09-12 06:20:52,879 INFO 2022-09-12 06:20:58,964 INFO 2022-09-12 06:21:05,067 INFO 2022-09-12 06:21:06,247 INFO 2022-09-12 06:21:07,410 INFO 2022-09-12 06:21:11,578 INFO 2022-09-12 06:21:15,736 INFO 2022-09-12 06:21:17,811 INFO 2022-09-12 06:21:19,904 INFO 2022-09-12 06:21:21,579 INFO 2022-09-12 06:21:23,266 INFO 2022-09-12 06:21:25,220 INFO 2022-09-12 06:21:27,170 INFO 2022-09-12 06:21:43,375 INFO 2022-09-12 06:21:59,608 INFO Align ja en Epoch 0 [Train Align Loss 0.577126|
Align ja en Epoch 1 [Train Align Loss 0.294240|
Align es en Epoch 0 [Train Align Loss 0.224772|
Align es en Epoch 1 [Train Align Loss 0.179589|
Align ja fr Epoch 0 [Train Align Loss 0.215952|
Align ja fr Epoch 1 [Train Align Loss 0.201016|
Align en fr Epoch 0 [Train Align Loss 0.184156|
Align en fr Epoch 1 [Train Align Loss 0.186263|
Align es fr Epoch 0 [Train Align Loss 0.188306|
Align es fr Epoch 1 [Train Align Loss 0.187599|
Align el ja Epoch 0 [Train Align Loss 0.207775|
Align el ja Epoch 1 [Train Align Loss 0.202363|
Align ja es Epoch 0 [Train Align Loss 0.202225|
Align ja es Epoch 1 [Train Align Loss 0.191727|
Align el fr Epoch 0 [Train Align Loss 0.222263|
Align el fr Epoch 1 [Train Align Loss 0.185255|
Align el en Epoch 0 [Train Align Loss 0.202170|
Align el en Epoch 1 [Train Align Loss 0.180836|
Align el es Epoch 0 [Train Align Loss 0.202020|
Align el es Epoch 1 [Train Align Loss 0.172025|
KG ja Epoch 0 [Train KG Loss 0.474746|
KG ja Epoch 1 [Train KG Loss 0.240753|
KG ja Epoch 2 [Train KG Loss 0.140252|
KG el Epoch 0 [Train KG Loss 0.430338|
KG el Epoch 1 [Train KG Loss 0.187315|
KG en Epoch 0 [Train KG Loss 0.360577|
KG en Epoch 1 [Train KG Loss 0.136722|
KG es Epoch 0 [Train KG Loss 0.280173|
KG es Epoch 1 [Train KG Loss 0.107179|
KG fr Epoch 0 [Train KG Loss 0.252925|
KG fr Epoch 1 [Train KG Loss 0.107643|
=== round 0
[ja]
Val: Hits@1 (8633 triples): 0.000811
Val: Hits@10 (8633 triples): 0.005444
Val: MRR (8633 triples): 0.003637
Test: Hits@1 (2162 triples): 0.001850
Test: Hits@10 (2162 triples): 0.006938
Test: MRR (2162 triples): 0.004615
BestVal! Epoch 0000 [Test seq] | Best mrr 0.004615| hits1 0.001850| hits10 0.006938|
BestTest! Epoch 0000 [Test seq] | Best mrr 0.004615| hits1 0.001850| hits10 0.006938|
Epoch: 1
Align ja en Epoch 0 [Train Align Loss 8.607937|
Align ja en Epoch 1 [Train Align Loss 5.972235|
Align es en Epoch 0 [Train Align Loss 6.809273|
Align es en Epoch 1 [Train Align Loss 6.584582|
Align ja fr Epoch 0 [Train Align Loss 6.047226|
Align ja fr Epoch 1 [Train Align Loss 5.838850|
Align en fr Epoch 0 [Train Align Loss 6.712930|
Align en fr Epoch 1 [Train Align Loss 6.519938|
Align es fr Epoch 0 [Train Align Loss 6.554777|
Align es fr Epoch 1 [Train Align Loss 6.365027|
Align el ja Epoch 0 [Train Align Loss 5.482096|
Align el ja Epoch 1 [Train Align Loss 5.316339|
Align ja es Epoch 0 [Train Align Loss 5.814239|
Align ja es Epoch 1 [Train Align Loss 5.632405|
Align el fr Epoch 0 [Train Align Loss 6.017292|
Align el fr Epoch 1 [Train Align Loss 5.835585|
Align el en Epoch 0 [Train Align Loss 6.054560|
Align el en Epoch 1 [Train Align Loss 5.876427|
Align el es Epoch 0 [Train Align Loss 5.998742|
Align el es Epoch 1 [Train Align Loss 5.883946|
KG ja Epoch 0 [Train KG Loss 0.133453|
KG ja Epoch 1 [Train KG Loss 0.072645|
KG ja Epoch 2 [Train KG Loss 0.048118|
KG el Epoch 0 [Train KG Loss 0.134991|
KG el Epoch 1 [Train KG Loss 0.078373|
KG en Epoch 0 [Train KG Loss 0.123032|
KG en Epoch 1 [Train KG Loss 0.080099|
KG es Epoch 0 [Train KG Loss 0.089405|
KG es Epoch 1 [Train KG Loss 0.066886|
KG fr Epoch 0 [Train KG Loss 0.096441|
KG fr Epoch 1 [Train KG Loss 0.069282|
=== round 1
[ja]
Val: Hits@1 (8633 triples): 0.000695
Val: Hits@10 (8633 triples): 0.018881
Val: MRR (8633 triples): 0.008707
Test: Hits@1 (2162 triples): 0.000925
Test: Hits@10 (2162 triples): 0.019426
Test: MRR (2162 triples): 0.008669
BestVal! Epoch 0001 [Test seq] | Best mrr 0.008669| hits1 0.000925| hits10 0.019426|
BestTest! Epoch 0001 [Test seq] | Best mrr 0.008669| hits1 0.000925| hits10 0.019426|
Epoch: 2
Align ja en Epoch 0 [Train Align Loss 7.293987|
Align ja en Epoch 1 [Train Align Loss 6.933926|
Align es en Epoch 0 [Train Align Loss 8.175304|
Align es en Epoch 1 [Train Align Loss 7.902112|
Align ja fr Epoch 0 [Train Align Loss 6.991634|
Align ja fr Epoch 1 [Train Align Loss 6.720252|
Align en fr Epoch 0 [Train Align Loss 8.128806|
Align en fr Epoch 1 [Train Align Loss 7.845694|
Align es fr Epoch 0 [Train Align Loss 7.800725|
Align es fr Epoch 1 [Train Align Loss 7.538444|
Align el ja Epoch 0 [Train Align Loss 5.946376|
Align el ja Epoch 1 [Train Align Loss 5.725021|
Align ja es Epoch 0 [Train Align Loss 6.610154|
Align ja es Epoch 1 [Train Align Loss 6.362963|
Align el fr Epoch 0 [Train Align Loss 6.986602|
Align el fr Epoch 1 [Train Align Loss 6.773473|
Align el en Epoch 0 [Train Align Loss 7.147122|
Align el en Epoch 1 [Train Align Loss 6.931155|
Align el es Epoch 0 [Train Align Loss 6.472295|
Align el es Epoch 1 [Train Align Loss 6.281886|
KG ja Epoch 0 [Train KG Loss 0.077415|
KG ja Epoch 1 [Train KG Loss 0.053311|

ZijieH · Answer 1 · Mon Sep 12 2022 15:01:15 GMT+0800 (China Standard Time)

Have you installed all the packages with the right version?
I got the following results from the first epoch in one saved model:

2022-03-28 13:41:56 INFO === round 0
2022-03-28 13:41:56 INFO [ja]
2022-03-28 13:42:04 INFO Val: Hits@1 (8633 triples): 0.069501
2022-03-28 13:42:04 INFO Val: Hits@10 (8633 triples): 0.197730
2022-03-28 13:42:04 INFO Val: MRR (8633 triples): 0.115438
2022-03-28 13:42:09 INFO Test: Hits@1 (2162 triples): 0.077243
2022-03-28 13:42:09 INFO Test: Hits@10 (2162 triples): 0.198427
2022-03-28 13:42:09 INFO Test: MRR (2162 triples): 0.122049
2022-03-28 13:42:09 INFO Epoch 0000 [Test seq] | Best mrr 0.122049| hits1 0.077243| hits10 0.198427|

jiazhaojun · Answer 2 · Mon Sep 12 2022 15:29:37 GMT+0800 (China Standard Time)

The main package versions are as follows: torch 1.11.0 cuda 11.3 python 3.7/3.8 PIP install directly used by torch geometric and torch scatter torch spark torch cluster There were some errors when I first ran the code, so I modified some places, such as:  line 184 of the source\ssaga_model.py script neg_ loss = torch. mean(neg_losses, dim=-1) target = torch. tensor([-1], dtype=torch.long, device=self.device) Changed to neg_ loss = torch.unsqueeze(torch.mean(neg_losses, dim=-1), 1)target = torch.tensor([[-1]*pos_loss.size(0)], dtype=torch. long, device=self.device) Because they and pos_loss The dimension is different, resulting in loss = self. criterion_ Kg (pos_loss, neg_loss, target)

…

------------------ 原始邮件 ------------------ 发件人: "amzn/ss-aga-kgc" ***@***.***>; 发送时间: 2022年9月12日(星期一) 下午3:01 ***@***.***>; ***@***.******@***.***>; 主题: Re: [amzn/ss-aga-kgc] When training "alignment model", the result is poor (Issue #11) Have you installed all the packages with the right version? I got the following results from the first epoch in one saved model: 2022-03-28 13:41:56 INFO === round 0 2022-03-28 13:41:56 INFO [ja] 2022-03-28 13:42:04 INFO Val: ***@***.*** (8633 triples): 0.069501 2022-03-28 13:42:04 INFO Val: ***@***.*** (8633 triples): 0.197730 2022-03-28 13:42:04 INFO Val: MRR (8633 triples): 0.115438 2022-03-28 13:42:09 INFO Test: ***@***.*** (2162 triples): 0.077243 2022-03-28 13:42:09 INFO Test: ***@***.*** (2162 triples): 0.198427 2022-03-28 13:42:09 INFO Test: MRR (2162 triples): 0.122049 2022-03-28 13:42:09 INFO Epoch 0000 [Test seq] | Best mrr 0.122049| hits1 0.077243| hits10 0.198427| — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

ZijieH · Answer 3 · Mon Sep 12 2022 17:02:54 GMT+0800 (China Standard Time)

Please refer to the readme file to install the corresponding version for running the code.

jiazhaojun · Answer 4 · Mon Sep 12 2022 17:08:43 GMT+0800 (China Standard Time)

OK，thanks

…

---Original--- From: ***@***.***> Date: Mon, Sep 12, 2022 17:03 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [amzn/ss-aga-kgc] When training "alignment model", the result ispoor (Issue #11) Please refer to the readme file to install the corresponding version for running the code. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>