Leaderboard for heterogeneous graphs is missing

Question

Leaderboard for heterogeneous graphs is missing

amblee0306 opened this issue a year ago · comments

In the Simple-HGN code's README (https://github.com/THUDM/CogDL/tree/master/examples/simple_hgn), it mentioned that there is a leaderboard but the link is not working. Do you know where I can find this heterogeneous graphs leaderboard?

cenyk1230 · Answer 1 · Mon Sep 25 2023 09:17:02 GMT+0800 (China Standard Time)

Hi @amblee0306 ,

You can find the cogdl leaderboard from this link. If you want to see the original repo of Simple-HGN, please check this link

Amber Lee · Answer 2 · Mon Sep 25 2023 16:21:09 GMT+0800 (China Standard Time)

Hi @cenyk1230

Thanks for your reply, the leaderboard stated that the models were evaluated using macro-f1 scores, can I check how can I change the code such that the macro-f1 scores are returned? Currently when I run the simple-hgn from cogDL, I believe the test_acc is being utilized instead.

cenyk1230 · Answer 3 · Mon Sep 25 2023 19:26:28 GMT+0800 (China Standard Time)

Hi @amblee0306 ,

It seems that the results in the leaderboard should be micro-f1 scores (the same as accuracy in the multi-class setting) rather than macro-f1.

Amber Lee · Answer 4 · Tue Sep 26 2023 01:26:04 GMT+0800 (China Standard Time)

Hi @cenyk1230

Yes indeed. I would like to get the macro-f1 scores as well. Do you have any advice on how that is possible? Also, the score on the leaderboard, is it only for one instance or it refers to the average micro-f1 score? Cause I ran 10 instances and the average was lower than what was mentioned on the leaderboard.