Hyperparameters to replicate reported performance
Tiiiger opened this issue · comments
Thank you for the great work!
I would like to ask the correct hyperparameters for each of the datasets in order to replicate the paper reported result. Thank you!
Hi @Tiiiger, I would like to ask if you have been able to find the parameter settings for all the datasets? I ran MUTAG and found the default setting can (kind of) recreate the result. Given there is early stopping and whatnot, the mean accuracy can be higher and match the paper.
batch_size = 128
num_layers = 2
lr = 0.01
num_mlp_layers = 1
hidden_dim = 64
# --epochs can be way smaller than the default 350 (~90)mean acc = 0.8775
std acc = 0.0422
I try to tune some hyperparameters but possibly due to high variance, the result averaged over 10 folds is usually worse than the paper reported number. Also, using MLP and 1-layer do not seem to make a difference in my experiments.
I see. I am searching for COLLAB hyperparameters now. So far varying the number graph layers around 5 doesn't help. Mean acc = ~60% +- 2%. If I take only the highest test accuracy over all training process (assuming early stopping.....), mean acc = ~70%. There is 10% difference from the paper so please let me know if you found the setting that works for COLLAB.
Hi, thanks for your interest. There are number of hyper-parameters to be tuned. Please refer to our paper for the detailed procedure. For COLLAB, did you set --degree_as_tag?
Besides, the variance is indeed rather high due to the small data size. Still, all the results should be able to be reproduced by the exact procedure described in our paper.
@weihua916 Thanks! I indeed did not set --degree_as_tag for COLLAB since I switched from MUTAG and forgot to use the degree as feature vector. Btw, is the default 350 epochs necessary to achieve the results in the paper?
Great! I do not think 350 epochs are necessary for most of the datasets. :)
Cool! Thanks for making the code available btw :).