Giters

cvignac / DiGress

code for the paper "DiGress: Discrete Denoising diffusion for graph generation"

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Questions about Metrics (1. Scaf on MOSES & 2. FCD on GuacaMol)

TianBian95 opened this issue · comments

commented

Hi,

For MOSES dataset, I used the smiles file you put in the dir '/generated_samples/generated_smiles_moses.txt', and the metrics evaluation code provided by MOSES repo. For DiGress, Scaf metric on MOSES is around 0.9.

For GuacaMol dataset, I used the smiles file you put in the dir '/generated_samples/digress_guacamol_smiles.txt', and the code imported from fcd_torch. For DiGress, FCD metric on GuacaMol is around 1.78.

May I know why there is such a big gap with the results reported in your paper? Can you provide these metric codes in the repo?

I would also like to know if the results of other baseline methods in your paper are tested using the same code?

BTW, why the FCD metric on GuacaMol is indicated as the bigger is better in the Table 4 in your paper?

Thank you very much.

commented

Hello,
we use in both cases the software provided by Guacamol and MOSES to make sure that all models are evaluated in the same way. I invite you to do the same to avoid unfair comparison.

Guacamol reports scores, so that higher is always better, even for FCD.