Performance of cell-level clustering
nathalia-kim opened this issue · comments
Hello,
I've been trying to reproduce the results of cell-level clustering from the paper.
According to Table II, for dataset A, your method achieved purity = 0.855
, entropy = 0.750
and f-score = 0.863
.
I have run your code with different configurations, but for some reason, I can't achieve the same results as those reported in the paper.
-
According to the paper, the model was trained with
n_epoch = 10
,batchsize = 64
,lr = 2e-4
- With these parameters, I get
purity = 0.511
,entropy = 1.548
andf_score = 0.589
- With these parameters, I get
-
Using the defaults of the code:
n_epoch = 50
,batchsize = 10
,lr = 1e-4
- I get
purity = 0.725
,entropy = 1.137
andf_score = 0.732
- I get
I also ran the code with other configurations to investigate:
-
n_epoch = 10
,batchsize = 64
,lr = 1e-4
purity = 0.535
,entropy = 1.451
,f_score = 0.617
-
n_epoch = 50
,batchsize = 64
,lr = 1e-4
purity = 0.668
,entropy = 1.087
,f_score = 0.689
-
n_epoch = 50
,batchsize = 10
,lr = 2e-4
purity = 0.725
,entropy = 1.145
,f_score = 0.733
It seems that for all the runs, my results are significantly lower than the results you got. Could you please provide me some guidance on what could be causing this discrepancy? Any thoughts would be highly appreciated!
Thank you,
Nathalia
Hello Nathalia,
I would suggest you produce generated images for each cluster and investigate whether it is a good representation. You can also divide images into clusters and investigate visually.
It is difficult to evaluate performance by arbitrary parameters.
Could you also run longer time? Please also refer to the learning curve.
But the most straightforward way is to see if the produced images are a good representation.
Best,
Bo
Hello Bo,
The parameters I mentioned in my last comment were only to serve as examples, but I actually trained the model with all the parameter values from the paper and the default values of the code and wasn't able to achieve the same results as reported in the paper. So my question wasn't meant to be about arbitrary parameters, but why do you think this might be the case even using these two sets of parameters? (paper and defaults)
Thank you,
Nathalia
I cannot answer questions like:
'Why the results are not good enough' or 'I changed the parameters this way but the results are not good enough.'
I can answer questions such as:
'This is my cell-level representations, what went wrong?' or 'the codes are outdated what should I do to make it work.'
I'm not provided with enough information. I don't know how to answer you specifically.
I'm working on other projects so please raise questions about the mathematics, codes, and experiment details specifically. I cannot answer questions just given the numbers. I can only answer questions if I'm provided enough information.