Unable to re-produce LANE results on SCAN length split

Question

Unable to re-produce LANE results on SCAN length split

ag1988 opened this issue 3 years ago · comments

Hello,

I am running the training command as instructed in LANE:
CUDA_VISIBLE_DEVICES=0 python main.py --mode train --checkpoint length_model --task length
but am not able to get the expected results. I tried this with random seeds 1, 2.
I am attaching the log files - can you please let me know what I might be doing wrong.
length_model_seed2.log
length_model.log

Qian · Answer 1 · Fri Mar 26 2021 12:27:51 GMT+0800 (China Standard Time)

@ag1988 Thanks for your interest on our work! I will look into this problem and get back here in 2 days. Please stay tuned!

Qian · Answer 2 · Mon Mar 29 2021 18:01:33 GMT+0800 (China Standard Time)

@ag1988 Hi, sorry for the mistake on curriculum setting. I have updated the code, and you're expected to reproduce the results of length stably. Please try again, Thanks ☕

Attached the log file: model_length.log

Ankit Gupta · Answer 3 · Mon Mar 29 2021 22:15:02 GMT+0800 (China Standard Time)

Thanks for the prompt response Qian! I'll run the updated code and get back to you if the issue persists. Regards, Ankit

…

On Mon, Mar 29, 2021, 6:01 AM Qian ***@***.***> wrote: @ag1988 <https://github.com/ag1988> Hi, sorry for the mistake on curriculum setting. I have updated the code, and you're expected to reproduce the results of length stably. Please try again, Thanks ☕ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AID3IGHZ5CBQEHEZEHYSCTLTGBFY7ANCNFSM4ZXYI7NQ> .

Ankit Gupta · Answer 4 · Wed Mar 31 2021 00:06:34 GMT+0800 (China Standard Time)

I was able to able reproduce 100% accuracy on length split with the update hyperparams. Thanks!