GEO and ATIS results

Question

GEO and ATIS results

todpole3 opened this issue 6 years ago · comments

Victoria X Lin commented 6 years ago

Hi Li,

I run the code myself and reproduced the results on Django and WikiSQL without any problem.

However, I wasn't able to reproduce the results on GEO and ATIS by running the commands given.

I got acc 86.4 on GEO and 86.2 on ATIS. Would you please take a look at this issue? Thanks.

Li Dong · Answer 1 · Thu Aug 02 2018 21:47:07 GMT+0800 (China Standard Time)

Hi @todpole3 ,

I am wondering whether they are the results of pre-trained models, or new models?

./pretrain.sh geoqueries 1
./pretrain.sh atis 1

Thanks!

Victoria X Lin · Answer 2 · Fri Aug 03 2018 01:44:34 GMT+0800 (China Standard Time)

They are the results of the new models:

./run geoqueries 1
./run atis 1

I have tried the pretrained models and their output matched those in the paper.

Li Dong · Answer 3 · Fri Aug 03 2018 23:06:15 GMT+0800 (China Standard Time)

I tried to run the script ./run.sh, and it works fine. There might be some fluctuations due to the difference of the used environments, as the size of these two datasets is not that large. The results on Django and WikiSQL should be more robust. Thanks!

Victoria X Lin · Answer 4 · Sat Aug 04 2018 04:15:35 GMT+0800 (China Standard Time)

I see. Thanks for taking a look at this.