shawnwun / NNDIAL

NNDial is an open source toolkit for building end-to-end trainable task-oriented dialogue models. It is released by Tsung-Hsien (Shawn) Wen from Cambridge Dialogue Systems Group under Apache License 2.0.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Test results are different from the paper.

robotzheng opened this issue · comments

LIDM_70
Venue Match Rate : 91.9%
Task Success Rate : 74.3%
BLEU : 0.2337
Semantic Match : 63.7%

LIDM-RL_70
Venue Match Rate : 92.6%
Task Success Rate : 75.7%
BLEU : 0.2459
Semantic Match : 63.4%

LIDM-RL_100 (just set I = 100)
Venue Match Rate : 89.7%
Task Success Rate : 70.6%
BLEU : 0.2541
Semantic Match : 60.3%

not like the paper's result:
LIDM Models
LIDM, I = 50 66.9 0.238
LIDM, I = 70 61.0 0.246
LIDM, I = 100 63.2 0.242
LIDM Models + RL
LIDM, I = 50, +RL 82.4 0.231
LIDM, I = 70, +RL 81.6 0.230
LIDM, I = 100, +RL 84.6 0.240

shawnwun, thanks.