Multi-Turn Response Selection in Retrieval-Based Chatbots
Multi-turn response selection in retrieval-based chatbots is a task which aims to select the best-matched response from a set of candidates, given the context of a conversation. This task is attracting more and more attention in academia and industry. However, no one has maintained a leaderboard and a collection of popular papers and datasets yet. The main objective of this repository is to provide the reader with a quick overview of benchmark datasets and the state-of-the-art studies on this task, which serves as a stepping stone for further research.
The World is Not Binary: Learning to Rank with Grayscale Data for Dialogue Response Selection. EMNLP 2020. [paper]
MSN (Yuan et al., 2019)
-
0.800
0.899
0.978
Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots. EMNLP 2019. [paper][code]
IOI (Tao et al., 2019)
0.947
0.796
0.894
0.974
One Time of Interaction May Not Be Enough: Go Deep with an Interaction-over-Interaction Network for Response Selection in Dialogues. ACL 2019. [paper][code]
IMN (Gu et al., 2019)
0.946
0.794
0.889
0.974
Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. CIKM 2019. [paper][code]
U2U-IMN (Gu et al., 2019)
0.945
0.790
0.886
0.973
Utterance-to-Utterance Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. TASLP 2019. [paper][code]
MRFN (Tao et al., 2019)
0.945
0.786
0.886
0.976
Multi-Representation Fusion Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. WSDM 2019. [paper][code]
IACMN (Wang et al., 2019)
0.944
0.782
0.886
0.973
Multi-Turn Response Selection in Retrieval-Based Chatbots with Iterated Attentive Convolution Matching Network. CIKM 2019. [paper][code]
DAM (Zhou et al., 2018)
0.938
0.767
0.874
0.969
Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network. ACL 2018. [paper][code]
DUA (Zhang et al., 2018)
-
0.752
0.868
0.962
Modeling Multi-Turn Conversation with Deep Utterance Aggregation. COLING 2018. [code]
SMN (Wu et al., 2017)
0.926
0.726
0.847
0.961
Sequential Matching Network: A New Architecture for Multi-Turn Response Selection in Retrieval-Based Chatbots. ACL 2017. [paper][code]
Ubuntu Dialogue Corpus V2
Model
R_2@1
R_10@1
R_10@2
R_10@5
Paper and Code
Cross-encoder (Humeau et al., 2020)
-
0.865
-
0.991
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring. ICLR 2020. [paper][code]
Thread-bi (Jia et al., 2020)
-
0.838
0.924
0.985
Multi-turn Response Selection using Dialogue Dependency Relations. EMNLP 2020. [paper][code]
SA-BERT (Gu et al., 2020)
0.963
0.830
0.919
0.985
Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots. CIKM 2020. [paper][code]
IMN (Gu et al., 2019)
0.945
0.771
0.886
0.979
Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. CIKM 2019. [paper][code]
U2U-IMN (Gu et al., 2019)
0.943
0.762
0.877
0.975
Utterance-to-Utterance Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. TASLP 2019. [paper][code]
HRDE-LTC (Yoon et al., 2018)
0.915
0.652
0.815
0.966
Learning to Rank Question-Answer Pairs using Hierarchical Recurrent Encoder with Latent Topic Clustering. NAACL 2018. [paper][code]
Douban Conversation Corpus
Model
MAP
MRR
P@1
R_10@1
R_10@2
R_10@5
Paper and Code
BERT-FP (Han et al., 2021)
0.644
0.680
0.512
0.324
0.542
0.870
Fine-grained Post-training for Improving Retrieval-based Dialogue Systems. NAACL 2021. [paper][code]
SA-BERT+HCL (Su et al., 2021)
0.639
0.681
0.514
0.330
0.531
0.858
Dialogue Response Selection with Hierarchical Curriculum Learning. ACL 2021. [paper][code]
UMS_BERT+ (Whang et al., 2020)
0.625
0.664
0.499
0.318
0.482
0.858
Do Response Selection Models Really Know What’s Next? Utterance Manipulation Strategies for Multi-turn Response Selection. AAAI 2021. [paper][code]
SA-BERT (Gu et al., 2020)
0.619
0.659
0.496
0.313
0.481
0.847
Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots. CIKM 2020. [paper][code]
DCM (Li et al., 2020)
0.611
0.649
-
0.294
0.498
0.842
Deep context modeling for multi-turn response selection in dialogue systems. Information Processing & Management 2020. [paper][code]
BERT-SPIDER (Zhang et al., 2021)
0.609
0.650
0.475
0.296
0.488
0.836
Structural Pre-training for Dialogue Comprehension. ACL 2021. [paper][code]
RoBERTa-BASE-SS-DA (Lu et al., 2020)
0.602
0.646
0.460
0.280
0.495
0.847
Improving Contextual Language Models for Response Retrieval in Multi-Turn Conversation. SIGIR 2020. [paper][code]
G-MSN (Lin et al., 2020)
0.599
0.645
0.476
0.308
0.468
0.826
The World is Not Binary: Learning to Rank with Grayscale Data for Dialogue Response Selection. EMNLP 2020. [paper]
Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots. EMNLP 2019. [paper][code]
IOI (Tao et al., 2019)
0.573
0.621
0.444
0.269
0.451
0.786
One Time of Interaction May Not Be Enough: Go Deep with an Interaction-over-Interaction Network for Response Selection in Dialogues. ACL 2019. [paper][code]
IACMN (Wang et al., 2019)
0.571
0.621
0.448
0.269
0.453
0.783
Multi-Turn Response Selection in Retrieval-Based Chatbots with Iterated Attentive Convolution Matching Network. CIKM 2019. [paper][code]
MRFN (Tao et al., 2019)
0.571
0.617
0.448
0.276
0.435
0.783
Multi-Representation Fusion Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. WSDM 2019. [paper][code]
IMN (Gu et al., 2019)
0.570
0.615
0.433
0.262
0.452
0.789
Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. CIKM 2019. [paper][code]
U2U-IMN (Gu et al., 2019)
0.564
0.611
0.429
0.259
0.430
0.791
Utterance-to-Utterance Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. TASLP 2019. [paper][code]
DAM (Zhou et al., 2018)
0.550
0.601
0.427
0.254
0.410
0.757
Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network. ACL 2018. [paper][code]
DUA (Zhang et al., 2018)
0.551
0.599
0.421
0.243
0.421
0.780
Modeling Multi-Turn Conversation with Deep Utterance Aggregation. COLING 2018. [code]
SMN (Wu et al., 2017)
0.529
0.569
0.397
0.233
0.396
0.724
Sequential Matching Network: A New Architecture for Multi-Turn Response Selection in Retrieval-Based Chatbots. ACL 2017. [paper][code]
E-commerce Corpus
Model
R_10@1
R_10@2
R_10@5
Paper and Code
BERT-FP (Han et al., 2021)
0.870
0.956
0.993
Fine-grained Post-training for Improving Retrieval-based Dialogue Systems. NAACL 2021. [paper][code]
BERT-SL (Xu et al., 2020)
0.776
0.919
0.991
Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues. AAAI 2021. [paper]
UMS_BERT+ (Whang et al., 2020)
0.762
0.905
0.986
Do Response Selection Models Really Know What’s Next? Utterance Manipulation Strategies for Multi-turn Response Selection. AAAI 2021. [paper][code]
SA-BERT+HCL (Su et al., 2021)
0.721
0.896
0.993
Dialogue Response Selection with Hierarchical Curriculum Learning. ACL 2021. [paper][code]
BERT-SPIDER (Zhang et al., 2021)
0.708
0.853
0.986
Structural Pre-training for Dialogue Comprehension. ACL 2021. [paper][code]
SA-BERT (Gu et al., 2020)
0.704
0.879
0.985
Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots. CIKM 2020. [paper][code]
DCM (Li et al., 2020)
0.685
0.864
0.982
Deep context modeling for multi-turn response selection in dialogue systems. Information Processing & Management 2020. [paper][code]
Improving Contextual Language Models for Response Retrieval in Multi-Turn Conversation. SIGIR 2020. [paper][code]
IMN (Gu et al., 2019)
0.621
0.797
0.964
Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. CIKM 2019. [paper][code]
U2U-IMN (Gu et al., 2019)
0.616
0.806
0.966
Utterance-to-Utterance Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. TASLP 2019. [paper][code]
G-MSN (Lin et al., 2020)
0.613
0.786
0.964
The World is Not Binary: Learning to Rank with Grayscale Data for Dialogue Response Selection. EMNLP 2020. [paper]
MSN (Yuan et al., 2019)
0.606
0.770
0.937
Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots. EMNLP 2019. [paper][code]
IOI (Tao et al., 2019)
0.563
0.768
0.950
One Time of Interaction May Not Be Enough: Go Deep with an Interaction-over-Interaction Network for Response Selection in Dialogues. ACL 2019. [paper][code]
DUA (Zhang et al., 2018)
0.501
0.700
0.921
Modeling Multi-Turn Conversation with Deep Utterance Aggregation. COLING 2018. [code]
SMN (Wu et al., 2017)
0.453
0.654
0.886
Sequential Matching Network: A New Architecture for Multi-Turn Response Selection in Retrieval-Based Chatbots. ACL 2017. [paper][code]
Papers
In addition to the studies mentioned above, there are stil a lot of great studies on multi-turn response selection worth reading. We list a part of them below.
Distilling Knowledge for Fast Retrieval-based Chat-bots. Amir Vakili Tahami, Kamyar Ghajar, Azadeh Shakery. SIGIR 2020.
Conversational Word Embedding for Retrieval-Based Dialog System. Wentao Ma, Yiming Cui, Ting Liu, Dong Wang, ShijinWang, Guoping Hu. ACL 2020.
IART: Intent-aware Response Ranking with Transformers in Information-seeking Conversation Systems. Liu Yang, Minghui Qiu, Chen Qu, Cen Chen, Jiafeng Guo, Yongfeng Zhang, W. Bruce Croft, Haiqing Chen. WWW 2020.
Dually Interactive Matching Network for Personalized Response Selection in Retrieval-Based Chatbots. Jia-Chen Gu, Zhen-Hua Ling, Xiaodan Zhu, Quan Liu. EMNLP 2019.
Sampling Matters! An Empirical Study of Negative Sampling Strategies for Learning of Matching Models in Retrieval-based Dialogue Systems. Jia Li, Chongyang Tao, wei wu, Yansong Feng, Dongyan Zhao, Rui Yan. EMNLP 2019.
Learning a Matching Model with Co-teaching for Multi-turn Response Selection in Retrieval-based Dialogue Systems. Jiazhan Feng, Chongyang Tao, Wei Wu, Yansong Feng, Dongyan Zhao, Rui Yan. ACL 2019.
Training Neural Response Selection for Task-Oriented Dialogue Systems. Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, Pei-Hao Su. ACL 2019.
DSTC7 Task 1: Noetic End-to-End Response Selection. Chulaka Gunasekara, Jonathan K. Kummerfeld, Lazaros Polymenakos, Walter Lasecki. ACL 2019 Workshop.
A Document-grounded Matching Network for Response Selection in Retrieval-based Chatbots. Xueliang Zhao, Chongyang Tao, Wei Wu, Can Xu, Dongyan Zhao, Rui Yan. IJCAI 2019.
Sequential Attention-based Network for Noetic End-to-End Response Selection. Qian Chen, Wen Wang. AAAI 2019 Workshop on DSTC 7.
Building Sequential Inference Models for End-to-End Response Selection. Jia-Chen Gu, Zhen-Hua Ling, Yuping Ruan, Quan Liu. AAAI 2019 Workshop on DSTC 7.
Training Millions of Personalized Dialogue Agents. Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, Antoine Bordes. EMNLP 2018.
Personalizing Dialogue Agents: I have a dog, do you have pets too?. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, Jason Weston. ACL 2018.
Learning Matching Models with Weak Supervision for Response Selection in Retrieval-based Chatbots. Yu Wu, Wei Wu, Zhoujun Li, Ming Zhou. ACL 2018.
Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems. Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, W. Bruce Croft, Jun Huang, Haiqing Chen. SIGIR 2018.
Improving Response Selection in Multi-turn Dialogue Systems by Incorporating Domain Knowledge. Debanjan Chaudhuri, Agustinus Kristiadi, Jens Lehmann, Asja Fischer. CONLL 2018.
Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus. Jianxiong Dong, Jim Huang. ArXiv.
Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System. Rui Yan, Yiping Song, Hua Wu. SIGIR 2016.
Update
Although we work very hard to list more work, the studies we select to present in this repository are by no means complete. To this end, we welcome more people to participate in the maintenance of this project. Please feel free to open issues, pull requests or contact us (gujc@mail.ustc.edu.cn).
About
Leaderboards, Datasets and Papers for Multi-Turn Response Selection in Retrieval-Based Chatbots