XrosLiang / KENLG-Reading

Reading list for knowledge-enhanced text generation, with a survey

Knowledge-enriched Text Generation Reading-List

Here is a list of recent publications about Knowledge-enhanced text generation. (Update on Nov 5th, 2020)

-- We will continue to add and update related papers and codes on this page.

-- indicates available code and indicates high citation in recent years.

Survey paper

A Survey of knoweldge-enhanced Text Generation. Wenhao Yu (ND), Chenguang Zhu (Microsoft), Zaitang Li (CUHK), Zhiting Hu (UCSD), Qingyun Wang (UIUC), Heng Ji (UIUC), Meng Jiang (ND). arXiv. 2010.04389

To the best of our knowledge, our survey is the first work that presents a comprehensive reviewof knowledge-enhanced text generation. It aims to provide NLG researchers a synthesis and pointer to related researches. Our survey also includes a detailed discussion about how NLG can benefit from recent progress in deep learning and artificial intelligence, including technologies such as graph neural network, reinforcement learning, neural topic modeling and so on.

Basic NLG papers and codes

(For new learners, some important papers for general NLG/KENLG.)

• [Seq2Seq] Sequence to Sequence Learning with Neural Networks
  • Ilya Sutskever (Google) et al, In NeurIPS 2014.
• [SeqAttn] Neural Machine Translation by Jointly Learning to Align and Translate
  • Dzmitry Bahdanau (Jacobs University) et al, In ICLR 2015.
• [CopyNet] Incorporating Copying Mechanism in Sequence-to-Sequence Learning
  • Jiatao Gu (The University of Hong Kong) et al, In ACL 2016.
• [PointerNet] Get To The Point: Summarization with Pointer-Generator Networks
  • Abigail See (Stanford University) et al, In ACL 2017.
• [Transformer] Attention Is All You Need
  • Ashish Vaswani (Google) et al, In NeurIPS 2017.

Pretrained language generation models

• [GPT-2] Language Models are Unsupervised Multitask Learners
  • Alec Radford (OpenAI) et al, On OpenAI blog 2019. [official code](tensorflow) [huggingface](pytorch)
• [UniLM] Unified Language Model Pre-training for Natural Language Understanding and Generation
  • Li Dong (Microsoft) et al, In NeurIPS 2019. [official code](pytorch)
• [BART] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Mike Lewis (Facebook AI) et al, On arXiv 2019. [fairseq](pytorch) [huggingface](pytorch)
• [T5] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  • Colin Raffel (Google) et al, In JMLR 2020. [official code](tensorflow) [huggingface](pytorch)

Controllable generation leanrng methods

• [Posterior Regularization] Deep Generative Models with Learnable Knowledge Constraints
  • Zhiting Hu (Carnegie Mellon University) et al, In NeurIPS 2018.
• [Plug and Play] Plug and Play Language Models: A Simple Approach to Controlled Text Generation
  • Sumanth Dathathri (Uber AI) et al, In ICLR 2020. [code](pytorch)
• [Backprop-based Decoding] Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
  • Lianhui Qin (University of Washington) et al, In EMNLP 2020.
• [Weakly Supervision] Summarizing Text on Any Aspects: A Knowledge-Informed Weakly-Supervised Approach
  • Bowen Tan (Carnegie Mellon University) et al, In EMNLP 2020. [code]

Topic-enhanced text generation

• [Dialogue System] Topic Aware Neural Response Generation
  • Chen Xing (Nankai University) et al, In AAAI 2017.
• [Dialogue System] A Neural TopicalExpansion Framework for Unstructured Persona-oriented Dialogue Generation
  • Minghong Xu (Shandong University) et al, In ECAI 2020. [code](tensorflow)
• [Summarization] Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
  • Shashi Narayan (University of Edinburgh) et al, In EMNLP 2018. [code](pytorch)
• [Summarization] Topic-Guided Variational Autoencoders for Text Generation
  • Wenlin Wang (Duke University) et al, In NAACL 2019.
• [Summarization] Document Summarization with VHTM: Variational Hierarchical Topic-Aware Mechanism
  • Xiyan Fu (Nankai Univeristy) et al, In AAAI 2020.
• [Summarization] A Topic Augmented Text Generation Model: Joint Learning of Semantics and Structural Features
  • Peng Cui (Harbin Institute of Technology) et al, In COLING 2020.
• [Machine Translation] Topic-Informed Neural Machine Translation
  • Jian Zhang, (Dublin City University) et al, In COLING 2016.
• [Machine Translation] Translating with Bilingual Topic Knowledge for Neural Machine Translation
  • Xiangpeng Wei (Chinese Academy of Sciences) et al, In AAAI 2019.
• [Topic Transfer] A Topic Augmented Text Generation Model: Joint Learning of Semantics and Structural Features
  • Hongyin Tang (Chinese Academy of Sciences) et al, In EMNLP 2019.

Keyword-enhanced text generation

• [Dialogue System] Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation
  • Lili Mou (Peking University) et al, In COLING 2016. [code](tensorflow)
• [Dialogue System] Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
  • Hao Zhou (Tsinghua University) et al, In AAAI 2018. [code](tensorflow)
• [Dialogue System] Generating Responses with a Specific Emotion in Dialog
  • Zhenqiao Song (Fudan University) et al, In ACL 2019.
• [Summarization] Guiding Generation for Abstractive Text Summarization based on Key Information Guide Network
  • Chenliang Li (Beijing University of Posts and Telecommunications) et al, In NAACL 2018.
• [Summarization] Inferring Search Queries from Web Documents via a Graph-Augmented Sequence to Attention Network
  • Fred X. Han (University of Alberta) et al, In WWW 2019.
• [Summarization] Coherent Comment Generation for Chinese Articles with a Graph-to-Sequence Model
  • Wei Li (Peking University) et al, In ACL 2019. [code](pytorch)
• [Summarization] Keywords-Guided Abstractive Sentence Summarization
  • Haoran Li (JD AI Research) et al, In AAAI 2020.

Knowledge base-enhanced text generation

• [Question Answering] Generating Natural Answers by Incorporating Copying and Retrieving Mechanisms in Sequence-to-Sequence Learning
  • Shizhu He (Chinese Academy of Sciences) et al, In ACL 2017.
• [Question Answering] Natural answer generation with heterogeneous memory
  • Yao Fu (Peking University) et al, In NAACL 2018.
• [Dialogue System] Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems
  • Andrea Madotto (Hong Kong University of Science and Technology) et al, In ACL 2019. [code](pytorch)
• [Dialogue System] Global-to-local Memory Pointer Networks for Task-Oriented Dialogue
  • Chien-Sheng Wu (Hong Kong University of Science and Technology) et al, In ICLR 2019. [code](pytorch)
• [Dialogue System] Improving Knowledge-aware Dialogue Generation via Knowledge Base Question Answering
  • Jian Wang (South China University of Technology) et al, In AAAI 2020. [code](pytorch)
• [Dialogue System] Learning to Select Knowledge for Response Generation in Dialog Systems
  • Rongzhong Lian (Baidu) et al, In IJCAI 2019.
• [Dialogue System] Diverse and Informative Dialogue Generation with Context-Specific Commonsense Knowledge Awareness
  • Sixing Wu (Peking University) et al, In ACL 2020. [code](tensorflow)
• [Dialogue System] TopicKA: Generating Commonsense Knowledge-Aware Dialogue Responses Towards the Recommended Topic Fact
  • Sixing Wu (Peking University) et al, In IJCAI 2020. [code]
• [Content Manipulation] Learning to Select Bi-Aspect Information for Document-Scale Text Content Manipulation
  • Xiaocheng Feng (Harbin Institute of Technology) et al, In AAAI 2020. [code]
• [Content Manipulation] Fact-based Text Editing
  • Hayate Iso (Nara Institute of Science and Technology) et al, In ACL 2020. [code]
• [Summarization] Exploring Human-Like Reading Strategy for Abstractive Text Summarization
  • Min Yang (Chinese Academy of Sciences) et al, In AAAI 2019.
• [Table-to-text] Describing a Knowledge Base
  • Qingyun Wang (UIUC) et al, in INLG 2018. [Official code] (pytorch) [Slides]

Knowledge graph-enhanced text generation

• [Dialogue System] Commonsense Knowledge Aware Conversation Generation with Graph Attention
  • Hao Zhou (Tsinghua University) et al, In IJCAI 2018. [code](tensorflow) [code](pytorch)
• [Dialogue System] Knowledge Aware Conversation Generation with Explainable Reasoning over Augmented Graphs
  • Zhibin Liu, (Baidu) et al, In EMNLP 2019.
• [Dialogue System] DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic Knowledge Graphs
  • Yi-Lin Tuan (National Taiwan University) et al, In EMNLP 2019. [code](tensorflow)
• [Dialogue System] Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs
  • Houyu Zhang (Brown University) et al, In ACL 2020. [code](pytorch)
• [Question Answering] Commonsense for Generative Multi-Hop Question Answering Tasks
  • Lisa Bauer (University of North Carolina at Chapel Hill) et al, In EMNLP 2018. [code](tensorflow)
• [Scientific Writing] Text Generation from Knowledge Graphs with Graph Transformers
  • Rik Koncel-Kedziorski (University of Washington) et al, In NAACL 2018. [code](pytorch)
• [Scientific Writing] PaperRobot: Incremental Draft Generation of Scientific Ideas
  • Qingyun Wang (Rensselaer Polytechnic Institute) et al, In ACL 2019. [code](pytorch)
• [Story Generation] Story Ending Generation with Incremental Encoding and Commonsense Knowledge
  • Jian Guan (Tsinghua University) et al, In AAAI 2019. [code](tensorflow)
• [Story Generation] Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph
  • Haozhe Ji (Tsinghua University) et al, In EMNLP 2020. [code]
• [Story Generation] KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning
  • Ye Liu (University of Illinois at Chicago) On arXiv 2020.
• [Machine Translation] Knowledge Graphs Enhanced Neural Machine Translation
  • Yang Zhao (Chinese Academy of Sciences) et al, In IJCAI 2020.
• [Summarization] Incorporating Commonsense Knowledge into Abstractive Dialogue Summarization via Heterogeneous Graph Networks
  • Xiachong Feng (Harbin Institute of Technology) et al, On arXiv 2020.

Open knowledge graph-enhanced text generation
(Knowledge graph constructed by OpenIE)

• [Question Answering] Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs
  • Angela Fan (Facebook AI Research) et al, In EMNLP 2019.
• [Summarization] Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward
  • Luyang Huang (Northeastern University) et al, In ACL 2020. [code](pytorch)
• [Summarization] Boosting Factual Correctness of Abstractive Summarization with Knowledge Graph
  • Chenguang Zhu (Microsoft Research) et al, On arXiv 2020.
• [Summarization] Heterogeneous Graph Neural Networks for Extractive Document Summarization
  • Danqing Wang (Fudan University) et al, In ACL 2020. [code](pytorch)

Grounded text-enhanced text generation

• [Dialogue System] A Knowledge-Grounded Neural Conversation Model
  • Marjan Ghazvininejad (University of Southern California) et al, In AAAI 2018. [data]
• [Dialogue System] Wizard of Wikipedia: Knowledge-Powered Conversational agents
  • Emily Dinan (Facebook AI Research) et al, In ICLR 2019. [code](pytorch)
• [Dialogue System] Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue
  • Byeongchang Kim (Seoul National University) et al, In ICLR 2020. [code](tensorflow)
• [Dialogue System] DeepCopy: Grounded Response Generation with Hierarchical Pointer Networks
  • Semih Yavuz (University of California, Santa Barbara) et al, In SIGDIAL 2019.
• [Dialogue System] Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading
  • Lianhui Qin (University of Washington) et al, In ACL 2019. [code](pytorch)
• [Dialogue System] RefNet: A Reference-aware Network for Background Based Conversation
  • Chuan Meng (Shandong University) et al, In AAAI 2020. [code](tensorflow)
• [Dialogue System] Thinking Globally, Acting Locally: Distantly Supervised Global-to-Local Knowledge Selection for Background Based Conversation
  • Pengjie Ren (University of Amsterdam) et al, In AAAI 2020. [code](pytorch)
• [Question Answering] Generating Well-Formed Answers by Machine Reading with Stochastic Selector Networks
  • Bin Bi (Alibaba) et al, In AAAI 2020.
• [Summarization] Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization
  • Ziqiang Cao (The Hong Kong Polytechnic University) et al, In ACL 2018. [data]
• [Summarization] BiSET: Bi-directional Selective Encoding with Template for Abstractive Summarization
  • Kai Wang (Sun Yat-sen University) et al, In ACL 2019. [code](pytorch)
• [Paraphrase] Paraphrase Generation by Learning How to Edit from Samples
  • Amirhossein Kazemnejad (Iran University of Science and Technology) et al, In ACL 2020.

Knowledge-enhanced pretraining

• [KG + LM] A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
  • Jian Guan (Tsinghua University) et al, In TACL 2020. [code](tensorflow)
• [Commonsense + LM] Abductive Commonsense Reasoning
  • Chandra Bhagavatula (Allen Institute for AI) et al, In ICLR 2020. [code](pytorch)
• [Table + LM] TAPAS: Weakly Supervised Table Parsing via Pre-training
  • Jonathan Herzig (Google) et al, in ACL 2020. [official code] (tensorflow) [video]
• [KG + LM] JAKET: Joint Pre-training of Knowledge Graph and Language Understanding
  • Donghan Yu (Carnegie Mellon University) et al, On arXiv 2020.

Citation

@article{yu2020survey,
  title={A Survey of Knowledge-Enhanced Text Generation},
  author={Yu, Wenhao and Zhu, Chenguang and Li, Zaitang and Hu, Zhiting and Wang, Qingyun and Ji, Heng and Jiang, Meng},
  journal={arXiv preprint arXiv:2010.04389},
  year={2020}
}

Acknowledgement

This page is contributed by Wenhao Yu([e-mail]) and Qingyun Wang([e-mail]).