ruotianluo / PromptPapers

Must-read papers on prompt-based tuning for pre-trained language models.

PromptPapers

Must-read papers on prompt-based tuning for pre-trained language models. The paper list is mainly mantained by Ning Ding and Shengding Hu.

Contents

• PromptPapers
  • Contents
  • Introduction
  • Papers
    • Prompt Designing
    • Prompt Analysis
    • Prompt Application
  • Other Contributors

Introduction

This is a paper list about prompt-based tuning for large-scale pre-trained language models. Different from traditional fine-tuning that uses an explicit classifier, prompt-based tuning directly uses the pre-trained models to conduct the pre-training tasks for classification or regression.

Papers

Prompt Designing

1. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR.

   Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. [pdf], [project], (T5). 2019.10.

2. Parameter-Efficient Transfer Learning for NLP. ICML 2019.

   Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly. [pdf], [project], 2019.6

3. How Can We Know What Language Models Know? TACL 2020.

   Zhengbao Jiang, Frank F. Xu, Jun Araki, Graham Neubig. [pdf], [project], 2019.11

4. Language Models are Few-shot Learners. NeurIPS 2020.

   Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei. [pdf], [website] (GPT-3), 2020.5

5. It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. NAACL 2021.

   Timo Schick, Hinrich Schütze. [pdf], [project], 2020.9

6. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. Preprint.

   Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, Sameer Singh. [pdf], [website] (AutoPrompt), 2020.10

7. Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification. COLING 2020.

   Timo Schick, Helmut Schmid, Hinrich Schütze [pdf] [project]

8. Making Pre-trained Language Models Better Few-shot Learners. ACL 2021.

   Tianyu Gao, Adam Fisch, Danqi Chen. [pdf], [project] (LM-BFF), 2020.12

9. Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference. EACL 2021.

   Timo Schick, Hinrich Schütze. [pdf], [project] (PET), 2021.1

10. Prefix-tuning: Optimizing continuous prompts for generation. Preprint.

    Xiang Lisa Li, Percy Liang. [pdf], [project], 2021.1

11. Calibrate Before Use: Improving Few-Shot Performance of Language Models. Preprint.

    Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, Sameer Singh. [pdf], [project], 2021.2

12. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. Preprint.

    Laria Reynolds, Kyle McDonell. [pdf], 2021.2

13. Improving and Simplifying Pattern Exploiting Training. Preprint.

    Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, Colin Raffel. [pdf]

14. The Power of Scale for Parameter-Efficient Prompt Tuning. Preprint.

    Brian Lester, Rami Al-Rfou, Noah Constant. [pdf], [implementation], 2021.4

15. Learning How to Ask: Querying LMs with Mixtures of Soft Prompts. NAACL 2021.

    Guanghui Qin, Jason Eisner [pdf] [project], 2021.4

16. GPT understands, too. Preprint.

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, Jie Tang. [pdf], [project] (P-tuning), 2021.3

17. Factual Probing Is [MASK]: Learning vs. Learning to Recall. NAACL 2021.

    Zexuan Zhong, Dan Friedman, Danqi Chen. [pdf], [project], 2021.4

18. AdaPrompt: Adaptive Prompt-based Finetuning for Relation Extraction. Preprint.

    Xiang Chen, Xin Xie, Ningyu Zhang, Jiahuan Yan, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, Huajun Chen. [pdf], 2021.4

19. PTR: Prompt Tuning with Rules for Text Classification. Preprint.

    Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, Maosong Sun. [pdf] (PTR), 2021.5

Prompt Analysis

1. Language Models as Knowledge Bases? EMNLP 2019.

   Fabio Petroni, Tim Rocktaschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, Sebastian Riedel. [pdf], [project] (LAMA), 2019.9

2. What Makes Good In-Context Examples for GPT-3?. Preprint.

   Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, Weizhu Chen. [pdf] 2021.1

3. How Many Data Points is a Prompt Worth? NAACL 2021. Preprint.

   Teven Le Scao, Alexander M. Rush. [pdf], [project], 2021.3

4. Surface Form Competition-Why the Highest Probability Answer Isn’t Always Right. Preprint. Preprint.

   Ari Holtzman, Peter West, Vered Schwartz, Yejin Choi, Luke Zettlemoyer. [pdf] [project], 2021.4

5. Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions. Preprint.

   Swaroop Mishra, Daniel Khashabi, Chitta Baral, Hannaneh Hajishirzi. [pdf] [project] 2021.4

6. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. Preprint. Preprint.

   Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, Pontus Stenetorp. [pdf] 2021.4

Prompt Application

1. Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2. Preprint.

   Gregor Betz, Kyle Richardson, Christian Voigt. [pdf] 2021.3

2. GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation. Preprint.

   Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, Woomyeong Park. [pdf] 2021.4

3. PADA: A Prompt-based Autoregressive Approach for Adaptation to Unseen Domains Preprint.

   Eyal Ben-David, Nadav Oved, Roi Reichart. [pdf] [project] 2021.5

Other Contributors

We thank Yujia Qin, Xiachong Feng, Chenglei Si for the paper recommendation. Pull requests and issues are welcomed!