louieworth / awesome-rlhf

An index of algorithms for reinforcement learning from human feedback (rlhf))

awesome-rlhf

Welcome to our curated collection of research and review papers focused on Reinforcement Learning from Human Feedback (RLHF). We encourage you to star, fork, and contribute to this repository. We're actively seeking additional contributors and maintainers!

Maintained by:

• Li Jiang, McGill University

Please follow this format for contributions:

- [Paper Title](paper link) [Additional Links]
  - Author1, Author2, and Author3. arXiv/Conference/Journal, Year.

For any inquiries, don't hesitate to contact: [e-mail]

Some notes:

• This resource is dedicated to the latest papers and does not include past academic works, even those published earlier in 2023. For a review of prominent historical papers and other sources, please refer to hugging face blog and this link from OpendiLab.
• Most of the paper collections is credited to RLHF papers.

Table of Contents

• Papers
  • Review/Survey/Position Papers
  • RLHF for LLMs: Theory / Methods
  • RLHF for Other Domains
  • Datasets
• Blogs/Talks/Reports
  • Blogs
  • Talks
  • Reports
• Open Source Software/Implementations

Papers

Review/Survey

• AI Alignment: A Comprehensive Survey
  • Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O'Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, Wen Gao. arXiv, 2023.
• Aligning Large Language Models with Human: A Survey
  • Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, Qun Liu. arXiv, 2023.
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  • Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. arXiv, 2023.

RLHF for LLMs: Theory / Methods

• A Minimaximalist Approach to Reinforcement Learning from Human Feedback
  • Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, Alekh Agarwal. arXiv, 2024.
• Preference as Reward, Maximum Preference Optimization with Importance Sampling.
  • Zaifan Jiang, Xing Huang, Chao Wei. arXiv, 2023.
• Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
  • Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, Huaimin Wang. arXiv, 2023.
• Aligning Large Language Models with Human Preferences through Representation Engineering
  • Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, Xuanjing Huang. arXiv, 2023.
• The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback.
  • Nathan Lambert, Roberto Calandra. arXiv, 2023.
• Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration
  • Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, Willie Neiswanger. arXiv, 2023.
• On the Exploitability of Reinforcement Learning with Human Feedback for Large Language Models
  • Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, Chaowei Xiao. arXiv, 2023.
• Policy Optimization in RLHF: The Impact of Out-of-preference Data.
  • Ziniu Li, Tian Xu, Yang Yu. arXiv, 2023.
• Interpreting Reward Models in RLHF-Tuned Language Models Using Sparse Autoencoders.
  • Luke Marks, Amir Abdullah, Luna Mendez, Rauno Arike, Philip Torr, Fazl Barez. arXiv, 2023.
• SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF.
  • Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, Oleksii Kuchaiev. arXiv, 2023.
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models.
  • Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, Noah Fiedel. arXiv, 2023.
• Nash Learning from Human Feedback.
  • Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, Bilal Piot. arXiv, 2023.
• Interpreting Reward Models in RLHF-Tuned Language Models Using Sparse Autoencoders.
  • Luke Marks, Amir Abdullah, Luna Mendez, Rauno Arike, Philip Torr, Fazl Barez. arXiv, 2023.
• Adversarial Preference Optimization
  • Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, Nan Du. arXiv, 2023.
• Black-Box Prompt Optimization: Aligning Large Language Models without Model Training
  • Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, Minlie Huang. arXiv, 2023.
• Fake Alignment: Are LLMs Really Aligned Well?
  • Yixu Wang, Yan Teng, Kexin Huang, Chengqi Lyu, Songyang Zhang, Wenwei Zhang, Xingjun Ma, Yu-Gang Jiang, Yu Qiao, Yingchun Wang. arXiv, 2023.
• Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment
  • Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, Ji-Rong Wen. arXiv, 2023.
• Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback
  • Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang. arXiv, 2023.
• Is RLHF More Difficult than Standard RL?
  • Yuanhao Wang, Qinghua Liu, Chi Jin. arXiv, 2023.
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, R´emi Munos. arXiv, 2023.
• COPF: Continual Learning Human Preference through Optimal Policy Fitting
  • Han Zhang, Lin Gui, Yuanzhao Zhai, Hui Wang, Yu Lei, Ruifeng Xu. arXiv, 2023.
• SuperHF: Supervised Iterative Learning from Human Feedback.
  • Gabriel Mukobi, Peter Chatain, Su Fong, Robert Windesheim, Gitta Kutyniok, Kush Bhatia, Silas Alberti. arXiv, 2023.
• Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
  • Kai Chen, Chunwei Wang, Kuo Yang, Jianhua Han, Lanqing Hong, Fei Mi, Hang Xu, Zhengying Liu, Wenyong Huang, Zhenguo Li, Dit-Yan Yeung, Lifeng Shang, Xin Jiang, Qun Liu. arXiv, 2023.
• Entangled Preferences: The History and Risks of Reinforcement Learning and Human Feedback
  • Nathan Lambert, Thomas Krendl Gilbert, Tom Zick. arXiv, 2023.
• Group Preference Optimization: Few-Shot Alignment of Large Language Models
  • Siyan Zhao, John Dang, Aditya Grover. arXiv, 2023.
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
  • Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, Yaodong Yang. arXiv, 2023.
• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models.
  • Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, Zhi-Quan Luo. arXiv, 2023.
• Stabilizing RLHF through Advantage Model and Selective Rehearsal
  • Baolin Peng, Linfeng Song, Ye Tian, Lifeng Jin, Haitao Mi, Dong Yu. arXiv, 2023.
• Beyond One-Preference-for-All: Multi-Objective Direct Preference Optimization for Language Models.
  • Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, Yu Qiao. arXiv, 2023.
• A General Theoretical Paradigm to Understand Learning from Human Preferences.
  • Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, Rémi Munos. arXiv, 2023.
• Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment
  • Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, Jiantao Jiao. arXiv, 2023.
• Understanding the Effects of RLHF on LLM Generalisation and Diversity
  • Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, Roberta Raileanu. arXiv, 2023.
• Improving Language Models with Advantage-based Offline Policy Gradients
  • Ashutosh Baheti, Ximing Lu, Faeze Brahman, Ronan Le Bras, Maarten Sap, Mark Riedl. arXiv, 2023.
• SALMON: Self-Alignment with Principle-Following Reward Models
  • Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan. arXiv, 2023.
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. NeurIPS, 2023.
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment.
  • Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang. arXiv, 2023.
• Preference Ranking Optimization for Human Alignment.
  • Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, Houfeng Wang. arXiv, 2023.
• RRHF: Rank Responses to Align Language Models with Human Feedback without tears
  • Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang. NeurIPS, 2023.
• Reward Model Ensembles Help Mitigate Overoptimization
  • Thomas Coste, Usman Anwar, Robert Kirk, David Krueger. arXiv, 2023.
• Learning Optimal Advantage from Preferences and Mistaking it for Reward
  • W. Bradley Knox, Stephane Hatgis-Kessell, Sigurdur Orn Adalgeirsson, Serena Booth, Anca Dragan, Peter Stone, Scott Niekum. arXiv, 2023.
• Enable Language Models to Implicitly Learn Self-Improvement From Data
  • Ziqi Wang, Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, Heng Ji. arXiv, 2023.
• The Trickle-down Impact of Reward (In-)consistency on RLHF
  • Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, Dong Yu. arXiv, 2023.
• Aligning Language Models with Offline Reinforcement Learning from Human Feedback
  • Jian Hu, Li Tao, June Yang, Chandler Zhou. arXiv, 2023.
• Human Feedback is not Gold Standard
  • Tom Hosking, Phil Blunsom, Max Bartolo. arXiv, 2023.
• Fine-Tuning Language Models with Advantage-Induced Policy Alignment
  • Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I. Jordan, Jiantao Jiao. arXiv, 2023.
• Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons.
  • Banghua Zhu, Jiantao Jiao, Michael I. Jordan. arXiv, 2023.
• Making PPO even better: Value-Guided Monte-Carlo Tree Search decoding
  • Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, Asli Celikyilmaz. arXiv, 2023.
• Stabilizing RLHF through Advantage Model and Selective Rehearsal
  • Baolin Peng, Linfeng Song, Ye Tian, Lifeng Jin, Haitao Mi, Dong Yu. arXiv, 2023.
• Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF
  • Simeng Sun, Dhawal Gupta, Mohit Iyyer. arXiv, 2023.
• RAIN: Your Language Models Can Align Themselves without Finetuning
  • Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, Hongyang Zhang, arXiv, 2023.
• Statistical Rejection Sampling Improves Preference Optimization
  • Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, Jialu Liu. arXiv, 2023.
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  • Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, Abhinav Rastogi. arXiv, 2023.
• Reinforced Self-Training (ReST) for Language Modeling
  • Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, Nando de Freitas. arXiv, 2023.
• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  • Hritik Bansal, John Dang, Aditya Grover. arXiv, 2023.
• Let Me Teach You: Pedagogical Foundations of Feedback for Language Models
  • Beatriz Borges, Niket Tandon, Tanja Käser, Antoine Bosselut. arXiv, 2023.
• Generalized Knowledge Distillation for Auto-regressive Language Models
  • Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, Olivier Bachem. arXiv, 2023.
• Secrets of RLHF in Large Language Models Part I: PPO
  • Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang. arXiv, 2023.
• Learning to Generate Better Than Your LLM
  • Jonathan D. Chang, Kiante Brantley, Rajkumar Ramamurthy, Dipendra Misra, Wen Sun. arXiv, 2023.
• Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  • Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, Hannaneh Hajishirzi. NeurIPS, 2023.
• Continually Improving Extractive QA via Human Feedback
  • Ge Gao, Hung-Ting Chen, Yoav Artzi, Eunsol Choi. arXiv, 2023.
• SLiC-HF: Sequence Likelihood Calibration with Human Feedback
  • Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, Peter J. Liu. arXiv, 2023.
• The Wisdom of Hindsight Makes Language Models Better Instruction Followers
  • Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, Joseph E. Gonzalez. arXiv, 2023.

RLHF for Other Domains

• Contrastive Preference Learning: Learning from Human Feedback without RL.
  • Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, Dorsa Sadigh. arXiv, 2023.
• Beyond Reward: Offline Preference-guided Policy Optimization.
  • Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, Donglin Wang. arXiv, 2023.
• PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback
  • Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, Qianxiang Wang. arXiv, 2023.
• WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
  • Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Dongmei Zhang. arXiv, 2023.
• Shepherd: A Critic for Language Model Generation
  • Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, Asli Celikyilmaz. arXiv, 2023.
• Reinforcement Learning with Human Feedback for Realistic Traffic Simulation
  • Yulong Cao, Boris Ivanovic, Chaowei Xiao, Marco Pavone. arXiv, 2023.
• Aligning Large Multimodal Models with Factually Augmented RLHF
  • Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell. arXiv, 2023.
• Motif: Intrinsic Motivation from Artificial Intelligence Feedback
  • Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, Mikael Henaff. arXiv, 2023.

Datasets

• UltraFeedback: Boosting Language Models with High-quality Feedback
  • Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, Maosong Sun. arXiv, 2023.
• OpenAssistant Conversations - Democratizing Large Language Model Alignment.
  • Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, Alexander Julian Mattick. NeurIPS 2023.

Blogs/Talks/Reports

Blogs

• The N Implementation Details of RLHF with PPO
  • Tianlin Liu, Tianlin Liu, Leandro von Werra. Hugging Face blog, 2023.
• Illustrating Reinforcement Learning from Human Feedback (RLHF)
  • Nathan Lambert, Hugging Face blog, 2022.
• Reinforcement Learning for Language Models
  • Yoav Goldberg, April 2023.
• RLHF: Reinforcement Learning from Human Feedback
  • Chip Huyen, 2023.

Talks

• Reinforcement Learning from Human Feedback: From Zero to chatGPT
• Nathan Lambert, Hugging Face, 2022.
• Reinforcement Learning from Human Feedback: Progress and Challenges
  • John Schulman, OpenAI, 2023.
• Scalable Oversight for Large Language Models
  • Samuel Bowman, New York University, 2023

Reports

• Llama 2: Open Foundation and Fine-Tuned Chat Models
  • Meta, 2023.
• GPT-4 Technical Report
  • OpenAI, 2023.

Open Source Software/Implementations

• trl
  • Train transformer language models with reinforcement learning.
• OpenRLHF
  • A Ray-based High-performance RLHF framework (for 34b+ models)