A Comprehensive Survey Of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO And More

Wang Zhichao James, Bi Bin James, Pentyala Shiva Kumar James, Ramnath Kiran James, Chaudhuri Sougata James, Mehrotra Shubham James, Zixu James, Zhu Claire, Mao Xiang-bo Claire, Asur Sitaram Claire, Na Claire, Cheng. Arxiv 2024

[Paper]    
Fine Tuning Interpretability And Explainability Model Architecture Pretraining Methods Reinforcement Learning Survey Paper Training Techniques Transformer

With advancements in self-supervised learning, the availability of trillions tokens in a pre-training corpus, instruction fine-tuning, and the development of large Transformers with billions of parameters, large language models (LLMs) are now capable of generating factual and coherent responses to human queries. However, the mixed quality of training data can lead to the generation of undesired responses, presenting a significant challenge. Over the past two years, various methods have been proposed from different perspectives to enhance LLMs, particularly in aligning them with human expectation. Despite these efforts, there has not been a comprehensive survey paper that categorizes and details these approaches. In this work, we aim to address this gap by categorizing these papers into distinct topics and providing detailed explanations of each alignment method, thereby helping readers gain a thorough understanding of the current state of the field.

Similar Work