Mixture Of Insightful Experts (mote): The Synergy Of Thought Chains And Expert Mixtures In Self-alignment

Liu Zhili, Gou Yunhao, Chen Kai, Hong Lanqing, Gao Jiahui, Mi Fei, Zhang Yu, Li Zhenguo, Jiang Xin, Liu Qun, Kwok James T.. Arxiv 2024

[Paper]    
Agentic Efficiency And Optimization Fine Tuning Model Architecture Pretraining Methods Reinforcement Learning Training Techniques

As the capabilities of large language models (LLMs) have expanded dramatically, aligning these models with human values presents a significant challenge. Traditional alignment strategies rely heavily on human intervention, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), or on the self-alignment capacities of LLMs, which usually require a strong LLM’s emergent ability to improve its original bad answer. To address these challenges, we propose a novel self-alignment method that utilizes a Chain of Thought (CoT) approach, termed AlignCoT. This method encompasses stages of Question Analysis, Answer Guidance, and Safe Answer production. It is designed to enable LLMs to generate high-quality, safe responses throughout various stages of their development. Furthermore, we introduce the Mixture of insighTful Experts (MoTE) architecture, which applies mixture of experts to enhance each component of the AlignCoT process, markedly increasing alignment efficiency. The MoTE approach not only outperforms existing methods in aligning LLMs with human values but also highlights the benefits of using self-generated data, revealing the dual benefits of improved alignment and training efficiency.

Similar Work