In-context Example Selection With Influences

Nguyen Tai, Wong Eric. Arxiv 2023

[Paper]    
Ethics And Bias Few Shot In Context Learning Prompting Tools

In-context learning (ICL) is a powerful paradigm emerged from large language models (LLMs). Despite its promises, ICL performance is known to be highly sensitive to input examples. In this work, we use \(\textit{in-context influences}\) to analyze few-shot ICL performance directly from the in-context examples. Our proposed influence-based example selection method can identify both positive and negative examples, outperforming several baselines when evaluated on 9 SuperGLUE tasks. Our analysis uncovers up to a \(16.3%\) performance gap between using the most negative in-context examples compared to the most positive. In a case study, we apply our influence-based framework to quantify the phenomena of recency bias in example ordering for few-shot ICL.

Similar Work