Fewnlu: Benchmarking State-of-the-art Methods For Few-shot Natural Language Understanding

Zheng Yanan, Zhou Jing, Qian Yujie, Ding Ming, Liao Chonghua, Li Jian, Salakhutdinov Ruslan, Tang Jie, Ruder Sebastian, Yang Zhilin. Arxiv 2021

[Paper]    
Applications Attention Mechanism Few Shot Model Architecture Tools

The few-shot natural language understanding (NLU) task has attracted much recent attention. However, prior methods have been evaluated under a disparate set of protocols, which hinders fair comparison and measuring progress of the field. To address this issue, we introduce an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. Under this new evaluation framework, we re-evaluate several state-of-the-art few-shot methods for NLU tasks. Our framework reveals new insights: (1) both the absolute performance and relative gap of the methods were not accurately estimated in prior literature; (2) no single method dominates most tasks with consistent performance; (3) improvements of some methods diminish with a larger pretrained model; and (4) gains from different methods are often complementary and the best combined model performs close to a strong fully-supervised baseline. We open-source our toolkit, FewNLU, that implements our evaluation framework along with a number of state-of-the-art methods.

Similar Work