Llambert: Large-scale Low-cost Data Annotation In NLP

Csanády Bálint, Muzsai Lajos, Vedres Péter, Nádasdy Zoltán, Lukács András. Arxiv 2024

[Paper]    
BERT Fine Tuning GPT Model Architecture Pretraining Methods RAG Survey Paper Training Techniques Transformer

Large Language Models (LLMs), such as GPT-4 and Llama 2, show remarkable proficiency in a wide range of natural language processing (NLP) tasks. Despite their effectiveness, the high costs associated with their use pose a challenge. We present LlamBERT, a hybrid approach that leverages LLMs to annotate a small subset of large, unlabeled databases and uses the results for fine-tuning transformer encoders like BERT and RoBERTa. This strategy is evaluated on two diverse datasets: the IMDb review dataset and the UMLS Meta-Thesaurus. Our results indicate that the LlamBERT approach slightly compromises on accuracy while offering much greater cost-effectiveness.

Similar Work