LUNA: A Framework For Language Understanding And Naturalness Assessment
Saidov Marat, Bakalova Aleksandra, Taktasheva Ekaterina, Mikhailov Vladislav, Artemova Ekaterina. Arxiv 2024
[Paper]
The evaluation of Natural Language Generation (NLG) models has gained increased attention, urging the development of metrics that evaluate various aspects of generated text. LUNA addresses this challenge by introducing a unified interface for 20 NLG evaluation metrics. These metrics are categorized based on their reference-dependence and the type of text representation they employ, from string-based n-gram overlap to the utilization of static embeddings and pre-trained language models. The straightforward design of LUNA allows for easy extension with novel metrics, requiring just a few lines of code. LUNA offers a user-friendly tool for evaluating generated texts.
Similar Work