Gpt-4v(ision) As A Generalist Evaluator For Vision-language Tasks

Zhang Xinlu, Lu Yujie, Wang Weizhi, Yan An, Yan Jun, Qin Lianke, Wang Heng, Yan Xifeng, Wang William Yang, Petzold Linda Ruth. Arxiv 2023

[Paper]    
GPT Interpretability And Explainability Model Architecture Multimodal Models RAG Reinforcement Learning

Automatically evaluating vision-language tasks is challenging, especially when it comes to reflecting human judgments due to limitations in accounting for fine-grained details. Although GPT-4V has shown promising results in various multi-modal tasks, leveraging GPT-4V as a generalist evaluator for these tasks has not yet been systematically explored. We comprehensively validate GPT-4V’s capabilities for evaluation purposes, addressing tasks ranging from foundational image-to-text and text-to-image synthesis to high-level image-to-image translations and multi-images to text alignment. We employ two evaluation methods, single-answer grading and pairwise comparison, using GPT-4V. Notably, GPT-4V shows promising agreement with humans across various tasks and evaluation methods, demonstrating immense potential for multi-modal LLMs as evaluators. Despite limitations like restricted visual clarity grading and real-world complex reasoning, its ability to provide human-aligned scores enriched with detailed explanations is promising for universal automatic evaluator.

Similar Work