More Robots Are Coming: Large Multimodal Models (chatgpt) Can Solve Visually Diverse Images Of Parsons Problems

Hou Irene, Man Owen, Mettille Sophie, Gutierrez Sebastian, Angelikas Kenneth, Macneil Stephen. Arxiv 2023

[Paper]    
GPT Interpretability And Explainability Model Architecture Multimodal Models Prompting RAG Reinforcement Learning Tools

The advent of large language models is reshaping computing education. Recent research has demonstrated that these models can produce better explanations than students, answer multiple-choice questions at or above the class average, and generate code that can pass automated tests in introductory courses. These capabilities have prompted instructors to rapidly adapt their courses and assessment methods to accommodate changes in learning objectives and the potential for academic integrity violations. While some scholars have advocated for the integration of visual problems as a safeguard against the capabilities of language models, new multimodal language models now have vision and language capabilities that may allow them to analyze and solve visual problems. In this paper, we evaluate the performance of two large multimodal models on visual assignments, with a specific focus on Parsons problems presented across diverse visual representations. Our results show that GPT-4V solved 96.7% of these visual problems, struggling minimally with a single Parsons problem. Conversely, Bard performed poorly by only solving 69.2% of problems, struggling with common issues like hallucinations and refusals. These findings suggest that merely transitioning to visual programming problems might not be a panacea to issues of academic integrity in the generative AI era.

Similar Work