Natural Language Programming In Medicine: Administering Evidence Based Clinical Workflows With Autonomous Agents Powered By Generative Large Language Models

Vaid Akhil, Lampert Joshua, Lee Juhee, Sawant Ashwin, Apakama Donald, Sakhuja Ankit, Soroush Ali, Bick Sarah, Abbott Ethan, Gomez Hernando, Hadley Michael, Lee Denise, Landi Isotta, Duong Son Q, Bussola Nicole, Nabeel Ismail, Muehlstedt Silke, Muehlstedt Silke, Freeman Robert, Kovatch Patricia, Carr Brendan, Wang Fei, Glicksberg Benjamin, Argulian Edgar, Lerakis Stamatios, Khera Rohan, Reich David L., Kraft Monica, Charney Alexander, Nadkarni Girish. Arxiv 2024

[Paper]    
Agent Agentic GPT Model Architecture Prompting RAG Reinforcement Learning Tools

Generative Large Language Models (LLMs) hold significant promise in healthcare, demonstrating capabilities such as passing medical licensing exams and providing clinical knowledge. However, their current use as information retrieval tools is limited by challenges like data staleness, resource demands, and occasional generation of incorrect information. This study assessed the potential of LLMs to function as autonomous agents in a simulated tertiary care medical center, using real-world clinical cases across multiple specialties. Both proprietary and open-source LLMs were evaluated, with Retrieval Augmented Generation (RAG) enhancing contextual relevance. Proprietary models, particularly GPT-4, generally outperformed open-source models, showing improved guideline adherence and more accurate responses with RAG. The manual evaluation by expert clinicians was crucial in validating models’ outputs, underscoring the importance of human oversight in LLM operation. Further, the study emphasizes Natural Language Programming (NLP) as the appropriate paradigm for modifying model behavior, allowing for precise adjustments through tailored prompts and real-world interactions. This approach highlights the potential of LLMs to significantly enhance and supplement clinical decision-making, while also emphasizing the value of continuous expert involvement and the flexibility of NLP to ensure their reliability and effectiveness in healthcare settings.

Similar Work