Supporting Qualitative Analysis With Large Language Models: Combining Codebook With GPT-3 For Deductive Coding

Xiao Ziang, Yuan Xingdi, Liao Q. Vera, Abdelghani Rania, Oudeyer Pierre-yves. Arxiv 2023

[Paper]    
Fine Tuning GPT Model Architecture Pretraining Methods Prompting Reinforcement Learning Tools Training Techniques

Qualitative analysis of textual contents unpacks rich and valuable information by assigning labels to the data. However, this process is often labor-intensive, particularly when working with large datasets. While recent AI-based tools demonstrate utility, researchers may not have readily available AI resources and expertise, let alone be challenged by the limited generalizability of those task-specific models. In this study, we explored the use of large language models (LLMs) in supporting deductive coding, a major category of qualitative analysis where researchers use pre-determined codebooks to label the data into a fixed set of codes. Instead of training task-specific models, a pre-trained LLM could be used directly for various tasks without fine-tuning through prompt learning. Using a curiosity-driven questions coding task as a case study, we found, by combining GPT-3 with expert-drafted codebooks, our proposed approach achieved fair to substantial agreements with expert-coded results. We lay out challenges and opportunities in using LLMs to support qualitative coding and beyond.

Similar Work