Instructpipe: Building Visual Programming Pipelines With Human Instructions

Zhou Zhongyi, Jin Jing, Phadnis Vrushank, Yuan Xiuxiu, Jiang Jun, Qian Xun, Zhou Jingtao, Huang Yiyi, Xu Zheng, Zhang Yinda, Wright Kristen, Mayes Jason, Sherwood Mark, Lee Johnny, Olwal Alex, Kim David, Iyengar Ram, Li Na, Du Ruofei. Arxiv 2023

[Paper]    

Visual programming provides beginner-level programmers with a coding-free experience to build their customized pipelines. Existing systems require users to build a pipeline entirely from scratch, implying that novice users need to set up and link appropriate nodes all by themselves, starting from a blank workspace. We present InstructPipe, an AI assistant that enables users to start prototyping machine learning (ML) pipelines with text instructions. We designed two LLM modules and a code interpreter to execute our solution. LLM modules generate pseudocode of a target pipeline, and the interpreter renders a pipeline in the node-graph editor for further human-AI collaboration. Technical evaluations reveal that InstructPipe reduces user interactions by 81.1% compared to traditional methods. Our user study (N=16) showed that InstructPipe empowers novice users to streamline their workflow in creating desired ML pipelines, reduce their learning curve, and spark innovative ideas with open-ended commands.

Similar Work