Instruction-vit: Multi-modal Prompts For Instruction Learning In Vit

Xiao Zhenxiang, Chen Yuzhong, Zhang Lu, Yao Junjie, Wu Zihao, Yu Xiaowei, Pan Yi, Zhao Lin, Ma Chong, Liu Xinyu, Liu Wei, Li Xiang, Yuan Yixuan, Shen Dinggang, Zhu Dajiang, Liu Tianming, Jiang Xi. Arxiv 2023

[Paper]    
Fine Tuning Model Architecture Pretraining Methods Prompting Training Techniques Transformer

Prompts have been proven to play a crucial role in large language models, and in recent years, vision models have also been using prompts to improve scalability for multiple downstream tasks. In this paper, we focus on adapting prompt design based on instruction tuning into a visual transformer model for image classification which we called Instruction-ViT. The key idea is to implement multi-modal prompts (text or image prompt) related to category information to guide the fine-tuning of the model. Based on the experiments of several image captionining tasks, the performance and domain adaptability were improved. Our work provided an innovative strategy to fuse multi-modal prompts with better performance and faster adaptability for visual classification models.

Similar Work