Gaud\'i: Conversational Interactions With Deep Representations To Generate Image Collections

Bursztyn Victor S., Healey Jennifer, Vinay Vishwa. Arxiv 2021

[Paper]    
GPT Language Modeling Model Architecture Multimodal Models Reinforcement Learning

Based on recent advances in realistic language modeling (GPT-3) and cross-modal representations (CLIP), Gaud'i was developed to help designers search for inspirational images using natural language. In the early stages of the design process, with the goal of eliciting a client’s preferred creative direction, designers will typically create thematic collections of inspirational images called “mood-boards”. Creating a mood-board involves sequential image searches which are currently performed using keywords or images. Gaud'i transforms this process into a conversation where the user is gradually detailing the mood-board’s theme. This representation allows our AI to generate new search queries from scratch, straight from a project briefing, following a theme hypothesized by GPT-3. Compared to previous computational approaches to mood-board creation, to the best of our knowledge, ours is the first attempt to represent mood-boards as the stories that designers tell when presenting a creative direction to a client.

Similar Work