Do Vision And Language Models Share Concepts? A Vector Space Alignment Study
Li Jiaang, Kementchedjhieva Yova, Fierro Constanza, Søgaard Anders. Arxiv 2023
[Paper]
Large-scale pretrained language models (LMs) are said to lack the ability
to connect utterances to the world'' (Bender and Koller, 2020), because they do
not havemental models of the world’ ‘(Mitchell and Krakauer, 2023). If so,
one would expect LM representations to be unrelated to representations induced
by vision models. We present an empirical evaluation across four families of
LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures
(ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge
towards representations isomorphic to those of vision models, subject to
dispersion, polysemy and frequency. This has important implications for both
multi-modal processing and the LM understanding debate (Mitchell and Krakauer,
2023).
Similar Work