Semantic Composition in Visually Grounded Language Models
| Author | Rohan Shrirang Pandey |
|---|---|
| Published | 2023 |
| Format | Research paper |
| Original | Read the original |
TL;DR — Rohan Pandey’s 2023 Carnegie Mellon honors thesis on why vision-language AI models fail to compose meaning from word order and relations, with new benchmarks, probes and training fixes.
The thesis, submitted to CMU’s Language Technologies Institute for School of Computer Science honors and supervised by Louis-Philippe Morency and Paul Pu Liang, starts from a known weakness: large language models show real compositional ability, but models that link language to images largely fail to represent compositional structure — for instance, telling “a mug in some grass” from “some grass in a mug.” It asks whether such models compose visually grounded meaning, how they might, and how to make them do it better.
It bundles five contributions. WinogroundVQA is a new compositional visual question-answering benchmark, on which BLIP and MiniGPT-4 score in single digits. Syntactic Neural Module Distillation measures how compositional a sentence-embedding model is, and Causal Tracing for image-captioning models locates the internal representations that matter for vision-language composition. Syntactic MeanPool adds a compositional bias to sentence embeddings, and Cross-modal Attention Congruence Regularization (CACR) is a training objective that aligns relations across text and image; it improves on the prior best method on the Winoground benchmark and outperforms a model nearly twice its size.
The short concluding chapter connects the work to neuroscience, psycholinguistics, formal semantics and philosophy, including the Nyāya and Mīmāṃsā schools of Indian philosophy of language, a thread the author also traces in the introduction alongside Frege.
What it is: An undergraduate honors thesis, posted to arXiv in May 2023 — careful and well supervised, and its acknowledgements thank reviewers of the separate papers it gathers, but a student thesis, not a peer-reviewed result. The models it tests are now superseded.
Where this came from
- Will Manidis’s “most underrated PDF” thread, July 2023 — routing record: The Most Underrated PDF on the Internet (July 2023)
50 pages. A copy is archived locally against link rot; the header links the original source.