Course / Act V · Applications Lecture 16

Vision and multimodal grounding.

A detector can put a box around every object in a scene and still not answer “the red cube left of the large sphere.” Seeing is not referring. This lecture is about the step in between: binding words to objects, attributes, and relations that a machine can inspect.

~50 min read 9 sections 2 figures 1 interactive lab Updated 2026-11-17

A sentence can point into a scene with more precision than any label. “The red cube left of the large sphere” does not ask for a caption. It asks for one object, and it asks the system to justify why that object and not another. The gap between captioning and referring is where vision becomes symbolic.

00 / The hookA sentence, a scene, a unique object.

Picture a tabletop with ten objects: a large slate sphere near the centre, two red cubes, a large orange cube, a teal cylinder, and a few others. A detector returns ten boxes with classes and scores. Ask it “which one is the red cube left of the large sphere?” and the question does not compute: it has no notion of the, no relation, and no set with exactly one member.

The phrase is a small program. Find the large sphere; find the cubes; keep the ones left of that sphere; return the set. If the set has one member, the referent is determined. If it has two, the sentence is ambiguous. If it has none, the sentence fails to refer. This is not a stylistic preference for programs. It is what the definite article means: the presupposes that there is exactly one thing satisfying the description.

The older name for attaching symbols to experience is the symbol grounding problem (Harnad, The symbol grounding problem, Physica D 42, 1990). The version here is narrower: attaching a phrase to an object in a scene model that perception itself produced.

filter( cube, filter( red, left_of( large_sphere ) ) )
(0.1)

The grounding step evaluates that program against a scene. It needs a representation (objects, attributes, relations), a semantics for the words (what counts as red, what counts as left), and a decision rule for zero, one, or many matches. A scene graph is one representation; a referring expression is one such program in natural language.

Now change one word: “the red cube to the robot’s left.” The phrase invokes a different frame of reference. If the robot faces the camera, its left is the viewer’s right, and the answer flips. The pixels did not change; the anchoring of the relation did. A spatial word is not a coordinate until a frame is chosen.

Definition · grounding

Given an expression e, a scene model M, and a frame f, grounding is the computation of the denotation ⟦e⟧, the set of objects in M that satisfy e. A referent exists when that set has exactly one member. Detection proposes candidates, captioning describes the whole image, grounding selects a candidate under a compositional interpretation.

The canonical demonstration is the Neuro-Symbolic Concept Learner. A neural perception module turns the image into object slots with attributes; a learned parser turns the question into a program; a symbolic executor runs the program over the slots. It is trained from question–answer pairs, without program annotations (Mao et al., arXiv:1904.12584). The symbolic part supplies composition; the neural part supplies the predicates.

Caveat · what the hook does not show

The scene is synthetic, the vocabulary is closed, and the relations are image-plane relations. The lab is simpler still: ten drawn objects and a hand-written parser. Neither shows how a system behaves in a kitchen. §4 and §6 take up that gap.

01 / The pipelineFrom pixels to predicates, one claim at a time.

The standard pipeline has five stages, and each one makes a different kind of claim. Keeping them separate is the difference between “the model was wrong” and “the model misread the colour.”

The perception-to-grounding pipeline An image is passed through detection, attribute prediction, relation prediction, scene-graph construction, and grounding. Under each stage is its output and its characteristic failure: missed objects, silent attribute errors, frame errors, annotation noise, and ambiguity. SCHEMATIC · THE PIPELINE IS AN ANALYSIS, NOT A REQUIRED ARCHITECTURE image Detect box · class · score N proposals failure: missed object Attribute colour, size, material unary predicates failure: silent flip Relate left, above, behind pairwise predicates failure: wrong frame Structure objects · edges scene graph failure: noisy edge Ground parse · execute denotation failure: ambiguity what is where what it is like how it relates what the scene is which one ERRORS ARE LOCAL, BUT THEY COMPOSE: a missed object removes a candidate forever; a flipped attribute changes the denotation silently; a wrong frame selects a different object while every local prediction stays correct.
Fig. 1.1 A schematic of the stages this lecture separates. Real systems fuse them, train them jointly, or replace the whole stack with one vision-language model. The separation is analytical: it tells you where to look when the answer is wrong. No stage is a photograph or a reproduction.

Detection answers what is where for a fixed label set. A detector scores candidate regions and regresses boxes; its output is a set of instances, each with a class and a score. The geometric part is usually measured by intersection over union:

IoU(A, B) = |A ∩ B| / |A ∪ B|
(1.1)

IoU says nothing about the class, the attributes, or the referent. It measures geometric agreement: a detector can have a high IoU and the wrong label. The metric is local by design.

Attributes answer what it is like: colour, size, material, state, pose. They are unary predicates over a detected instance, from a multi-head classifier or from matching the region embedding to text. Attribute errors are quiet: the box does not move, and the score often stays high. §6 shows why that matters.

Relations answer how it relates: left of, above, behind, larger than, holding, part of. They are predicates over pairs or tuples, and they need a frame and a way to resolve the other argument. A relation is not a property of a box. It is a property of a box under an interpretation.

Structure assembles the objects and relations into a scene graph. Grounding parses an expression and evaluates it against that structure. The last stage is the only one that can return “ambiguous”; the earlier stages cannot represent the question.

Caveat · the pipeline is a diagnostic, not a doctrine

Modern vision-language models do not expose these stages. They map image and text to an answer in one network, and the intermediate objects may not exist anywhere. The pipeline is still the right way to audit a failure: ask which stage could have produced it, then design an intervention that isolates it. A model that cannot answer “where did the colour come from?” is harder to audit, not exempt from the question.

02 / Attributes & scene graphsThe scene as a set of objects and edges.

An attribute is a unary property with a value from an ontology. Colour might be {red, orange, blue, cream, sand, slate, teal}; shape {cube, sphere, cylinder}; size {small, large}. The ontology decides what the system can say and distinguish: six colour names collapse two reds; sixty make annotation expensive.

A relation is an n-ary predicate. Spatial relations matter most for grounding: left of, right of, above, below, behind, in front of, near, touching, inside. Comparative (larger than), functional (holding), and part (wheel of) relations are the other common families, and each needs different evidence: geometry, a comparison, an action model, or a partonomy.

A scene graph packages the two into a typed graph. Let O be a set of object nodes, R a set of relation labels, and A a set of attribute assignments such as colour=red or size=large. A scene graph is a tuple:

G = (O, E, α),   E ⊆ O × R × O,   α : O → 2A
(2.1)

Each node carries attribute values; each edge is a triple (subject, relation, object). In the lab, a fragment is (o1, left_of, o4), (o1, colour, red), (o8, behind, o9). The graph is not a picture with labels; it is a data structure that supports queries, and “left of o4 and red?” becomes a join.

The best-known crowd-sourced resource of this kind is Visual Genome, which pairs images with region descriptions, objects, attributes, and relationships (Krishna et al., arXiv:1602.07332). It made scene graphs a standard signal and exposed the difficulty: the predicate vocabulary is open, the annotators are many, and the frames are not recorded. “Left of” in one annotation may be image-left; in another, the left side of a person.

The representation makes composition explicit: a question compiles into a graph pattern, executes, and checks. The cost is a lossy, static, 2D summary with no events, uncertainty, occlusion reasoning, or anything outside the frame. That is the price of turning a scene into symbols.

03 / Referring expressionsComposition, presupposition, and the frame.

A referring expression is a noun phrase used to pick out an object: the red cube, the large sphere behind the cylinder, the cup on the left. Its meaning can be given denotationally. Relative to a scene model M, the adjective red denotes the set of red objects, the noun cube denotes the set of cubes, and intersection composes them:

⟦red cube⟧M = ⟦red⟧M ∩ ⟦cube⟧M
(3.1)

A spatial modifier is a function from a reference set to a set of candidates. For a frame f, left of the sphere denotes every object that stands in the left_of relation to some member of the sphere’s denotation:

⟦left of x⟧M,f = { o ∈ O : ∃r ∈ ⟦x⟧M. left_off(o, r) }
(3.2)

This is the existential reading: the target only has to be left of one sphere. If the reference expression denotes several spheres, the phrase inherits the ambiguity; if it denotes none, the relation is empty and the whole expression denotes the empty set. The definite article adds the presupposition that the final set has exactly one member. With zero members the description fails to refer; with two it is ambiguous. Neither is a detection error.

The parser bridges sentence and program. A small grammar suffices for the lab: the [size] [colour] shape, optionally followed by a relation and a second expression. The program is a nest of filters; the executor evaluates them and returns the set. The key property is compositionality: the meaning of the whole is a function of the meanings of the parts and their syntactic combination. A model that answers without that structure can be right, but cannot show the combination.

Now the frame. The same scene admits at least three readings of left:

Table 3.1 · Three frames of reference. The frame is part of the predicate’s meaning; choosing it is a semantic decision, not a preprocessing detail.
FrameOrigin and axesExampleTypical failure
Viewer / relativeThe camera; axes are the image axes“the cube left of the sphere” in a photographTreating it as the object’s left, or ignoring that the camera may be rotated
Intrinsic / objectThe object’s front, back, left, right“the robot’s left”, “the car’s left side”Applying it to an object with no intrinsic front (a cube, a sphere)
Absolute / geocentricGravity, compass, or a fixed world frame“north of the building”, “uphill”Conflating image-up with world-up
Two frames of reference for the same word On the left, a viewer looks at a scene and “left of the sphere” means image-left. On the right, a robot faces the viewer, so the robot’s own left is the viewer’s right. The two frames select different objects. A · VIEWER FRAME B · INTRINSIC FRAME sphere A B “left of the sphere” → A viewer robot front faces you your left its left “the robot’s left” → the red cube on your right
Fig. 3.1 The same word, two frames, two referents. Panel A uses the viewer’s axes; panel B uses the robot’s intrinsic left. When the robot faces the camera, the two frames disagree. The drawing is a schematic, not a photograph of a deployment.

The psychology is not a curiosity. Spatial language is frame-dependent: the same above can be read relative to the viewer, the object, or gravity, and the preferred frame changes with scene and task (Carlson-Radvansky & Irwin, Cognition 46, 1993; Levinson, Space in Language and Cognition, CUP, 2003). An annotator who is not asked for a frame supplies one unconsciously. That is one reason relation labels are noisy even when the boxes are perfect.

There is a second ambiguity specific to images. In a photograph, above is usually image-up: smaller y. On a tabletop, image-up is often farther away, not higher. A relation computed from the 2D box is a relation in the image plane, not in the 3D scene. The lab keeps depth as a separate scene attribute: behind is not derivable from the box, and the pipeline must be given the depth. A real system gets depth from stereo, a sensor, or a monocular model, each with its own error.

04 / Open vocabularyNames that were not in the training set.

A closed-vocabulary detector ends in a classifier over K labels, with output p(c | x). An object whose category is outside K is forced into the nearest known class, labelled background, or dropped. None of these is a referential failure; the vocabulary simply had no name.

An open-vocabulary system removes the fixed final layer. It learns a joint embedding space for images and text, usually with a contrastive objective over image–caption pairs (CLIP; Radford et al., arXiv:2103.00020). A region and a phrase are compared by cosine similarity, so a query can be any phrase the text encoder represents:

p(ti | x) = exp(sim(f(x), g(ti)) / τ) / Σj exp(sim(f(x), g(tj)) / τ)
(4.1)

The equation is a softmax over candidate texts, not over a fixed ontology. That is the trick, and the limit. The text encoder is a function, not a lexicon: it maps teal, crimson, and a shade between blue and green to nearby vectors, but it does not tell you which objects are teal. Similarity is a ranking signal, not a probability and not a denotation.

Multimodal grounding adds the region. GLIP formulates detection as phrase grounding: region features and word features are aligned, and the output is a box for each phrase in a caption or prompt (Li et al., arXiv:2112.03857). It can localise phrases, but it does not produce a scene graph or evaluate a definite description.

Segment Anything separates segmentation from naming. It is a promptable, class-agnostic mask model: given points, boxes, or a coarse mask, it returns object masks (Kirillov et al., arXiv:2304.02643). It answers which pixels, not what is it. Naming needs another model; relations need geometry and a frame.

ConceptGraphs is the full stack for 3D: an object-centric map from posed RGB-D frames, open-vocabulary features per object, spatial and semantic edges, and language queries over the graph (Gu et al., arXiv:2309.16650). The symbolic object is the graph; the neural part supplies features and language embeddings; the interface is a pipeline ending in a graph query.

Table 4.1 · Closed and open vocabulary change what can be named. They do not change what has to be reasoned about.
QuestionClosed vocabularyOpen vocabulary
Where does the label set come from?Fixed at training timeSupplied at query time as text
What happens to a new word?Forced, rejected, or droppedEncoded and scored against regions
What is the output?A class distribution over KA similarity ranking over arbitrary phrases
Is the score calibrated?Often, with temperature scalingUsually not across vocabularies
Does it solve relations?NoNo; relations still need structure
Does it solve binding?NoNo; a phrase still has to denote a set
Caveat · open vocabulary is not an open world

The text encoder has its own visual vocabulary, learned from training. A word it has never seen in a visual context may map near the wrong region. A phrase that composes familiar words in an unfamiliar way can still fail: contrastive training on whole captions does not guarantee sensitivity to word order or swapped relations, and Winoground was designed to test exactly this (Thrush et al., arXiv:2204.03162). “Open” describes the label interface, not knowledge of the world.

05 / VQA & compositionQuestions that require a computation.

Visual question answering pairs an image with a natural-language question and asks for an answer: a word, a number, a short phrase. The task is deceptively broad. What colour is the cube? is a lookup. How many red cubes are left of the large sphere? is a count over a filtered set. Is the cylinder behind the cube that is left of the sphere? is a nested relation. A model can do well on the first and fail on the third; aggregate accuracy may not show it.

The compositional approach parses the question into a program and executes it over a scene representation. CLEVR is the cleanest benchmark for this: scenes are generated from object attributes, questions are generated from functional programs, and the ground-truth program and scene are known (Johnson et al., arXiv:1612.06890). The answer is the result of an execution:

a = execute(p, M),   p = parse(q)
(5.1)

CLEVR was designed to defeat the shortcuts of earlier VQA datasets. If every banana is yellow, a model can answer “what colour is the banana?” without looking. CLEVR varies the attributes in the generated scenes, so the language prior is weak and the image has to be used. It also includes a compositional split with novel colour–shape combinations, which tests compositional generalization rather than interpolation within a familiar combination.

NS-CL adds learning. The parser is learned from question–answer pairs, perception produces object slots, and the executor is symbolic. A curriculum over the compositional space teaches the system to ground concepts without attribute labels (Mao et al., arXiv:1904.12584). The intermediate object is inspectable: you can read the program, run it, and see which objects each filter kept.

Why is CLEVR diagnostic? Because scene and question are controlled, a failure can be attributed: if the model fails on a held-out combination, the failure is about composition, not blur or lighting. Why is it synthetic? The objects are rendered primitives, the vocabulary is tiny and closed, the relations are 2D image relations, and the questions come from a grammar. A model can learn the renderer’s regularities as well as the concepts. High accuracy on CLEVR is evidence about CLEVR.

GQA moves toward realism: real photographs, scene graphs derived from Visual Genome, and questions generated from those graphs (Hudson & Manning, arXiv:1902.09506). Real images with generated questions add clutter, but they are still not a deployment test. The generator’s grammar is part of the distribution.

06 / What failsSeven ways to be wrong, and what each one looks like.

The failures cluster: perceptual (the object or attribute was not recovered), semantic (the word was read under the wrong frame or ontology), and evidential (the answer was produced without the computation). The fixes differ.

Table 6.1 · A failure taxonomy for grounded vision. The “test” column is the intervention that would distinguish this failure from its neighbours.
FailureSymptomWhere it entersTest
Detection missNo candidate for the referent; the answer is “none” or another objectProposal stage; occlusion, scale, rare classRecall at the operating threshold; add a proposal and see if the answer recovers
Attribute errorWrong colour or size; the box is unchangedAttribute head; illumination, material, ontologySwap one attribute in the scene; does the answer track?
Relation / frame errorWrong object on a spatial phrase; the boxes are rightRelation module; viewer vs intrinsic frame; 2D vs 3DHold the scene fixed and change only the frame in the instruction
Binding errorAttributes attached to the wrong object; swapped rolesJoint embedding; attention over a bag of objectsPaired examples with swapped attributes or roles (Winoground-style)
Language priorCorrect answer from the question alone, wrong imageAnswer head; skewed training distributionBlind baseline: question only, no image
HallucinationAn object is asserted that is not presentGenerative decoder; prior over typical scenesAsk about absent objects (POPE-style negative probes)
Unfaithful rationaleA plausible explanation that does not match the computationChain-of-thought text; post-hoc narrationIntervene on the stated cause; does the answer change as claimed?

Detection is a ceiling. If the detector does not propose the object, no downstream symbol can refer to it: a grounding system cannot recover what perception never saw. The ceiling is measurable — recall at the operating threshold — and should be reported before any claim about reasoning. High accuracy on the objects it sees does not compensate for the objects it missed. When deployment differs from training, recall moves; that is distribution shift, not a reasoning failure.

Attribute errors are silent. The box does not move when the colour head flips. In the lab, at 55% the large red cube is observed as orange and the large orange cube as red, so “the large red cube” selects the wrong object with full confidence. Nothing flags the swap. A real system needs a calibrated attribute score propagated into the denotation, not a hard label.

Relations are the frame. A relation module can be geometrically correct and semantically wrong: it computes left_of in one frame while the instruction uses another. The fix is not a bigger model; it is to make the frame an explicit parameter and to test the same scene under both frames. The same applies to 2D versus 3D: the image-plane relation is not the tabletop relation.

Binding is not classification. A model can recognise red, cube, and sphere and still attach the adjective to the wrong noun. Winoground’s paired captions expose this: the same words in a different structure should select a different image, and a bag-of-words scorer will not reliably do so (Thrush et al., arXiv:2204.03162). POPE probes the complementary failure, asking about objects that are absent (Li et al., arXiv:2305.10355). Both test binding, not vocabulary.

Language priors are a baseline, not a bug. Early VQA models could answer many questions from the question alone because the answer distribution was skewed; VQA v2 paired complementary images to reduce the shortcut (Goyal et al., arXiv:1612.00837). Always report a question-only baseline. If it is close to the model, the benchmark is measuring the prior.

An answer is not a computation. What the answer shows is that the model’s output is consistent with the answer on this input. What it does not show is the referent, the frame, or the computation that would justify them. A vision-language model can produce a fluent answer and a fluent rationale without either being a trace of the computation. Chain-of-thought in language models can be unfaithful to the actual predictors (Turpin et al., arXiv:2305.04388), and attention weights are not a referent or an explanation (Jain & Wallace, arXiv:1902.10186). What counts is a checkable box, an executable program, a validatable structure, or an intervention that changes one factor and shows the answer tracking it.

Caveat · the taxonomy is not a scorecard

No benchmark covers all seven rows. CLEVR covers composition and some relations in a synthetic setting; GQA adds real images; POPE covers hallucination; Winoground covers binding; none covers the frame of reference in a real instruction. A claim that a system “grounds language in vision” should name the benchmark, the split, the baseline, and the failure row it tests.

07 / The tuple & casesFour systems, read the same way.

The course’s tuple works here without modification: (representation, inference, coupling, supervision, guarantee), plus two slots that matter for vision, vocabulary and frame. A system that cannot fill the vocabulary slot has not said what it can name; one that cannot fill the frame slot has not said what its spatial words mean.

Table 7.1 · Four grounded-vision systems along the tuple. “Guarantee” is what the system can promise, not what its demo suggests.
SystemSymbolic objectNeural partInterfaceVocabularyGuarantee
NS-CLProgram over object slotsPerception + parserArchitecture (executor)ClosedEmpirical; executor semantics are exact relative to the slots
GLIPPhrase–region alignmentDual encoder, detection headsJoint pretrainingOpen phrasesEmpirical; no scene graph
SAM + CLIPMasks, then namesPromptable segmentation + text encoderPipelineOpen namingEmpirical; no relations
ConceptGraphs3D object graphOpen-vocab features + LLM queryPipeline over a graphOpenEmpirical; graph queries are exact relative to the map

Read the guarantee column. None guarantees that the referent is correct in the world. The executor guarantees that the program’s result follows from the scene model; the graph query guarantees that the answer follows from the graph. Both are conditional on a model produced by a neural component with an error rate. This is Lecture 15’s planner again: correctness relative to a model is not correctness relative to the world.

The design checklist follows. What is the referent? Name the object set and the rule for 0, 1, and many. What is the frame? Write it down and test the same scene under another. What is the vocabulary? Closed, open, or mixed, and what happens outside it. What is the failure? Which taxonomy row the evaluation probes. What is logged? Candidates, parse, frame, scores, threshold. A system that logs only the final answer cannot be audited; interpretability here is a property of the log, not of a saliency map.

08 / RecapWhat you should be able to do now.

01

Separate grounding from detection: detection proposes, grounding denotes.

02

Name the five stages and the claim each one makes.

03

Define a scene graph and say what it loses.

04

Read a referring expression as a compositional program over sets.

05

Choose a frame of reference and explain how the referent changes with it.

06

Compare closed and open vocabulary, and say what open vocabulary does not fix.

07

Explain why CLEVR is diagnostic and why it is synthetic.

08

Attribute a failure to detection, attribute, relation, frame, binding, prior, or rationale.

Exercises

  1. Three denotations. In the lab, find one expression with zero referents, one with exactly one, and one with two. For each, say whether the cause is the scene, the description, or the vocabulary.
  2. Break the attribute. Set the noise to 55% and ground “the large red cube”. Which two colour flips make the orange cube the only match, and what does the third flip at that level change? Then set the noise to 100% and explain the result.
  3. Add a frame. Sketch the change to the lab that would let the user choose viewer versus intrinsic left. Which of the six presets change their referent, and which do not?
  4. From photograph to phrase. Take a photograph and write five referring expressions. Mark the frame of every spatial word and the presupposition of every definite article. Which have more than one referent?
  5. Audit a VLM. Ask two vision-language models ten paired questions with one attribute or relation swapped. Report answer accuracy and whether the stated rationale names the object that actually changed. Do not report a number you did not measure.

Quiz

Ten questions, with explanations. The score is not the point; the questions you cannot answer are the reading list.

Reading

The Neuro-Symbolic Concept Learner

Mao et al. · ICLR 2019 · programs, object slots, QA-only supervision

Grounding

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Johnson et al. · CVPR 2017 · the diagnostic synthetic benchmark

Benchmark

Learning Transferable Visual Models From Natural Language Supervision

Radford et al. · 2021 · the joint image-text embedding

Open vocab

Grounded Language-Image Pre-training

Li et al. · CVPR 2022 · detection as phrase grounding

Grounding

Segment Anything

Kirillov et al. · ICCV 2023 · promptable, class-agnostic masks

Segmentation

ConceptGraphs: Open-Vocabulary 3D Scene Graphs

Gu et al. · 2023 · objects, relations, and language queries in 3D

3D

Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

Thrush et al. · CVPR 2022 · swapped structure, not swapped words

Limits

Evaluating Object Hallucination in Large Vision-Language Models

Li et al. · 2023 · polling with absent objects

Limits
Source note · figures and scope

Both figures are schematic SVGs drawn for this course; no paper figure is reproduced. The CLEVR and Visual Genome citations identify the datasets, not samples from them. Every arXiv ID was checked against the arXiv API on 2026-11-17. The book and the 1993 Cognition article are cited by author, title, venue, and year, not by an invented DOI.