Lecture 16 · Applications

Vision and multimodal grounding.

A detector can box every object and still not answer “the red cube left of the large sphere.” Seeing is not referring. This is the step in between.

5 stages 3 frames 7 failure modes 1 lab
00 · Learning objectivesLecture 16 · Vision and multimodal grounding

By the end of this lecture

Four moves, one scene.

01

Separate the stages

Detection, attributes, relations, structure, grounding — and the claim each one can and cannot make.

02

Parse a referring expression

Read a phrase as a compositional program over sets, with 0, 1, or many referents.

03

Choose a frame

Viewer, intrinsic, absolute: the same spatial word can select different objects.

04

Judge open vocabulary

Say what open-vocabulary grounding fixes, and what it leaves exactly as hard as before.

S/V · Lecture 16
00 · The hookLecture 16 · Vision and multimodal grounding

A child can point. A detector cannot.

A box is not a referent.

“the red cube left of the large sphere”
Schematic scene, not a dataset image.

The detector sees five objects and returns five boxes. The phrase needs one object. The gap is a binding, not a detection.

The phrase is a small program: filter the large sphere, filter the red cubes, keep those left of it, return the set.

If the set has one member, it is the referent. Two: ambiguous. None: the phrase fails to refer.

S/V · Lecture 16
01 · The pipelineLecture 16 · Vision and multimodal grounding
01 / 08

From pixels to predicates.

Five stages, five claims, and the errors each one can and cannot explain.

S/V · Lecture 16
01 · The pipelineLecture 16 · Vision and multimodal grounding

Five stages, five claims

Each stage answers a different question.

01 · Detect

What is where? Box, class, score. Proposals, not referents.

02 · Attribute

What is it like? Unary predicates: colour, size, material.

03 · Relate

How does it stand? Pairwise predicates; needs a frame.

04 · Structure

What is the scene? Objects and edges: a scene graph.

05 · Ground

Which one? Parse and execute; 0, 1, or many.

The stages are not a scorecard. They are an error-attribution device: when the answer is wrong, ask which claim failed before changing the model.

S/V · Lecture 16
01 · The pipelineLecture 16 · Vision and multimodal grounding

Stage 01 · detection

A proposal, not a referent.

A detector scores candidate regions against a fixed label set and regresses their boxes. Output: box, class, score — one row per proposal.

Geometric agreement is measured by intersection over union: IoU = |A∩B| / |A∪B|. It says nothing about the class, the attributes, or the phrase.

Recall at the operating threshold is a hard ceiling on grounding.

prediction reference ∩ overlap · still not a referent
S/V · Lecture 16
01 · The pipelineLecture 16 · Vision and multimodal grounding

Stage 02 · attributes

Unary predicates, silent errors.

Colour

An ontology, not a spectrum. Six names collapse two reds; sixty names make annotators disagree.

Shape

Cube, sphere, cylinder. Often the detector’s class, which is why the boundary between “class” and “attribute” is blurry.

Size · material

Small / large, metal / rubber. Relative and context-dependent; a “large” cube is large for a cube.

The box does not move when the attribute flips. That is why attribute errors are the quietest failure in the stack — and why the lab has a noise slider.
S/V · Lecture 16
01 · The pipelineLecture 16 · Vision and multimodal grounding

Stage 03 · relations

A relation is a predicate over two objects.

left_of(o1, o4) is not a property of o1. It is a property of the pair, evaluated in a frame.

Spatial: left, right, above, below, behind, in front. Comparative: larger, same colour. Functional: holding, wearing. Part: wheel of.

Each family needs different evidence: geometry, a comparison, an action model, a partonomy.

The frame parameter

left_ofviewer(o1, o4) = true
left_ofintrinsic(o1, o4) = false

Same boxes. Same scene. Different word meaning.

S/V · Lecture 16
02 · Attributes & scene graphsLecture 16 · Vision and multimodal grounding

Stage 04 · structure

Objects, edges, attribute sets.

o1 · cube o4 · sphere o8 · cylinder o5 · cube left_of right_of in_front_of

G = (O, E, α)
E ⊆ O × R × O
α : O → 2A

Each edge is a triple. Each node carries an attribute set. A question becomes a graph pattern: “red cubes left of o4” is a join.

Visual Genome made this a standard supervision signal (arXiv:1602.07332).

S/V · Lecture 16
02 · Attributes & scene graphsLecture 16 · Vision and multimodal grounding

What the graph buys and costs

A lossy summary, by design.

Buys

Composition made explicit; queries as graph patterns; structure that can be checked, logged, and reused across tasks.

Costs

No events, no uncertainty, no occlusion reasoning, no outside-frame knowledge. Annotation frames are unrecorded; predicate vocabularies drift.

Two dimensions

Image-plane relations are not tabletop relations. “Above” can mean farther away; “behind” is not in the box.

A crowd underneath

Visual Genome’s edges are crowd-sourced. The graph looks formal; the labels are noisy and frame-inconsistent.

S/V · Lecture 16
03 · Referring expressionsLecture 16 · Vision and multimodal grounding
03 / 08

A phrase is a program.

Composition, presupposition, and the frame that decides what “left” means.

S/V · Lecture 16
03 · Referring expressionsLecture 16 · Vision and multimodal grounding

Compositional semantics

Intersect the parts; relate the sets.

⟦red cube⟧ = ⟦red⟧ ∩ ⟦cube⟧
⟦left of x⟧ = { o : ∃r∈⟦x⟧. left_of(o,r) }

the adds: the set must have exactly one member. Zero → failure to refer. Two → ambiguity.

Existential reading: the target may be left of any member of the reference set.

The program

filter(cube,
  filter(red,
    left_of(large_sphere)))

Parse tree in, set out. The executor is symbolic; the predicates may be learned.

ExpressionDenotationFailure mode
red cubeintersection of two setsnone — it is a set, not a choice
the red cube (one)a singletonzero → failure to refer
the red cube (two)a two-element setambiguity — the system must not pick silently
S/V · Lecture 16
03 · Referring expressionsLecture 16 · Vision and multimodal grounding

The result is a set, not a box

Three outcomes, three obligations.

0

No referent

The description is unsatisfied. Report failure, or ask; do not substitute the nearest box.

1

Referent

The presupposition is satisfied. Return the object and the evidence that selected it.

2+

Ambiguous

The description is underspecified. Report the set, or ask a clarifying question.

A detector has no representation for “2+”. That is why the last stage cannot be replaced by a better detector.

S/V · Lecture 16
03 · Referring expressionsLecture 16 · Vision and multimodal grounding

The parameter everyone forgets

Three frames, one word.

FrameOrigin / axesExample
Viewer / relativeCamera; image axes“the cube left of the sphere” in a photo
Intrinsic / objectThe object’s front and left“the robot’s left”, “the car’s left side”
Absolute / geocentricGravity, compass, world“north of the building”, “uphill”

A cube has no intrinsic left. A robot does. The phrase, not the detector, decides which frame applies — and an annotator who is not asked will choose one silently.

S/V · Lecture 16
03 · Referring expressionsLecture 16 · Vision and multimodal grounding

Same scene, same sentence

The frame decides.

viewer frame: left = image-left viewer
front faces you your left its left

“The red cube to the robot’s left” selects the cube on your right. Nothing in the image changed; the meaning of left did.

S/V · Lecture 16
04 · Open vocabularyLecture 16 · Vision and multimodal grounding

The baseline

A fixed label set has three fates for the unknown.

Forced

The new object is squeezed into the nearest known class. The label is wrong with high confidence.

Background

The object is treated as stuff, not an instance. No symbol exists to refer to.

Dropped

The proposal is discarded. The recall ceiling falls; the error is invisible downstream.

In the lab, “the teal cylinder” fails in the closed vocabulary before any grounding happens. The word is not in the lexicon; the parser cannot build the filter.

S/V · Lecture 16
04 · Open vocabularyLecture 16 · Vision and multimodal grounding

The open interface

Score a region against a phrase.

A joint image–text embedding is trained contrastively on image–caption pairs. A region and a phrase become vectors; cosine similarity ranks them.

p(tᵢ | x) = softmax( sim(f(x), g(tᵢ)) / τ )

A similarity is a ranking signal, not a probability and not a denotation.

region “a red cube” “a teal cylinder” “a blue sphere” embed
S/V · Lecture 16
04 · Open vocabularyLecture 16 · Vision and multimodal grounding

Two components, not one model

Localise the phrase. Segment the thing.

GLIP · arXiv:2112.03857

Detection as phrase grounding

Region features and word features are aligned; the output is a box for each phrase. It localises, but it does not build a scene graph or evaluate a definite description.

SAM · arXiv:2304.02643

Class-agnostic masks

Promptable segmentation returns object masks. It answers which pixels, not what is it. Naming needs another model; relations need geometry and a frame.

The combination is a pipeline: segment, name, relate, ground. Each box in that sentence is a chance for a different error.

S/V · Lecture 16
04 · Open vocabularyLecture 16 · Vision and multimodal grounding

What “open” does not buy

A wider lexicon, not a solved world.

Binding

Recognising red and cube does not attach the adjective to the right noun. Swapped-structure pairs still fail.

Relations

Text similarity is weak on order and spatial relations. A phrase is not a scene graph.

Calibration

Scores are not comparable across vocabularies. A threshold tuned on one prompt set may not transfer.

Open world

The text encoder has a visual vocabulary of its own. A genuinely unseen concept can map near the wrong region.

S/V · Lecture 16
05 · VQA & compositionLecture 16 · Vision and multimodal grounding
05 / 08

Questions that require a computation.

CLEVR, NS-CL, and the difference between an answer and an execution.

S/V · Lecture 16
05 · VQA & compositionLecture 16 · Vision and multimodal grounding

The compositional route

Parse, execute, answer.

q → p → execute(p, M) → a

How many red cubes are left of the large sphere?

count(filter(cube,
  filter(red,
    left_of(large_sphere))))

CLEVR supplies the ground-truth program; NS-CL learns the parser from QA pairs.

Why it is diagnostic

Scenes and questions are generated, so a failure on a held-out combination is about composition — not blur, lighting, or an unlucky photo.

Why it is synthetic: rendered primitives, closed vocabulary, 2D relations, generated grammar.

QuestionProgram shapeWhat it tests
How many red cubes …?count(filter(filter(…)))counting + two attribute filters
Is the cube left of the sphere?left_of(cube, sphere)a spatial relation
What colour is the cube behind the cylinder?colour(filter(cube, behind(cylinder)))relation + attribute readout
S/V · Lecture 16
05 · VQA & compositionLecture 16 · Vision and multimodal grounding

Read the benchmark, not the score

Diagnostic because controlled. Synthetic for the same reason.

It isolates

Composition, counting, comparison, and spatial relations — with ground-truth programs and a novel-combination split.

It excludes

Clutter, occlusion, material ambiguity, real lighting, open vocabulary, intrinsic frames, and anything outside the grammar.

It supports

Claims about systematic recombination within the generator’s distribution.

It cannot support

Claims about grounding in a kitchen, a street, or a hospital. High accuracy is evidence about CLEVR.

GQA adds real images but keeps generated questions (arXiv:1902.09506). Realism and control trade off; say which one your claim needs.

S/V · Lecture 16
05 · VQA & compositionLecture 16 · Vision and multimodal grounding

The evidential question

An answer is behaviour, not a trace.

Not a referent

“Yes” does not name the box, the frame, or the set that made it true.

Not a proof

A fluent rationale can be unfaithful to the predictors (arXiv:2305.04388).

Not calibrated

Confidence in text is not a probability over the answer being grounded.

What would count: a box that can be checked, a program that can be executed, or an intervention that changes one factor and shows the answer tracking it.

S/V · Lecture 16
06 · What failsLecture 16 · Vision and multimodal grounding

Seven rows, seven fixes

Name the failure before changing the model.

FailureSymptomTest
Detection missNo candidate; answer is “none” or another objectRecall at threshold; add a proposal
Attribute errorWrong colour or size; box unchangedSwap one attribute; does the answer track?
Relation / frameWrong object; every box is rightHold the scene, change the frame
BindingAttributes on the wrong objectSwapped-structure pairs (Winoground)
Language priorRight answer from the question aloneQuestion-only baseline
HallucinationAn absent object is assertedProbe with absent objects (POPE)
Unfaithful rationaleExplanation does not match computationIntervene on the stated cause
S/V · Lecture 16
06 · What failsLecture 16 · Vision and multimodal grounding

Interactive · parse, noise, vocabulary

Break the grounding yourself.

0%

Ready. The default phrase selects the large red cube.

Deterministic schedule: at 60% the red and orange cubes swap observed colours. Teal is outside the closed lexicon.

S/V · Lecture 16
07 · The tuple & casesLecture 16 · Vision and multimodal grounding

Four systems, one reading grid

Read the guarantee, not the demo.

SystemSymbolic objectNeural partVocabularyGuarantee
NS-CLProgram over slotsPerception + parserClosedEmpirical; exact relative to slots
GLIPPhrase–region alignmentDual encoderOpen phrasesEmpirical; no scene graph
SAM + CLIPMasks, then namesSegmentation + textOpen namingEmpirical; no relations
ConceptGraphs3D object graphFeatures + LLM queryOpenEmpirical; exact relative to map

Every “exact” in the table is conditional on a model produced by a neural component with an error rate. Correctness relative to the model is not correctness relative to the world.

S/V · Lecture 16
07 · The tuple & casesLecture 16 · Vision and multimodal grounding

The sentence to keep

A box is a hypothesis. A referent is a decision.

The decision needs a scene model, a parse, a frame, and a rule for zero, one, and many.

A VLM that skips the decision can still be fluent. It just cannot show you the binding.

S/V · Lecture 16
08 · RecapLecture 16 · Vision and multimodal grounding

What you can do now

Four moves, one lab.

Separate

Detection, attribute, relation, structure, grounding — and the claim each makes.

Parse

A phrase is a program over sets; the definite article adds a cardinality requirement.

Frame

Viewer, intrinsic, absolute. The same word can select different objects.

Judge

Open vocabulary widens the lexicon; binding and relations remain.

S/V · Lecture 16
Speaker notes