Course / Lecture 16 / Lab Lab 16

Grounding: bind the words to the scene.

A fixed synthetic scene, a small grammar, and a rule-based grounder. Type a referring expression, watch it parse into filters, and see which object the phrase actually denotes — before and after the attributes go wrong.

10 objects 6 preset expressions 0–100% attribute noise No install · runs in the browser

The referring-expression lab

A synthetic scene of ten objects Ten flat vector objects on a grid: a large red cube, a small red cube, a small blue sphere, a large slate sphere, a large sand cube, a small teal sphere, a large orange cube, a large teal cylinder, a large cream cube, and a small slate sphere. Object identifiers run from o1 to o10. The scene is drawn back to front by a depth attribute.
referent ambiguous set relation anchor depth is a scene attribute, not a pixel coordinate

Grammar. the [size] [colour] shape, optionally followed by left of, right of, above, below, behind, or in front of and a second expression. Adjectives are intersected; the relation is existential over the reference set.

0%
Vocabulary
Ready. Ground “the large red cube”.
Pipeline trace
    WHAT TO NOTICE

    Three toggles, three different failures.

    EXPRESSION 01 · AMBIGUITY

    “the red cube”

    Two objects satisfy the description. Every local prediction is correct; the phrase is underdetermined. A system that silently picks one has resolved an ambiguity it did not acknowledge.

    EXPRESSION 02 · NOISE

    “the large red cube” at 55%

    The true large red cube is observed as orange; the large orange cube is observed as red. The pipeline now returns a different object with the same confidence. Attribute errors are silent: the box is unchanged.

    EXPRESSION 03 · VOCABULARY

    “the teal cylinder”

    Closed: teal is not a word in the lexicon, so the parse fails before any grounding. Open: the word resolves, and the cylinder is selected. An open lexicon widens the names; it does not add evidence.

    What this lab is not

    The scene is drawn, the “perception” is a fixed corruption schedule, and the “open vocabulary” is a hand-written synonym table plus one new colour. There is no detector, no CLIP, no learned parser, and no 3D frame. The lab isolates the mechanism — parse, filter, relate, denote — and shows where the errors enter. It does not estimate how often a real system fails, and it does not prove that any particular model grounds. The guide (§4 and §6) says what the real components add and what they do not fix.

    The one sentence

    Grounding is a binding, not a caption. A phrase denotes a set of objects in a scene model; a referent is what remains when that set has exactly one member. Detection supplies the candidates, attributes and relations supply the predicates, and the frame supplies the meaning of the spatial words.

    Read the frames-of-reference section · Read the open-vocabulary section · Read the failure taxonomy