Grounding: bind the words to the scene.
A fixed synthetic scene, a small grammar, and a rule-based grounder. Type a referring expression, watch it parse into filters, and see which object the phrase actually denotes — before and after the attributes go wrong.
The referring-expression lab
Three toggles, three different failures.
“the red cube”
Two objects satisfy the description. Every local prediction is correct; the phrase is underdetermined. A system that silently picks one has resolved an ambiguity it did not acknowledge.
“the large red cube” at 55%
The true large red cube is observed as orange; the large orange cube is observed as red. The pipeline now returns a different object with the same confidence. Attribute errors are silent: the box is unchanged.
“the teal cylinder”
Closed: teal is not a word in the lexicon, so the parse fails before any grounding. Open: the word resolves, and the cylinder is selected. An open lexicon widens the names; it does not add evidence.
The scene is drawn, the “perception” is a fixed corruption schedule, and the “open vocabulary” is a hand-written synonym table plus one new colour. There is no detector, no CLIP, no learned parser, and no 3D frame. The lab isolates the mechanism — parse, filter, relate, denote — and shows where the errors enter. It does not estimate how often a real system fails, and it does not prove that any particular model grounds. The guide (§4 and §6) says what the real components add and what they do not fix.
Grounding is a binding, not a caption. A phrase denotes a set of objects in a scene model; a referent is what remains when that set has exactly one member. Detection supplies the candidates, attributes and relations supply the predicates, and the frame supplies the meaning of the spatial words.
Read the frames-of-reference section · Read the open-vocabulary section · Read the failure taxonomy