Grounding and composition, made visible.
One grid, two questions. Which object does the expression pick out — and does the rule that picks it out survive a combination it has never seen? Change the frame, then change the training set.
The grounding and composition lab
Grounding: which object?
The grid has six objects. The expression names a colour, a shape, and optionally a relation to a reference object. The frame decides what “left” means; the status says whether the result is unique.
Composition: which rule?
Four expressions, four colour–shape combinations. A holistic lookup memorises the expressions it was trained on. A compositional program filters by colour, then shape, and executes any of the four. Tick the training examples and run the comparison.
| Expression | Training | Holistic lookup | Compositional program |
|---|
The program is given grounded primitives (red, cube); the lookup is not. The comparison isolates the combination rule, not perception or learning.
Training examples
Checked expressions are memorised by the lookup. The program ignores this list: it composes the same two predicates every time.
What this shows
- A sentence can be satisfied by different objects under different frames; the frame is an argument the sentence leaves implicit.
- An undeclared frame has a set of referents, not a best referent. Returning the set is a decision the interface can make visible.
- A holistic lookup fails on a held-out recombination by having no representation for the question, not by answering it incorrectly.
- A program with grounded primitives applies the same colour-then-shape rule to every combination; the rule, not the example list, carries the generalisation.
What it does not
- It does not show that the program’s primitives are learned, correct, or grounded in perception. They are given.
- It does not measure compositional generalisation on natural language or real images; it isolates one rule in a toy grid.
- It does not prove that the compositional program is better on every task. It shows what each interface can represent.
- The lookup is a lookup, not a trained classifier. Its failure is a limit of its representation, stated plainly rather than dressed as a benchmark score.
Back to grounding · Back to compositionality · Lecture slides