Course / Lecture 05 / Lab Lab 05

Grounding and composition, made visible.

One grid, two questions. Which object does the expression pick out — and does the rule that picks it out survive a combination it has never seen? Change the frame, then change the training set.

6 × 5 grid world 3 frames 4 combinations No install · runs in the browser

The grounding and composition lab

How to use it. Resolve an expression under a declared frame, then under “undeclared,” and compare the referent sets. In the second panel, choose the examples the lookup is trained on and run the comparison.

01

Grounding: which object?

The grid has six objects. The expression names a colour, a shape, and optionally a relation to a reference object. The frame decides what “left” means; the status says whether the result is unique.

Filled shape: red Dark shape: blue Dot: facing direction Dashed ring: reference
Frame of reference
Resolving the default expression with an undeclared frame.
02

Composition: which rule?

Four expressions, four colour–shape combinations. A holistic lookup memorises the expressions it was trained on. A compositional program filters by colour, then shape, and executes any of the four. Tick the training examples and run the comparison.

ExpressionTrainingHolistic lookupCompositional program

The program is given grounded primitives (red, cube); the lookup is not. The comparison isolates the combination rule, not perception or learning.

Training examples

Checked expressions are memorised by the lookup. The program ignores this list: it composes the same two predicates every time.

Two examples are checked: red cube and blue sphere.

What this shows

  • A sentence can be satisfied by different objects under different frames; the frame is an argument the sentence leaves implicit.
  • An undeclared frame has a set of referents, not a best referent. Returning the set is a decision the interface can make visible.
  • A holistic lookup fails on a held-out recombination by having no representation for the question, not by answering it incorrectly.
  • A program with grounded primitives applies the same colour-then-shape rule to every combination; the rule, not the example list, carries the generalisation.

What it does not

  • It does not show that the program’s primitives are learned, correct, or grounded in perception. They are given.
  • It does not measure compositional generalisation on natural language or real images; it isolates one rule in a toy grid.
  • It does not prove that the compositional program is better on every task. It shows what each interface can represent.
  • The lookup is a lookup, not a trained classifier. Its failure is a limit of its representation, stated plainly rather than dressed as a benchmark score.

Back to grounding · Back to compositionality · Lecture slides