Course / Act II · The two worlds Lecture 05

The interface problems grounding · compositionality · data · trust

A model can be right about the words and wrong about the world. The four interface problems are the places where that happens — and where a design decision, not a slogan, decides the outcome.

~45 min read 4 interface problems 1 interactive lab Updated 2026-11-16

Put a red cube on the left of a red sphere, put another red cube above the sphere, and ask for “the red cube left of the sphere.” From where you are sitting, one cube is the answer. From where the sphere is facing, the other cube is the answer. Nothing in the sentence changed. Nothing in the scene changed. The meaning did.

00 / A sentence, two worldsThe missing argument is the frame.

The sentence is a referring expression: a determiner, a colour, a shape, a relation, a reference object. It looks complete. It is not. Left of is not a two-place relation until someone supplies a frame of reference — the viewer’s frame, the reference object’s intrinsic frame, or a world frame fixed by gravity or a map. English leaves the frame implicit because speakers share a body, a viewing direction, and a pile of conventions. A machine shares none of that unless we give it to the machine.

The same scene, two frames of reference, two referents Two panels show the same arrangement: a red sphere facing right, a red cube A at the sphere's left, and a red cube B above the sphere. In the viewer frame, left means smaller horizontal coordinate, so cube A is the referent. In the object frame, left is the sphere's intrinsic left, which points up because the sphere faces right, so cube B is the referent. VIEWER FRAME OBJECT FRAME A · referent B sphere left = smaller x A B · referent sphere faces → left = sphere’s left
Fig. 0.1 Schematic. The same objects, the same sentence, two admissible semantics. In the viewer frame, left is a screen coordinate; in the object frame, it is a body-relative direction.

This is the smallest version of the first interface problem, and it is not a trick of English. The same structure appears whenever a symbol has to reach the world. Grounding asks which object a symbol picks out. Compositionality asks how the parts determine the whole. Data asks which examples, labels, and splits taught the mapping. Trust asks what anyone is entitled to conclude from the output. The four are not stages in a pipeline; they are four ways the same interface can be underspecified.

The history is a sequence of attempts to make one of these arguments explicit. Winograd’s SHRDLU (1971–72) manipulated a blocks world from English, with the world and the lexicon built by hand. Harnad’s symbol grounding problem (1990) asked how a symbol could mean anything without that hand-built dictionary. Fodor and Pylyshyn (1988) argued that connectionist systems lack the systematicity composition requires. CLEVR (Johnson et al., arXiv:1612.06890) made composition controlled; SCAN (Lake and Baroni, arXiv:1711.00350) and COGS (Kim and Linzen, arXiv:2010.05465) made the held-out combination the unit of evaluation. CLIP (Radford et al., arXiv:2103.00020) made caption alignment a scalable grounding strategy; SayCan (Ahn et al., arXiv:2204.01691) tested plans against affordances; the Embodied Agent Interface (Li et al., arXiv:2410.07166) separated planning from grounding errors. The vocabulary changes; the four questions do not.

A symbol is a promise about the world. An interface is the set of conditions under which the promise can be checked.
Caveat · what the hook does not show

Two referents in a toy scene do not prove that a real system is ambiguous; they show that the sentence alone does not determine an answer. Real ambiguity is worse: the reference object may be misdetected, the relation may be graded, and the speaker may use a frame the system has never seen.

01 / GroundingWhich object does the symbol pick out?

Harnad’s argument is circularity. Look up horse and you find equine mammal; look up equine and you find horse. The definitions never leave the symbol system. Something outside the dictionary has to connect the word to a class of things in the world: perception, action, or a convention that is itself grounded in social practice.

Definition · grounding and anchoring

Grounding is the relation between a symbol and what it denotes — an object, a property, a relation, or an action — established through perception, action, or convention. Anchoring is the narrower act of binding a symbol occurrence to one entity in one scene: this token of red cube to object 4 at time t. A system can have grounded predicates and still anchor them wrongly.

Make the semantics explicit. For a scene S with objects Obj(S), a unary predicate denotes a subset:

⟦red⟧S = { o ∈ Obj(S) : o has colour red }
(1.1)

A binary relation denotes a set of pairs, and the frame is part of the denotation, not an afterthought:

⟦left_of⟧S,F = { (a,b) : a,b ∈ Obj(S) and leftF(a,b) }
(1.2)

For the viewer frame, with image coordinates increasing right and down, leftV(a,b) is xa < xb. For the object frame, let fb be the unit vector the reference object faces. Its intrinsic left is lb = (fb,y, −fb,x) — a quarter turn, so an object facing east has its left to the north. Then:

leftO(a,b) ⇔ (pa − pb) · lb > 0
(1.3)

In Fig. 0.1 the sphere sits at (3,2) and faces east, so l = (0,−1). Cube A at (1,2) gives a dot product of zero: it is beside the sphere, not to its intrinsic left. Cube B at (3,1) gives (0,−1)·(0,−1) = 1: it is to the sphere’s left. The viewer frame gives the opposite answer. Both are correct relative to their frame; the sentence alone does not choose.

From pixels to symbols

A grounding pipeline has more places to fail than the word suggests. A detector proposes regions; an attribute classifier assigns colour, shape, size, and material; a relation module assigns spatial predicates; an anchorer binds the symbols to object identities; a parser maps the expression onto those symbols. Each stage is a place where a symbol can attach to the wrong thing, or to nothing.

A grounding pipeline and its failure points Five boxes in a row: pixels, detections, attributes, relations and scene graph, symbols and program. Arrows connect them. Below the arrows, labels name the failure at each stage: missed or merged objects, confused attributes, wrong relations, wrong anchor or frame. pixels image or video detections boxes, masks attributes colour, shape, size scene graph objects + relations symbols anchors + program missed or merged confused attribute wrong relation wrong anchor or frame Each layer is a symbol-to-world commitment. A downstream program cannot recover an object that the detector never proposed. The scene graph is the shared object both camps can inspect; it is also where perceptual errors become symbolic facts.
Fig. 1.1 Schematic of a grounding pipeline. The orange stages are symbolic; the teal stages are perceptual. Every arrow is a place where an error can be introduced and then hidden by a fluent downstream component.

The scene graph is the most useful intermediate representation here, and the most dangerous. It is useful because it turns an image into objects, attributes, and relations a program can query. It is dangerous because it looks like facts. A detector’s confidence, a confusion between red and orange, and a relation module’s depth assumption all become edges that the symbolic component treats as true. NS-CL (Mao et al., arXiv:1904.12584) makes the scene representation the interface between perception and program execution, which is why its errors are inspectable — and why it inherits the limits of that representation.

Ambiguity is a result, not an error

If an expression has two referents, the correct output is not the more likely one. It is the set, plus a statement that the expression is ambiguous under the current frame. A system that returns argmax has made a choice the language did not authorize. The lab makes this concrete: with the frame undeclared, one expression selects two objects, and the interface says so.

Table 1.1 · Six grounding probes. Each probe changes one thing and asks whether the referent changes for the right reason.
ProbeWhat you changeWhat a pass looks likeWhat it catches
Frame swapViewer ↔ object frameThe referent changes only when the expression is frame-relativeA hard-coded screen-space left
Distractor insertionAdd a same-colour or same-shape objectAmbiguity or a changed set, not a silent flipArgmax reference resolution
Attribute counterfactualChange one object’s colour or shapeThe referent changes only if the expression mentions itShortcut use of an irrelevant cue
Paraphrase“left of” → “to the left of”The same referent set, or a stated parse failureSurface-pattern matching
OcclusionHide part of the reference objectConfidence drops or the system abstainsOverconfident perception
Anchor persistenceMove the scene between framesThe anchor follows the object, not the pixel regionIdentity confusion across time
Caveat · scene graphs are not grounding in Harnad’s sense

A scene graph is a symbol-to-sensor mapping. That is necessary but not sufficient for the stronger claim that a symbol has a causal role in behaviour, or means what a human means by it. The stronger claim needs action, feedback, and convention.

02 / CompositionalityUnseen combinations are the test.

The principle of compositionality, usually attributed to Frege, says that the meaning of a complex expression is a function of the meanings of its parts and the way they are combined. For a learning system, that is a constraint on the function to be learned: the same part must mean the same thing in a new context, and the combination rule must apply to parts it has not seen together.

⟦S⟧ = fsyn(⟦w1⟧, …, ⟦wn⟧)
(2.1)

Fodor and Pylyshyn (1988) turned the principle into an argument about architectures. A system that can represent red cube and blue sphere has the constituents red, blue, cube, sphere, and the rules that combine them. If it cannot represent red sphere and blue cube without new training, the representation is not compositional in their sense; it is a lookup over whole structures. They called the required property systematicity: the ability to entertain novel combinations of familiar constituents.

A program makes the composition rule explicit. For an object set S and a predicate c:

⟦filter(c)⟧(S) = { o ∈ S : c(o) }    and    ⟦filter(c2) ∘ filter(c1)⟧(S) = ⟦filter(c2)⟧(⟦filter(c1)⟧(S))
(2.2)

The program is compositional by construction: filter(cube) after filter(red) is a function composition, and it applies to any object set, including one it has never seen. The neural network that supplies red and cube need not be compositional. The interface makes the whole system systematically generalise — if the primitives are right and the parser produces the right program.

Interpolation is not recombination

Most benchmark scores measure interpolation: the test set is drawn from the same distribution, and a new example is a new point near old points. Compositional generalisation asks a different question. Hold out a combination of constituents that each appear in training with other partners, and ask the system to handle it. The held-out point is not near the training points in the space of combinations, even if it is near them in the space of pixels or tokens.

A 2 by 2 recombination test for colour and shape A two by two grid of colour and shape combinations. Red cube and blue sphere are seen in training. Red sphere and blue cube are held out. A holistic lookup has entries only for the seen cells; a compositional program filters by colour and then by shape, so it handles all four. TRAINING COMBINATIONS cube sphere red blue seen held out held out seen TWO MODELS ON THE TEST CELLS holistic lookup red cube → 1 blue sphere → 1 red sphere → — blue cube → — no entry for the recombination program filter(colour) → filter(shape) red cube → 1 blue sphere → 1 red sphere → 1 · blue cube → 1 same rule, four inputs The lookup is not wrong about the cells it saw. It has no representation for the question in the other two. The program is given grounded primitives; the comparison isolates the combination rule, not perception or learning. That is the interface difference the lab makes visible.
Fig. 2.1 A 2×2 recombination test. The diagonal is seen; the off-diagonal is held out. The lookup has no entry for the off-diagonal; the program applies the same rule to all four cells.

What the benchmarks actually test

SCAN (Lake and Baroni, 2018) showed that sequence-to-sequence models trained on command–action pairs can fail on commands that combine a known primitive with a known modifier in a new way. COGS (Kim and Linzen, 2020) sharpened the question by testing semantic interpretation on novel syntactic structures, not only novel lexical combinations. Both made a methodological point that outlives their grammars: if the test set is an i.i.d. sample, a high score says almost nothing about composition.

Faith and Fate (Dziri et al., arXiv:2305.18654) adds that on multi-step compositional tasks, transformer performance tends to degrade as the reasoning chain grows, consistent with pattern completion over training-like structures rather than execution of a general procedure. The claim is about a family of tasks and models, not a proof that no network can compose; treat it as a diagnostic signal, not a theorem.

The constructive counterexample is NS-CL. It learns a scene representation and a question parser from question–answer pairs, then executes the parsed program over the scene. On CLEVR’s compositional grammar, that division supports recombination beyond the exact questions seen in training. The caveat is equally important: the scene is synthetic, the grammar is closed, and the concepts are a small finite set.

Table 2.1 · Five splits, five different questions. “Held out” means absent from training, not merely rare.
SplitWhat is held outQuestion it asksWhat a high score does not show
I.i.d.A random sample of the same distributionCan the model interpolate?Anything about recombination
Atom held outA colour, shape, or wordCan it handle a new primitive?That the combination rule transfers
Combination held outA pair of seen primitives, never seen togetherCan it recombine?That the parse is faithful or the primitives are right
Length extrapolationLonger chains than any training exampleDoes the procedure scale?That it is not pattern-completing on shorter cases
Template held outA syntactic frame with new fillersDoes structure drive the answer?That the vocabulary is grounded
Caveat · a compositional score is not a compositional mechanism

A model can pass a held-out-combination test by learning a shortcut that happens to cover the test cells: a colour bias, a position cue, a question template. The stronger evidence is a trace — a parse, a program, an execution — that can be perturbed independently of the answer.

03 / DataWhat supervision buys, and what it hides.

Every interface problem has a data shadow. A grounded predicate needs examples of the property under the conditions the system will meet. A compositional parser needs evidence about the combination rule, not only the constituents. A trust claim needs an evaluation set that can falsify it. The question is not “how much data?” but “which distinctions does this data make identifiable?”

Table 3.1 · Supervision regimes. The costs are qualitative; the failure modes are the ones that matter here.
RegimeWhat is labelledWhat it teaches wellCharacteristic failure
Program annotationThe executable program for each inputComposition and parse structureExpensive; biases the parser toward the annotated grammar
Execution traceIntermediate statesFaithfulness of each stepTraces can be plausible without being causal
Question–answer pairsOnly the final answerEnd-to-end behaviour at scaleMany programs fit the same answers; the parse is underdetermined
Image–caption pairsFree-form natural languageBroad, cheap alignmentCo-occurrence is not reference; captions omit and imply
Weak supervisionNoisy or aggregate labelsCoverage when clean labels are scarceLabel noise is correlated with the shortcut the model will learn
Active learningLabels for selected inputsEfficiency under a label budgetSelection follows uncertainty, which may be miscalibrated

The case for symbolic structure as a prior is a case about the hypothesis space. Let H be the class of programs or predictors, and K a knowledge base the system treats as true. The knowledge restricts the class:

HK = { h ∈ H : h is consistent with K }    so    |HK| ≤ |H|
(3.1)

If K is true, a smaller hypothesis class can be learned from fewer examples. If K is false, the restriction removes the right answer and no amount of data recovers it. Knowledge is a prior, and a prior is a bet: worth making when the domain is stable and the constraints are auditable, a liability when the world is open or the knowledge is stale.

Programs versus answers

Question–answer supervision is cheap, and dangerous because the mapping from a question to an answer is many-to-one over programs. “How many red cubes are left of the sphere?” can be answered by filtering red, then cubes, then the relation; by computing the relation first; or by a lookup that associates the template with a number. Only some of those programs generalise to a new scene. NS-CL’s training procedure matters because it learns a parser from answers under a compositional curriculum, rather than assuming the answer identifies the program. The curriculum does work the labels do not.

Program annotations invert the trade. They identify the intended computation, which makes composition learnable from fewer examples and the trace checkable. They also import the annotator’s grammar: if the annotation language cannot express a valid parse, the parser will never learn it. The data interface is never neutral; it specifies what counts as a correct intermediate step.

Coverage, leakage, provenance

A dataset for a compositional system should be described by its coverage, not only its size. Build the matrix of attribute combinations the interface can express, mark which cells appear in training, and check that every cell you intend to test is absent from training by construction. The matrix in Fig. 2.1 is the smallest version. At real scale it has more axes — object types, relations, templates, lighting, viewpoints — and the empty cells decide whether a “compositional” claim is testable.

Benchmark leakage makes a coverage matrix lie. It takes several forms: exact or near-duplicate inputs across splits, template overlap that lets a model memorise the question shape, scene-level leakage where the same image appears with different questions, and, for LLMs, pretraining contamination that cannot be audited from the paper. None is visible in an aggregate accuracy number. A leakage audit hashes inputs, searches for near-duplicates, splits by scene or grammar rather than by row, and reports what could not be checked.

Multimodal pretraining is the strongest available answer to the data problem and the weakest to the grounding problem. CLIP learns a shared embedding from natural-language supervision over images; the alignment is broad, cheap, and useful for zero-shot recognition. But an image–caption pair asserts that a caption is associated with an image, not that a phrase refers to a region. That is enough for retrieval and not enough for “the red cube left of the sphere” when the frame is undeclared.

Caveat · data cannot make an unrepresented variable identifiable

If the frame is not in the input and not in the label, no dataset can teach the system to choose it. More data will fit the training frames more accurately and will not distinguish “left means screen-left” from “left means the object’s left.” The fix is representational before it is statistical.

04 / TrustWhat is a guarantee a guarantee of?

Trust is not a property a model has in isolation. It is a relation between a claim, a specification, an evaluation, and a person or institution entitled to rely on the claim. A system can be accurate on a benchmark and untrustworthy in a hospital; a system can be formally verified and useless because the specification omitted the hazard that matters. The engineering question is not “is it trustworthy?” but “which claim, relative to which specification, on which domain, with what evidence?” This is a position, not a theorem, and it is the position this lecture adopts.

Interpretable, or explained afterwards

Rudin (arXiv:1811.10154) argues that for high-stakes decisions one should use models interpretable by construction, rather than explaining a black box afterwards. Part of the argument is fidelity: a post-hoc explanation is a second model fitted to the first, and it can be wrong about the first model’s computation. LIME (Ribeiro et al., arXiv:1602.04938) is the canonical local-approximation method: it samples perturbed inputs around a prediction, fits a simple surrogate, and reports its important features. It is a useful diagnostic and a weak guarantee — the surrogate’s fidelity is local, the perturbation distribution is a choice, and the explanation can be stable while the decision is not.

The neurosymbolic version is concrete. A program trace can be interpretable: you can read the filters, the relation, and the count. But an interpretable trace can be wrong — the parser may have chosen the wrong attachment, or a predicate may be grounded on the wrong object — and a fluent explanation can make a wrong trace persuasive. Explainability is a communication layer on top of a representation, and it inherits whatever that representation hides.

Specification is the hidden premise

A verification claim has the form “system S satisfies specification φ on domain D.” The specification is not a comment on the system; it is part of the theorem. If φ says “the robot never collides with a visible obstacle,” a transparent obstacle may be outside the specification and inside the accident. If φ says “the answer is one of the objects in the scene,” a system that returns the right count for the wrong reason still satisfies it. Verification is a proof about a model; the model-to-world gap is not verified by the same proof.

Definition · three kinds of guarantee

Empirical: the system was tested on a sample; the claim is statistical, about that sample and its distribution. Formal: the system satisfies a specification for all inputs in a stated domain, relative to a model. Operational: a monitor checks the running system and can intervene. Each is valuable; none implies the others.

Confidence is not correctness

A classifier’s softmax score is not a probability that the answer is correct unless it has been calibrated against outcomes. Calibration is a property of the score distribution: among all cases where the system says 0.8, roughly 80% should be correct. The standard summary is expected calibration error, which bins predictions by confidence and averages the gap between confidence and accuracy:

ECE = Σb (nb / N) · |acc(b) − conf(b)|
(4.1)

The formula is simple; the interpretation is not. A low ECE on an i.i.d. test set does not imply calibration under distribution shift, and a system can be well calibrated on average while being badly calibrated where it is used. Uncertainty conflates noise in the input, ambiguity in the label, and ignorance about the domain. A scene with two valid referents is not “uncertain” like a blurry image; it is underspecified. Returning a low-confidence argmax conflates the two.

Underspecification (D’Amour et al., arXiv:2011.03395) adds a structural problem. Many models can fit the same training distribution and agree on the test set while relying on different features; they diverge under a shift the evaluation never exercised. A single accuracy number hides that disagreement. For a neurosymbolic system, the analogue is a set of programs that all fit the answers but compute different things; the trace is the only way to see which one you got.

Robustness and shortcuts

Shortcut learning (Geirhos et al., arXiv:2004.07780) is the tendency to use a cue that correlates with the label in training but does not express the intended rule: background texture, object position, a spurious word in the question. The shortcut is not an optimizer bug; it is what empirical risk minimisation does when a cheaper cue is available. Compositional and grounding tasks are especially vulnerable because the intended rule is relational and the shortcut is often perceptual. A model can answer “left of the sphere” by looking at the cube’s absolute position, which works until the camera moves.

SayCan is a useful example of trust by construction rather than trust by explanation. A language model scores candidate skills; learned value functions estimate whether each skill is feasible in the current state; the product chooses the next action. The language model supplies the plan, the affordances supply the grounding, and the value functions check executability. The guarantee is still empirical and local to the skill library, but the interface makes the failure modes legible: a wrong skill score, a missing affordance, or a grammatical but infeasible plan. The Embodied Agent Interface goes further and separates planning errors from grounding errors in a common taxonomy — the diagnostic move this lecture is building toward.

Caveat · trust is not a model property

No architecture is trustworthy in the abstract. A proof is trustworthy relative to a specification and a model; a benchmark is informative relative to a distribution and a split; an explanation is useful relative to a user and a decision. State the relation, not the adjective.

05 / The diagnostic frameFour failures, four probes.

When a neurosymbolic system gives a wrong answer, the useful question is not “which component is bad?” but “which interface assumption failed?” Four failure types cover most cases, and each has a characteristic probe. The categories are diagnostic aids, not an ontology: one wrong answer can have more than one cause.

Table 5.1 · The four interface failures. The probe is the smallest experiment that distinguishes this failure from its neighbours.
FailureSymptomFirst probeWhat to logDesign response
GroundingWrong object, attribute, or relationOracle perception; if the answer becomes right, the error is upstreamCandidate referents; frame; anchor ids; confidencesDeclare the frame; return referent sets; add ambiguity checks
CompositionRight objects, wrong order, scope, or numberCompare the parsed program with the intended one; perturb one operatorParse tree; program; trace; intermediate countsTyped executor; program supervision; held-out combination tests
SpecificationThe system does what it was told; the instruction was wrongHave a second reader formalise the requirement independently; diff the twoSpec version; assumptions; out-of-spec conditions; overridesSpec review; monitor assumptions; make the spec executable
Distribution shiftPerformance drops when scene, language, or environment changesRe-run the same evaluation on the shifted domain, per sliceDomain id; input statistics; per-slice calibration; OOD scoreShift-aware evaluation; abstention; adaptation; coverage

A worked diagnosis

An instruction: “Put the red cube left of the bowl into the blue bin.” The robot picks up a red cube, carries it to the bin, and drops it beside the bin rather than inside. Four readings:

  • Grounding. There were two red cubes. The system chose the one that was left in the viewer frame; the speaker meant the one on the bowl’s left. The trace shows a single referent with high confidence — the tell that an ambiguous expression was resolved by argmax.
  • Composition. “Left of the bowl” attaches to “the red cube,” but the parser attached it to “put.” The objects are right; the program’s scope is wrong. The parse tree, not the perception, is the evidence.
  • Specification. “Into the bin” was formalised as “within a fixed radius of the bin,” because the planner’s containment predicate was coarse. The robot did what the specification said. The specification is the bug.
  • Distribution shift. The bin is transparent. The detector was trained on opaque bins and treated the transparent one as absent. The scene is in-distribution by category and out-of-distribution by material.

The four readings imply four fixes: a frame declaration and ambiguity check; a parser change or a clarification question; a specification review; a data or detector change. Retraining the policy would be the wrong response to the first three. That is the practical value of the frame: it stops a data problem from being treated as a reasoning problem, and a specification problem from being treated as a perception problem.

A diagnostic order for a wrong answer Five steps in order: reproduce the failure, swap in oracle perception, check the frame and the parse, review the specification, then test the shifted domain before retraining. An arrow loops back from the last step to the first. 01 reproduce same input, same trace 02 oracle swap perception vs reasoning 03 frame + parse referent set, program 04 spec review what was asked? 05 shift test then retrain if the fix changes the failure, the diagnosis was wrong The order is a heuristic, not a proof. Its point is to make the cheapest discriminating experiment come first.
Fig. 5.1 A diagnostic order. The loop is deliberate: a diagnosis that does not change the failure is a hypothesis, not a finding. The oracle swap separates perception from everything downstream.

What to log

A system that cannot produce a trace cannot be diagnosed, but a trace is not automatically trustworthy. The useful log records the interface decisions, not only input and output: input id and hash; the declared frame; candidate referents with scores and anchor ids; the parsed program; the execution trace with intermediate sets and counts; constraint and type checks; the calibration score; model, data, and specification versions; and any human override. The schema makes the difference between “the model was unsure” and “the question was underspecified” visible after the fact.

Two evaluation designs do most of the work. The oracle ablation replaces one component with a ground-truth version and asks whether the error survives. If true perception fixes the answer, the failure is grounding; if not, it is downstream. The counterfactual edit changes one property of the scene or sentence and checks that the output changes for the right reason. A system that is right for the wrong reason passes an accuracy test and fails a counterfactual test.

Three instruments

Read each of these as a probe, not as a leaderboard.

06 / Design patternsWhat to build when the problem is named.

Naming a problem is not a design. The design work is to decide which interface carries the response, what it costs, and what failure it introduces. The checklist is the set of questions a team should answer before it writes a paper or ships a system.

Checklist · six questions for any interface

1 · What is the symbol? A predicate, program, graph, constraint, proof. 2 · What grounds it? A detector, scene graph, lexicon, convention. 3 · What composes it? A parser, executor, solver — and is it typed? 4 · What data teaches it? Which regime, coverage, split? 5 · What checks it? A loss, monitor, verifier, calibration test — of what? 6 · What happens when it fails? Abstain, ask, fall back, log, alarm. A system with no failure behaviour has an implicit one: it guesses.

Table 6.1 · Four problems, the interface that most directly addresses each, and the cost of choosing it.
ProblemPrimary responseMechanismCost and new failure
GroundingArchitecture + dataScene graph, neural predicates, explicit frame, anchor idsPerception errors become symbolic facts; the frame must be maintained
CompositionArchitecture + searchParse to a typed program; execute or synthesise over a structured sceneParser cost; a wrong parse is a confident wrong answer
DataData interfaceCoverage matrix, compositional splits, provenance, active samplingAnnotation cost; a dataset can encode the annotator’s grammar
TrustVerifier + evaluationSpecification, monitor, calibration, counterfactual tests, logsSpec maintenance; monitor coverage; false alarms; runtime

The table is a default, not a rule. A grounding problem can sometimes be fixed with data; a composition problem with a loss; a trust problem by removing the model and using a rule. The right question is not “which interface is best?” but “which interface makes the failure detectable at the cost I can pay?”

A worked design: the tabletop referrer

  1. Grounding. Build a scene graph with object ids, attributes, and pairwise relations. Declare the frame in the interface: the caller says viewer, object, or unspecified. If unspecified, return the union of referents and an ambiguity flag. Never silently argmax.
  2. Composition. Parse the expression into a typed program: filter(colour), filter(shape), relate(left_of, ref), count, select. Type-check before execution. The trace is the explanation.
  3. Data. Train attribute classifiers on a coverage matrix: every colour × shape × lighting cell the deployment can meet. Hold out combinations for evaluation. Record the matrix in the model card.
  4. Trust. Calibrate attribute scores on a held-out set; abstain when the referent set is empty or plural; log the frame, candidates, program, and trace. For an irreversible action, require human confirmation. The monitor checks reachability and collision, not “correctness.”

The design is not impressive, and that is the point. Each response is an interface decision that makes a specific failure visible: a frame parameter makes ambiguity visible; a typed program makes scope errors visible; a coverage matrix makes dataset gaps visible; a trace and a monitor make accountability possible.

Anti-patterns

  • Argmax over ambiguity. One referent is returned when the expression has two; the confidence score hides the choice.
  • Explanation as trust. A fluent trace is offered as evidence that the answer is right. The trace can be wrong at the first predicate.
  • Benchmark as capability. An i.i.d. score is reported as evidence of composition or grounding. The split, not the score, is the claim.
  • Symbolic wrapper, no semantics. Rules filter outputs without constraining the computation.
  • Unlogged interface. The frame, parse, and candidate set are not recorded; the first failure is unreproducible.
Caveat · every interface adds a failure mode

A frame can be set wrong; a typed parser can reject a valid expression; a monitor can be blind to a hazard outside its specification; a coverage matrix can be incomplete. The checklist does not remove failure; it moves failure to a place where it can be seen.

07 / The five interfacesFour problems, five responses, no one-to-one map.

The five interfaces from Lecture 01 are not five solutions to five problems. They are five places to put the coupling, and each is better at some problems than others. This table is the bridge to Lecture 06.

Table 7.1 · The five interfaces as responses to the four problems. “Primary” means most direct, not sufficient.
InterfacePrimary problemHow it respondsCanonical systemsWhat it does not fix
Logic as lossData, compositionA constraint becomes a differentiable penalty; training satisfies it rather than filtering afterwardsSemantic loss, LTN, DL2Soft constraints can be traded away; a loss is not a proof
Logic as architectureGrounding, compositionNeural predicates feed a symbolic executor inside the computation graphDeepProbLog, NeurASP, NS-CL, ScallopInference cost; a wrong scene graph is a wrong proof
Logic as dataData, groundingSymbols structure the input, output, or embeddingKG embeddings, ontology embeddings, rule miningEmbedding distortion; schema drift
Logic as searchComposition, trustNeural proposals plus symbolic search or synthesis; the result is checkableILP, program synthesis, AlphaGeometrySearch cost; proposal quality; a proof of the wrong statement
Logic as verifierTrustA monitor, shield, or repair layer checks output against a specificationRuntime monitoring, STL, conformal boundsModel fidelity; monitor coverage; specification gaps

Grounding is most directly an architecture problem because the interface must carry a reference from symbol to entity; a loss can encourage consistency but cannot create a referent. Composition is an architecture or search problem because the rule must be executed or synthesised. Data is a data-interface problem because the split determines what can be learned and claimed. Trust is a verifier problem because a guarantee needs a specification and a check, not a better representation.

This explains why a system can be strong on one problem and weak on another. DeepProbLog has an exact probabilistic semantics and a clean architecture; it still inherits whatever its neural predicates get wrong. AlphaGeometry produces formal proofs; the proof is only as good as the formalisation, and the proposals can be wrong without making the final proof invalid. SayCan grounds language in affordances; it does not guarantee an optimal plan. The tuple, not the label, is the object to compare.

The lecture’s position · stated as a position

The four interface problems are not a taxonomy of the field and not a claim that every system must solve all four. They are a diagnostic frame: a way to ask what a system has made explicit and what it has left to the world. The claim defended here is narrower and testable — that most failures of neurosymbolic systems in deployment are failures of an interface assumption, not of the neural or symbolic component in isolation.

Lecture 06 compares systems along expressivity, complexity, coupling, guarantee, supervision, cost, data efficiency, and transparency. Those axes are the four problems made comparable. When a paper claims “compositional generalisation” or “trustworthy grounding,” ask: which interface made it true, on which split, against which specification, and at what cost?

08 / RecapWhat you should be able to do now.

01

Explain why a referring expression can have two correct referents, and name the missing argument: the frame.

02

Define grounding and anchoring, and distinguish a scene graph from grounding in Harnad’s stronger sense.

03

State the compositionality principle and test it with a held-out recombination, not an i.i.d. sample.

04

Compare supervision regimes by what they identify, not by how much data they supply.

05

Distinguish empirical, formal, and operational guarantees, and state the specification each hides.

06

Diagnose a wrong answer as grounding, composition, specification, or distribution shift, and name the probe.

07

Apply the six-question interface checklist to a small system and name the cost of each response.

08

Map the four problems onto the five interfaces, and say what each interface does not fix.

Exercises

  1. Two frames, one sentence. Draw a three-object scene in which “the red cube left of the sphere” has a different referent in the viewer and object frames. Specify coordinates and facing, and compute both answers with Eq. 1.3.
  2. Design a split. Take a grammar with three colours and three shapes. Write an i.i.d., an atom-held-out, and a combination-held-out split. Which cells are in each test set, and what does each split let you claim?
  3. Audit a benchmark. Pick a paper from the reading list that reports a compositional result. Find its split definition. Does it hold out a combination, a template, or only a random sample? If the paper does not say, write the question you would ask.
  4. Break the lab. Set the frame to “undeclared” and enter “the red cube left of the red sphere.” Switch to viewer and to object. Explain the change in one sentence. Then try an expression with no relation and explain its ambiguity.
  5. Write a specification. For the tabletop referrer, write three specifications: safety, task success, ambiguity handling. For each, say what it omits and what evidence would check it.
  6. Diagnose a failure. A robot told to “put the red cube left of the bowl into the blue bin” places the cube beside the bin. Give two diagnoses from §5, the probe that distinguishes them, and the log field that would settle it.

Check your understanding

Ten questions, with explanations. The score is not the point; the questions you cannot answer are the reading list.

Reading list

The first six are the spine of this lecture. The rest are the next layer: benchmarks, critiques, and systems that make the interface concrete.

Harnad, “The Symbol Grounding Problem”

Physica D 42, 335–346, 1990 · the dictionary argument

Grounding

Fodor & Pylyshyn, “Connectionism and cognitive architecture”

Cognition 28(1–2), 3–71, 1988 · systematicity

Composition

Generalization without Systematicity

Lake & Baroni · 2018 · SCAN, the held-out command split

Composition

COGS: A Compositional Generalization Challenge

Kim & Linzen · EMNLP 2020 · structural splits

Composition

The Neuro-Symbolic Concept Learner

Mao et al. · ICLR 2019 · scene graph, program, answer

Architecture

CLEVR: A Diagnostic Dataset for Compositional Reasoning

Johnson et al. · CVPR 2017 · the controlled benchmark

Benchmark

Faith and Fate: Limits of Transformers on Compositionality

Dziri et al. · NeurIPS 2023 · depth and pattern completion

Limits

Underspecification Presents Challenges for Credibility

D’Amour et al. · 2020 · many models, one score

Trust

Stop Explaining Black Box Models for High Stakes Decisions

Rudin · 2019 · interpretability as a design requirement

Trust

“Why Should I Trust You?”: Explaining Any Classifier

Ribeiro et al. · KDD 2016 · LIME, the local surrogate

Trust

Shortcut Learning in Deep Neural Networks

Geirhos et al. · Nature Machine Intelligence 2020 · the cheap cue

Robustness

Learning Transferable Visual Models From Natural Language

Radford et al. · 2021 · CLIP and caption grounding

Grounding

Do As I Can, Not As I Say

Ahn et al. · 2022 · SayCan, language grounded in affordances

Robotics

Embodied Agent Interface

Li et al. · 2024 · planning versus grounding errors

Diagnosis
Source note · what is redrawn and what is claimed

Figures 0.1, 1.1, 2.1, and 5.1 are schematics drawn for this lecture, not reproductions. The gallery diagrams are schematic redrawings. Citations identify systems and claims; no benchmark number is quoted, and no arXiv ID is used for a paper that does not have one. Harnad (1990) and Fodor and Pylyshyn (1988) are cited by author, title, venue, and year.