Lecture 21 · Frontiers

Benchmarks, evaluation, and how to read a NeSy paper.

A score is not a measurement of intelligence. It is a compressed argument. This lecture teaches you to unpack it — and to notice when a premise is missing.

6 benchmarks 5 questions 1 protocol Self-paced
00 · Learning objectivesLecture 21 · Benchmarks and evaluation

By the end of this lecture

Four things you can do.

01

Read a benchmark as a measurement

Say what it isolates, what it leaves out, and which claim it can support.

02

Audit splits and leakage

Name the held-out unit, the correlate, and the overlap that could inflate a score.

03

Tell baselines from ablations

Answer “better than what?” and “because of what?” with the right comparison.

04

Read a paper with a protocol

Apply the five questions, state the guarantee, and write a five-line review.

S/V · Lecture 21
00 · The problemLecture 21 · Benchmarks and evaluation

Two systems, one number

The score does not tell you where the interface is.

System A · composes

Parses the question into a program and executes it over a scene graph. An error is traceable to a step.

System B · retrieves

Maps the question to an answer pattern learned from similar examples. An error is not localizable.

Same headline number. The number is a marginal; the interface is a conditional. Which claim does the number support?

A schematic, not a result.

S/V · Lecture 21
00 · The problemLecture 21 · Benchmarks and evaluation

A score is a compressed argument

On D, with M, against B.

(benchmark, split, metric, baseline, ablation, guarantee)

Construct

Does the benchmark measure the named capability?

Internal

Is the effect caused by the named component?

External

Does it survive a change of distribution or embodiment?

Statistical

Is the difference larger than seed and sampling noise?

One missing slot does not make the claim false. It makes it under-specified.
S/V · Lecture 21
01 · BenchmarksLecture 21 · Benchmarks and evaluation
01 / 08

What does each instrument measure?

Six benchmarks, six blind spots, and one rule: a benchmark can be diagnostic without being realistic.

S/V · Lecture 21
01 · BenchmarksLecture 21 · Benchmarks and evaluation

Six instruments

What each one diagnoses.

BenchmarkTaskMetricDiagnosesDoes not
CLEVRQuestions over scenesAccuracyBinding, relations, countingOpen vocabulary, real images
SCANCommands → actionsExact sequenceLexical and structural compositionNatural language, perception
COGSSentences → logical formsExact formCompositional semantic parsingOpen domain, world knowledge
PlanBenchPDDL planningValidity, costPlanning and reasoning about changePerception, execution
ALFWorldHousehold tasksSuccess rateGrounded, long-horizon actionReal perception, control
KGLink predictionMRR, Hits@kRelational generalizationTruth, novelty, time

A benchmark is a trade: control for realism, or realism for control.

S/V · Lecture 21
01 · BenchmarksLecture 21 · Benchmarks and evaluation

CLEVR · composition in a closed world

Diagnostic, not realistic.

Measures

Attribute binding, relations, counting, and composition over generated scenes with functional-program annotations.

Does not measure

Open vocabulary, real images, ambiguity, or knowledge outside the scene. CoGenT holds out shape–colour combinations.

A high CLEVR score is evidence about CLEVR. It is not evidence about a kitchen, a street, or a hospital.

Johnson et al., arXiv:1612.06890

S/V · Lecture 21
01 · BenchmarksLecture 21 · Benchmarks and evaluation

Composition in language

Near-ceiling in distribution, far from compositional.

SCAN

Commands → action sequences. Splits hold out a primitive, length, or template. RNNs “mix and match” when the gap is small; fail spectacularly when it is systematic.

Lake & Baroni · arXiv:1711.00350

COGS

Sentences → logical forms. 96–99% in-distribution; 16–35% on generalization; ±6–8% seed sensitivity.

Kim & Linzen · arXiv:2010.05465

The gap is the phenomenon. A benchmark can be near-solved in distribution and still diagnostic out of it.
S/V · Lecture 21
01 · BenchmarksLecture 21 · Benchmarks and evaluation

Action · PlanBench and ALFWorld

A valid plan is not an executed plan.

PlanBench

PDDL plan generation and verification; validity and cost. Separates planning from retrieving common-sense knowledge. LLM plan generation “falls quite short.”

arXiv:2206.10498

ALFWorld

Text and embodied household tasks aligned to ALFRED; success rate. Exploration or memorization can produce success.

arXiv:2010.03768

Validity ≠ optimality; success ≠ grounding. Both are outcome measures, not interface diagnoses.
S/V · Lecture 21
01 · BenchmarksLecture 21 · Benchmarks and evaluation

Knowledge graphs · relational generalization

A number about a protocol.

Link prediction. FB15k-237, WN18RR, YAGO3-10; MRR and Hits@k. Usually transductive: test entities were seen.
Inverse relations. FB15k made many test triples near-duplicates of training triples; FB15k-237 exists to remove that.
Protocol bias. Some very high completion scores are protocol artefacts, not better reasoning.

Toutanova & Chen 2015 · Sun et al., arXiv:1911.03903 · Hogan et al., arXiv:2003.02320

S/V · Lecture 21
01 · BenchmarksLecture 21 · Benchmarks and evaluation

The blind spots

What no benchmark measures for you.

Open-world perception

Closed vocabularies and generated scenes do not test the long tail.

The guarantee

Accuracy is not a promise about deployment.

Cost and latency

A score does not report the inference bill.

Interface errors

The average hides the slice where the seam breaks.

Ask which of these the claim needs, and which of them the benchmark supplies.

S/V · Lecture 21
02 · SplitsLecture 21 · Benchmarks and evaluation
02 / 08

The test distribution is a design choice.

Interpolation, recombination, and the held-out unit that tells you which one you are measuring.

S/V · Lecture 21
02 · SplitsLecture 21 · Benchmarks and evaluation

Two test geometries

Interpolation or recombination.

INTERPOLATION test inside the cloud no parts required RECOMBINATION known parts, new combination a memorizer fails here
Schematic. Real attribute spaces are not square grids.

Interpolation. The test point is near the training cloud; a smooth model can get it without the parts.

Recombination. Known parts, new combination; a memorizer fails, a compositional model can succeed.

The split chooses which. It is a hypothesis about what “compositional” means.

S/V · Lecture 21
02 · SplitsLecture 21 · Benchmarks and evaluation

Four families

Name the held-out unit.

FamilyHeld outTestsExample
LexicalA primitive or wordExtending the lexiconSCAN add-jump
StructuralA template or depthApplying a rule recursivelySCAN length; COGS structure
AttributeA combinationBinding without a co-occurrence cueCLEVR-CoGenT
Entity / timeNodes or periodsReasoning about the unseenInductive KG; temporal split
“Compositional” is not a split. It is a family of hypotheses about which part is being recombined.

A model can pass one family and fail another. Report which one you used.

S/V · Lecture 21
02 · SplitsLecture 21 · Benchmarks and evaluation

Three ways a split fails

Unit, correlate, balance.

Unit

Splitting by question while the same scenes are in train and test measures question generalization, not scene generalization.

Correlate

If colour predicts shape in training, a held-out shape–colour cell can still be solved by colour. Check conditionals.

Balance

Twenty held-out examples give a noisy estimate; a 98% train split is a different experiment. Report per-cell counts.

A split is a designed distribution, not a natural one. It can be a good test and still be a narrow one.
S/V · Lecture 21
03 · LeakageLecture 21 · Benchmarks and evaluation

The test set is only as clean as its construction

Six kinds of leakage.

KindHow it inflatesCheck
ExampleNear-duplicates across train and testDeduplicate at the claim’s unit
SplitHeld-out cell predictable from a correlateCheck conditional distributions
EntityTest entities seen; inverse relationsInductive split; filter inverses
PretrainingBenchmark items in the corpusCutoff; private or perturbed test
Test-set tuningLeaderboard iterationPre-register; private holdout
MetricPriors, templates, frequencyTrivial baselines; per-slice checks

A leaked score is a validity threat, not a number you can subtract.

S/V · Lecture 21
03 · LeakageLecture 21 · Benchmarks and evaluation

The large-model version

Memorization looks like reasoning.

Why it is hard

Public benchmarks are in public and proprietary corpora. Training data is undisclosed. A recalled answer and a derived answer look the same in the output.

What to do

Evaluate after a cutoff; keep a private or perturbed test set; report a contamination check; use a canary string; report all runs, not the best.

Contamination is a threat to validity. It is not a score you can subtract; it is a reason to weaken the claim.

Deng et al., arXiv:2311.09783

S/V · Lecture 21
04 · BaselinesLecture 21 · Benchmarks and evaluation
04 / 08

Better than what? Because of what?

A baseline is a comparison; an ablation is an attribution. Most papers need both, and most tables mix them.

S/V · Lecture 21
04 · BaselinesLecture 21 · Benchmarks and evaluation

“Better than what?”

Six baselines, six jobs.

BaselineQuestionHold fixedTypical failure
TrivialHarder than the prior?Split, metricStrong on an imbalanced set
Symbolic-onlyWhat do rules buy?Knowledge base, itemsRules tuned on test
Neural-onlyWhat does learning buy?Data, parameters, computeSmaller and trained less
Published SOTABetter than the field?Split, metric, protocolDifferent split
Compute-matchedIs it just scale?Budget, tokens, tuningOnly one side tuned
OracleHow much is interface error?Ground-truth symbolsUnachievable, but bounds

A baseline makes “better” meaningful. It does not show which component caused the gain.

S/V · Lecture 21
04 · BaselinesLecture 21 · Benchmarks and evaluation

“Because of what?”

Remove one part. Hold everything else.

Remove the symbolic part

Replace it with an MLP of the same size. If the gain survives, it was not the logic.

Remove the neural part

Feed ground-truth symbols. The gap is the cost of perception, not of reasoning.

Shuffle the structure

Keep the capacity, destroy the semantics. If the score holds, the content was not used.

An ablation is a causal claim. If parameters, data, compute, or tuning change with the component, the difference is confounded.

A parameter-matched replacement is stronger than deletion; a shuffle control is stronger still.

S/V · Lecture 21
04 · BaselinesLecture 21 · Benchmarks and evaluation

Before you believe the delta

Six things that must not move.

Parameters

Same capacity, or a matched replacement.

Data

Same examples, same preprocessing.

Compute

Same tokens, epochs, hardware budget.

Tuning

Same search budget for both sides.

Seeds

Mean and spread, not one run.

Metric

Same split, same scorer, same protocol.

COGS: ±6–8 points across seeds. A three-point gain from one seed may be a seed.
S/V · Lecture 21
05 · Score vs interfaceLecture 21 · Benchmarks and evaluation

Why the number can lie

Accuracy is a marginal. The interface is a conditional.

Shortcut

The benchmark has a correlate; the model learns it. Answer priors, templates, inverse relations.

Aggregation

The mean hides the slice. 96–99% in distribution, 16–35% compositional.

Mismatch

A valid plan is not executable; success is not grounding; MRR is not reasoning.

Correction masking

The constraint repairs the misread digit; the neural error disappears from the score.

Report the slice, the trace, and the correction. The average is not a diagnosis.

S/V · Lecture 21
05 · Score vs interfaceLecture 21 · Benchmarks and evaluation

What to report instead

Diagnose the interface, not just the label.

DiagnosticQuestionExample
Per-slice / worst groupWho is the average hiding?Per question type, per attribute
CounterfactualDoes the answer follow the symbol?Swap an attribute, rename an entity
Interface traceWhere did the step go wrong?Program, plan, proof, derivation
Calibration and costIs confidence honest? At what price?Reliability diagram; latency
The Embodied Agent Interface separates goal interpretation, subgoal decomposition, action sequencing, and transition modelling — and reports hallucination, affordance, and planning errors separately.

Li et al., arXiv:2410.07166

S/V · Lecture 21
06 · Five questionsLecture 21 · Benchmarks and evaluation

From Lecture 01, sharpened

Five questions, five red flags.

QuestionLook forRed flag
1 · Neural component?Input, output, data, error rate“A neural network,” no specification
2 · Symbolic component?Language, semantics, knowledge base“Logic,” no axioms
3 · Interface?Loss, architecture, data, search, verifierA pipeline with nothing to inspect
4 · Guarantee?Empirical or formal; artefact; assumptions“Guarantees,” no theorem or scope
5 · Benchmark?Task, split, metric, baseline, ablationOne accuracy number, no split
Add a sixth: what would falsify the claim? If nothing would, the claim is not yet empirical.
S/V · Lecture 21
07 · ProtocolLecture 21 · Benchmarks and evaluation

Forty-five minutes

Eight outputs, one page.

StepTimeOutput
0 · State the claim2 minOne sentence with comparand and scope
1 · Draw the interface8 minNeural, symbolic, interface; what is trained
2 · Build the table8 minBenchmark × split × metric × baseline × seeds
3 · List ablations6 minEach removal and its question
4 · Write the guarantee6 minAssumptions, artefact, scope
5 · Check leakage6 minSplit unit, dedup, contamination
6 · Read errors6 minPer-slice, counterfactual, trace
7 · Write five lines5 minClaim, evidence, strongest, weakest, next
S/V · Lecture 21
07 · ProtocolLecture 21 · Benchmarks and evaluation

Interactive · audit the five slots

Build the report, one slot at a time.

A claim is ready only when all five are named. Toggle them and watch the verdict move. The verdict is a completeness ladder, not a quality score.

Not a claim yet 0 / 5

Start with the benchmark: without a task, the other slots have nothing to attach to.

The full version, with field-specific flags and a written report, is the lab: evaluation planner.

S/V · Lecture 21
08 · RecapLecture 21 · Benchmarks and evaluation

What you should be able to do

Four habits.

Read

A benchmark as an instrument with a construct, a scope, and a blind spot.

Audit

The split, the correlate, the overlap, and the contamination.

Compare

A baseline for direction; an ablation for attribution.

State

The guarantee, its artefact, its assumptions, and its failure mode.

S/V · Lecture 21
08 · RecapLecture 21 · Benchmarks and evaluation

Next

Read the next paper with a protocol.

Take a paper from the reading list. State the claim, draw the interface, fill the five slots, and write the five lines. The lab will tell you which slot is empty.

BenchmarkSplitBaselineAblationGuarantee

Five slots. One missing slot, one lower rung — the report says which.

S/V · Lecture 21
Speaker notes