Two papers report the same number on the same benchmark. One claims its model composes; the other claims it is accurate. The number cannot tell them apart. Evaluation is the discipline of restoring what the number leaves out: the split, the baseline, the ablation, the metric, and the guarantee.
00 / The evaluation problemA score is an argument with missing premises.
A benchmark is an instrument: a task, a dataset, a split, a metric, and usually a leaderboard. Evaluation is the inference from the instrument’s output to a claim. The instrument does not make the inference. A score on CLEVR gives you a number; it does not tell you what the model can do off the test distribution, or which component is responsible.
The inference has a standard form: on distribution D, metric M, system S beats comparand B by Δ, because of component C, under guarantee G. Every slot can be empty. An empty slot does not make the claim false; it makes it under-specified. The reader’s job is to find the empty slots before believing the sentence in the abstract.
Evaluation is the argument from a measurement to a claim. The benchmark supplies the measurement; the split supplies the scope; the baseline supplies the comparison; the ablation supplies the attribution; the guarantee supplies the promise.
Four validities fail for different reasons. Construct: does the benchmark operationalize the named capability? A CLEVR score is evidence about closed-world compositional VQA, not about visual reasoning in general. Internal: is the effect caused by the named component? That is the ablation question. External: does it survive a change of distribution or embodiment? Statistical: is the difference larger than seed noise?
Statistical validity is not a formality. COGS reports 96–99% in-distribution, 16–35% on its generalization splits, and ±6–8 points of seed sensitivity (Kim & Linzen, arXiv:2010.05465). A six-point swing is larger than many reported gains. A three-point gain from one seed may be a seed.
The asymmetry that organizes the lecture: a score is a marginal; an interface is a conditional. Accuracy averages over examples. An interface failure is about a subset: the unusual object, the unseen combination, the correction that was right for the wrong reason. A model can be right on average and wrong at the seam.
A thermometer has a calibration certificate. A benchmark has an argument. Treat the score as evidence, not as a measurement of the capability itself.
01 / What benchmarks measureSix instruments, six blind spots.
These six are not a survey; they are the ones this course has used. Each isolates a different claim, and each is silent about something the others measure.
| Benchmark | Task | Metric | Diagnoses | Does not |
|---|---|---|---|---|
| CLEVR | Questions over generated scenes | Accuracy, per type | Binding, relations, counting, composition | Open vocabulary, real images |
| SCAN | Commands → actions | Exact sequence | Lexical and structural composition | Natural language, perception |
| COGS | Sentences → logical forms | Exact form | Compositional semantic parsing | Open domain, world knowledge |
| PlanBench | PDDL planning | Validity, cost | Planning and reasoning about change | Perception, execution, uncertainty |
| ALFWorld | Household tasks | Success rate | Grounded, long-horizon action | Real perception, control |
| KG | Link prediction, queries | MRR, Hits@k | Relational generalization | Truth, novelty, time |
CLEVR: composition in a closed world
CLEVR was built against the biases of earlier visual question answering datasets. Its scenes are generated; its questions are generated from functional programs; and every question carries an annotation of the reasoning it requires. The paper claims “minimal biases,” not realism (Johnson et al., arXiv:1612.06890). It is a good instrument for attribute binding, relations, counting, and composition in a closed vocabulary. It is a poor instrument for open vocabulary, real images, ambiguity, or outside knowledge. Its CoGenT split holds out shape–colour combinations: a real test of binding, and a narrow one.
SCAN and COGS: composition in language
SCAN pairs navigation commands with action sequences in a tiny procedural language. Its splits hold out a primitive (“jump”), longer sequences, or new templates. Lake and Baroni report that seq2seq RNNs generalize when the test difference is small enough for “mix-and-match” heuristics, but fail “spectacularly” when the generalization is systematic (arXiv:1711.00350). Near-ceiling on the standard split is not compositional generalization.
COGS makes the point in semantic parsing. Its evaluation set contains new combinations of familiar words and structures, including deeper recursion. Transformers and LSTMs reach 96–99% in-distribution and 16–35% on the generalization splits, with ±6–8% seed sensitivity (arXiv:2010.05465). The gap is the phenomenon, not a bug. COGS is still a fragment of English: it says nothing about reference, ambiguity, or discourse.
PlanBench and ALFWorld: action
PlanBench evaluates planning in PDDL domains from the planning community. It asks for plan generation and verification, and reports validity and cost. The design answers a confound: common-sense tasks make it hard to tell planning from retrieving world knowledge. LLM plan generation “falls quite short,” even for state-of-the-art models (Valmeekam et al., arXiv:2206.10498). A PlanBench result is about valid plans in a formal domain — not perception, grounding, execution, or uncertainty. Validity is not optimality.
ALFWorld aligns text household tasks with ALFRED goals, so an agent can learn an abstract policy in text and execute it in a visual house (Shridhar et al., arXiv:2010.03768). The metric is task success. Success is a good outcome and a poor diagnostic: exploration, a memorized floor plan, or a templated action space can all produce it. In the text variant, observations are text; the actions are discrete, and the physics is simulated. It tests embodied, long-horizon action in a house; it does not test a real kitchen.
Knowledge graphs: relational generalization
Knowledge-graph benchmarks come in two shapes. Link prediction ranks a missing head or tail on FB15k-237, WN18RR, or YAGO3-10; the metrics are MRR and Hits@k. Query answering asks for the answer set of a multi-hop query. Both measure relational generalization over a fixed graph, and both are usually transductive: test entities were seen in training. A high MRR says the model ranks missing links among old nodes well; it does not say it can reason about a new entity, a new relation, or a changed schema.
The protocol matters. Toutanova and Chen showed that inverse relations in FB15k make many test triples near-duplicates of training triples, and that a simple inverse model scores well; FB15k-237 was built to remove that. Sun et al. found that some very high completion scores were attributable to an inappropriate evaluation protocol (arXiv:1911.03903). A KG number is a number about a protocol.
Benchmark descriptions and the COGS figures come from the cited papers, not from re-runs. Ranges are reproduced as ranges. The KG survey (Hogan et al., arXiv:2003.02320) supplies the taxonomy, not a score.
Every benchmark is a trade: control for realism, or realism for control. The question is not which is best, but which can support this claim and which part it leaves unsupported.
02 / Compositional splitsThe test distribution is a design choice.
A compositional split holds out a combination of known parts — or a part itself — so that the test set requires something other than interpolation. It distinguishes interpolation — a test point near the training cloud, solvable without representing the parts — from recombination — a new combination that requires them. The split is the operational part: it is a hypothesis about what “compositional” means for this task.
Four families test different things. Lexical: a held-out primitive (SCAN add-jump) tests extending the lexicon. Structural: a held-out template or depth (SCAN length/template, COGS structures) tests applying a rule recursively. Attribute: a held-out combination (CLEVR-CoGenT) tests binding without a co-occurrence cue. Entity or time: held-out nodes or periods test reasoning about the unseen.
| Family | Held out | Tests | Example |
|---|---|---|---|
| Lexical | A primitive | Extending the lexicon | SCAN add-jump |
| Structural | A template or depth | Applying a rule recursively | SCAN length; COGS structure |
| Attribute | A combination | Binding without a cue | CLEVR-CoGenT |
| Entity / time | Nodes or periods | Reasoning about the unseen | Inductive KG; temporal split |
A split can fail in three ways. The unit can be wrong: splitting by question while the same scenes are in train and test measures question generalization, not scene generalization. The correlate can leak: a held-out cell is only out of distribution if it is not predictable from a training cue; if colour predicts shape, the held-out shape–colour cell may be solvable by colour. The balance can fool you: twenty held-out examples give a noisy estimate, and a 98% train split is a different experiment from a balanced one. Report per-cell counts.
This is a designed distribution shift. It can be covariate, label, or concept shift; most compositional benchmarks are covariate by construction. They do not tell you what happens when the label semantics changes. “Compositional” is not a synonym for “robust.”
A model can pass lexical and fail structural; pass CLEVR-CoGenT and fail real images. Ask which split, at which unit, with which correlate controlled. If the answer is “the standard test set,” the claim has no split behind it.
03 / LeakageThe test set is only as clean as its construction.
Leakage is information in training, selection, or tuning that would not be available at test time, or any use of test information to make design choices. It is a family, not a bug.
| Kind | How it inflates | Check |
|---|---|---|
| Example | Near-duplicates across train and test | Deduplicate at the claim’s unit |
| Split | Held-out cell predictable from a correlate | Check conditional distributions |
| Entity | Test entities seen; inverse relations | Inductive split; filter inverses |
| Pretraining | Benchmark items in the corpus | Cutoff; private or perturbed test |
| Test-set tuning | Leaderboard iteration | Pre-register; private holdout |
| Metric | Priors, templates, frequency | Trivial baselines; per-slice checks |
Example leakage. CLEVR questions are generated per scene; if the same scene appears in train and test with a different question, the model can memorize the scene. Deduplicate at the unit of the claim and report overlap. Split leakage. If red predicts cube in training, a held-out red sphere can be answered from colour. Check the conditional distribution, not just the marginals.
Entity leakage. Transductive benchmarks such as FB15k-237 and WN18RR keep the entity set fixed; test entities were seen. That is a scope condition, but it becomes leakage when the claim is about new entities. In the older FB15k, inverse relations make it worse: the reversed triple may be in training, and a simple inverse model scores well. FB15k-237 was built to remove that; a paper reporting only FB15k is reporting a leakage-prone result. Pretraining contamination. Public benchmarks are in model corpora. Evaluate after the cutoff, keep a private or perturbed set, report a check. Deng et al. show the gap between inflated scores and actual performance and propose detection methods (arXiv:2311.09783). A canary string is a crude tripwire.
Test-set tuning is leakage by iteration: hundreds of submissions turn the test set into a validation set. Metric leakage: answer priors, templates, or frequency can drive the score. A trivial baseline is the cheapest check and the most often omitted.
You cannot usually correct a leaked score by subtracting a number. Change the design — a cleaner split, a private test set, an inductive protocol, a temporal cutoff — or weaken the claim.
04 / Baselines & ablations“Better than what?” and “because of what?”
A baseline makes “better” meaningful: better than a majority-class predictor, a symbolic planner, a neural-only model, the published state of the art, or a compute-matched system. An ablation removes or replaces one component to attribute the effect. A baseline answers compared with what; an ablation answers because of what.
| Comparison | Question | Hold fixed | Typical failure |
|---|---|---|---|
| Trivial | Harder than the prior? | Split, metric | Majority class is strong when imbalanced |
| Symbolic-only | What do rules buy? | Knowledge base, items | Rules tuned on test |
| Neural-only | What does learning buy? | Data, parameters, compute | Smaller and trained less |
| Published SOTA | Better than the field? | Split, metric, protocol | Different split or preprocessing |
| Compute-matched | Is it just scale? | Budget, tokens, tuning | Only one side tuned |
| Oracle | How much is interface error? | Ground-truth symbols | Unachievable, but bounds |
A paper that compares to a neural-only model has a baseline; it has not shown the symbolic component caused the gain. A paper that removes its own logic loss has an ablation; it has not shown it beats the alternatives. The strongest papers report both, in separate tables. Label each row with the question it answers; if it fits neither, it is doing neither.
A good ablation changes one thing and holds the rest fixed. If removing the symbolic part also removes parameters, the difference may be capacity; if it removes training time, optimization; if the ablated model is tuned less, the comparison is unfair. A parameter-matched replacement (an MLP of the same size for the logic layer) is stronger than deletion. A shuffle control — destroy the semantics, keep the capacity — is stronger still.
Report variance. One run is an anecdote, and COGS’s ±6–8 points is the reason. Report mean and standard deviation over seeds, or a confidence interval; use paired comparisons when the design allows; do not compare best-of-five to a single baseline. For language models, report prompts, decoding, samples, and cost: they are part of the intervention.
A baseline answers “better than what?”; an ablation answers “because of what?” If a comparison cannot be assigned to one, it is not yet evidence for either.
05 / Score vs interfaceWhy a high number can hide a broken seam.
Accuracy is a marginal; an interface is a conditional. The gap is where the interesting failures live. Four mechanisms recur.
Shortcut. The benchmark contains a correlate — an answer prior, a template, a scene statistic, an inverse relation — and the model learns it instead of the capability. Shortcut learning is what empirical risk minimization does when the shortcut is easier than the intended feature (Geirhos et al., arXiv:2004.07780). Accurate and ungrounded can coexist.
Aggregation. The mean hides the slice. COGS is the clean case: 96–99% in-distribution, 16–35% compositional. A paper that reports only the first number makes the interface invisible. Report per-type, per-split, per-attribute, and worst-group numbers.
Interface mismatch. The measured artefact is not the claimed one. A plan can be valid in PDDL but not executable because the grounding is wrong. ALFWorld success can come from exploration; it does not measure efficiency or grounding. A KG rank can come from an inverse relation. A shield can guarantee “no hazard” relative to its model and specification — not task success.
Correction masking is the NeSy-specific version. The all-different constraint fixes a misread digit; the planner repairs an infeasible action; the type checker rejects a malformed program. End-to-end accuracy looks good and the neural error is invisible. But the correction depends on the constraint being right and applicable. Report the neural error rate separately, and ablate the correction.
| Symptom | May hide | Diagnostic |
|---|---|---|
| High answers, low programs | Composition shortcut | Program and per-type accuracy |
| Valid plans, high cost | Poor planning quality | Cost and optimality gap |
| High success, many steps | Exploration, not grounding | Efficiency; per-task success |
| High MRR, transductive | Entity or inverse leakage | Inductive split; filtered ranking |
| High average, low worst group | Aggregation | Per-slice and worst-group |
| Confident wrong answers | Poor calibration | Reliability diagram; ECE |
Diagnostic evaluation reports more than one number: per-slice accuracy, counterfactual perturbations (swap an attribute, rename an entity, shuffle a relation), interface traces (program, plan, proof, derivation), ablations, and cost. The Embodied Agent Interface is the model: it separates goal interpretation, subgoal decomposition, action sequencing, and transition modelling, and reports hallucination, affordance, and planning errors separately (Li et al., arXiv:2410.07166).
Robustness is accuracy under a specified perturbation, against a specified baseline. If the perturbation is not described, “robust” is a mood. Underspecification is the sibling problem: many models fit the same data and differ off-distribution, so one held-out score does not identify which one you have (D’Amour et al., arXiv:2011.03395).
06 / The five questionsThe Lecture 01 questions, sharpened for evaluation.
The five questions still apply; evaluation adds a second column. Ask each about the system and about the evidence.
| Question | Look for | Red flag |
|---|---|---|
| 1 · Neural component? | Input, output, data, error rate, frozen or trained | “A neural network,” no specification |
| 2 · Symbolic component? | Language, semantics, knowledge base, assumptions, cost | “Logic,” no axioms |
| 3 · Interface? | Loss, architecture, data, search, verifier | A pipeline with nothing to inspect |
| 4 · Guarantee? | Empirical or formal; artefact; assumptions | “Guarantees,” no theorem or scope |
| 5 · Benchmark? | Task, split, metric, baseline, ablation, leakage | One accuracy number, no split |
If 3 has no answer, the system is a pipeline. If 4 has no answer, the claim is empirical, whatever the abstract says. If 5 has no answer, the result may be real but the lesson narrow. The questions locate the contribution; they are not a checklist for rejection.
Evaluation adds a sixth: what would falsify the claim? A split (fails on unseen combinations), a baseline (a trivial model matches it), an ablation (the gain survives removing the component), or a guarantee (the monitor misses a violation). Write it down before the results.
07 / The reading protocolForty-five minutes, eight outputs.
Reading is an evaluation task. This protocol fits 45–60 minutes and ends with notes you can defend. Times are a budget.
| Step | Time | Output |
|---|---|---|
| 0 · State the claim | 2 min | One sentence with a comparand and scope |
| 1 · Draw the interface | 8 min | Neural, symbolic, interface; what is trained |
| 2 · Build the table | 8 min | Benchmark × split × metric × baseline × seeds |
| 3 · List ablations | 6 min | Each removal and its question |
| 4 · Write the guarantee | 6 min | Assumptions, artefact, scope |
| 5 · Check leakage | 6 min | Split unit, dedup, contamination |
| 6 · Read error analysis | 6 min | Per-slice, counterfactual, trace |
| 7 · Write five lines | 5 min | Claim, evidence, strongest, weakest, next |
Step 0 is the one people skip: write the claim with a comparand — “S reaches M on D, better than B, because of C, under G.” If a slot is missing, the sentence shows you which one. Step 1 is a drawing, not a summary: neural, symbolic, interface, what is trained, where information crosses. If you cannot draw it, the method section is under-specified; that is a finding.
Step 4 is where reviews are too generous. A guarantee is always relative. Write: under A, S guarantees G for artefact Y at cost C; it does not guarantee G′ outside A. The table gives five cases; the pattern is the same — the guarantee is real and narrower than the abstract.
| Guarantee | Covers | Does not cover | Assumption |
|---|---|---|---|
| Runtime shield | No hazard relative to model and specification | Task success; unmodelled hazards | Model and specification correct |
| Conformal prediction | Marginal coverage at 1 − α | Conditional coverage per subgroup | Exchangeability |
| Theorem prover | Proof is checkable | That the formalization matches the informal problem | The formal statement is the intended one |
| Classical planner | Valid plan relative to the model | Executability in the world | Model and grounding correct |
| Benchmark score | Accuracy on D | Any guarantee outside D | D represents deployment |
A verifier is the clearest case. A monitor can be sound with respect to a specification: if it says “safe,” the trace satisfies the spec, assuming it observes what it thinks it observes. That is a real soundness property. It is not a property of the policy, the perception, or the world. The guarantee is about the artefact checked.
Step 7 is five lines: claim, evidence, strongest, weakest, next experiment. If the next experiment is obvious and cheap, the paper did its job. If it is “reproduce the benchmark,” the paper has not made a testable claim yet.
The evaluation planner asks for the five slots — benchmark, split, baseline, ablation, guarantee — and builds a structured report. It does not score the paper. It shows which slots are empty, which claims the filled slots support, and which sentence in the abstract is still unearned.
08 / RecapWhat you should be able to do now.
Read a score as an argument: task, distribution, metric, comparand, attribution, guarantee.
Say what six benchmarks measure — and what they do not.
Distinguish interpolation from recombination, and name a split’s held-out unit.
Audit example, split, entity, pretraining, test-set, and metric leakage.
Tell a baseline from an ablation, and state the question each answers.
Explain how a high score hides a broken interface, and name a diagnostic.
Apply the five questions and the falsifier question to any NeSy paper.
Run the 45-minute protocol and write a five-line review.
Exercises
- Audit one benchmark. Pick CLEVR or SCAN. Write three sentences on what it measures and three on what it does not. Then check whether a paper’s claim stays inside that boundary.
- Design a split. Hold out one attribute combination. Name two shortcuts that could still solve the held-out cell, and one design change that blocks each.
- Hunt leakage. Take a paper with public code. Check the dedup unit, the train/test overlap, and whether test entities or templates appear in training.
- Label the comparisons. Take a results table. Label each row baseline, ablation, or both, and write the question it answers. If it fits neither, say why.
- Write three guarantees. One sentence each for a shield, a conformal predictor, and a theorem prover. Name the artefact, the assumption, and one thing it does not cover.
- Run the protocol. Pick a paper from the reading list and spend 45 minutes on the eight steps. Compare your “weakest point” with the paper’s limitations section.
- Use the lab. Load a preset, build the report, and add the missing fields one at a time. Watch the verdict change. Then write the claim you would defend.
Check your understanding
Ten questions, with explanations. The score is not the point; the questions you cannot answer are the reading list.
Reading list
The first five are the benchmarks; the rest are the survey, the methods, and the framing.
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson et al. · CVPR 2017 · the closed-world visual benchmark
Generalization without systematicity
Lake & Baroni · ICML 2018 · SCAN and the add-primitive split
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
Kim & Linzen · EMNLP 2020 · the in-distribution/generalization gap
PlanBench
Valmeekam et al. · NeurIPS 2023 · planning in formal domains
ALFWorld: Aligning Text and Embodied Environments
Shridhar et al. · ICLR 2021 · text and embodied household tasks
Knowledge Graphs
Hogan et al. · ACM Computing Surveys 2021 · the task and protocol landscape
A Re-evaluation of Knowledge Graph Completion Methods
Sun et al. · ACL 2020 · why the protocol matters
Embodied Agent Interface
Li et al. · NeurIPS 2024 · diagnostic metrics for agents
Investigating Data Contamination in Modern Benchmarks
Deng et al. · NAACL 2024 · contamination detection
A Survey on Evaluation of Large Language Models
Chang et al. · ACM TIST 2024 · the broader landscape
On the Measure of Intelligence
Chollet · 2019 · what a benchmark can and cannot measure
A Gentle Introduction to Conformal Prediction
Angelopoulos & Bates · 2021 · what a coverage guarantee covers
Benchmark papers are cited by arXiv ID; the COGS percentages and the SCAN characterization are quoted from the abstracts, not re-run. Both figures are original schematics. The knowledge-graph protocol point follows Toutanova and Chen, “Observed versus latent features for knowledge base and text inference” (3rd Workshop on Vector Space Modeling for NLP, 2015), and Sun et al. (2020); the first has no arXiv ID. No benchmark score here is invented.