Course / Lectures / Lecture 21 / Lab Lab 21

The evaluation planner.

A claim is not ready until you can name the benchmark, the split, the baseline, the ablation, and the guarantee. Fill the five slots and the tool builds a structured report. Leave one empty and it shows you which part of the claim is still unearned.

5 required slots 6 verdict levels 4 preset claims Deterministic · no install

The evaluation planner lab

1 · Start from a claim

Include a comparand and a scope if you can: better than what, on which distribution.

2 · Fill the five slots

Which task and dataset does the claim live on?

What is held out, and at which unit?

What does the system beat?

What was removed or replaced to attribute the gain?

What is promised, about which artefact, under which assumptions?

Choose a preset or write a claim, fill the five slots, then build the report.

The report

Build a report to see the five slots, the claim-strength verdict, and the flags.

The verdict is a completeness ladder, not a quality score. A five-slot report can still be wrong; it is simply auditable.

HOW IT WORKS

The verdict is derived from the checklist.

LEVEL 0 · NO BENCHMARK

Not a claim yet

Without a benchmark there is no task to attach the other slots to. A split, a baseline, an ablation, or a guarantee that is not about a named task is a sentence without a subject.

LEVEL 1 · BENCHMARK

A demonstration

You know the task. You do not know the distribution the claim is supposed to hold on, so “better” has no scope yet.

LEVEL 2 · + SPLIT

A scoped demonstration

You know the task and the held-out distribution. There is still no comparand, so there is no direction to the claim.

LEVEL 3 · + BASELINE

A comparison

You know it beats something on a stated distribution. You do not know which component is responsible; the gain could be parameters, data, or training time.

LEVEL 4 · + ABLATION

An attribution

You know which component did the work, assuming the ablation held everything else fixed. There is still no statement of what is guaranteed outside the benchmark.

LEVEL 5 · + GUARANTEE

An auditable claim

Every slot is filled: task, distribution, comparand, attribution, and promise. This is completeness, not proof. The report can be checked; it can still be false.

What each slot licenses, and what its absence costs.
SlotIf presentIf missing
BenchmarkNames the task and the construct.The claim has no subject.
SplitNames the distribution and the held-out unit.“Better” has no scope.
BaselineNames the comparand.“Better” has no direction.
AblationNames the cause.The gain is not attributed.
GuaranteeNames the promise and its assumptions.The claim is empirical by default.
What the lab is not

It does not score a paper, verify a number, or know the true benchmark. It cannot tell whether the benchmark you selected actually measures the claim, whether the split is leak-free, or whether the guarantee is sound. It checks that the five slots are present and flags a few common mismatches between the claim text and the selected fields. The rest is reading.

Read the reading protocol · Read what the benchmarks measure · Slides