The evaluation planner.
A claim is not ready until you can name the benchmark, the split, the baseline, the ablation, and the guarantee. Fill the five slots and the tool builds a structured report. Leave one empty and it shows you which part of the claim is still unearned.
The evaluation planner lab
1 · Start from a claim
Include a comparand and a scope if you can: better than what, on which distribution.
2 · Fill the five slots
Which task and dataset does the claim live on?
What is held out, and at which unit?
What does the system beat?
What was removed or replaced to attribute the gain?
What is promised, about which artefact, under which assumptions?
Choose a preset or write a claim, fill the five slots, then build the report.
The report
Build a report to see the five slots, the claim-strength verdict, and the flags.
The verdict is a completeness ladder, not a quality score. A five-slot report can still be wrong; it is simply auditable.
The verdict is derived from the checklist.
Not a claim yet
Without a benchmark there is no task to attach the other slots to. A split, a baseline, an ablation, or a guarantee that is not about a named task is a sentence without a subject.
A demonstration
You know the task. You do not know the distribution the claim is supposed to hold on, so “better” has no scope yet.
A scoped demonstration
You know the task and the held-out distribution. There is still no comparand, so there is no direction to the claim.
A comparison
You know it beats something on a stated distribution. You do not know which component is responsible; the gain could be parameters, data, or training time.
An attribution
You know which component did the work, assuming the ablation held everything else fixed. There is still no statement of what is guaranteed outside the benchmark.
An auditable claim
Every slot is filled: task, distribution, comparand, attribution, and promise. This is completeness, not proof. The report can be checked; it can still be false.
| Slot | If present | If missing |
|---|---|---|
| Benchmark | Names the task and the construct. | The claim has no subject. |
| Split | Names the distribution and the held-out unit. | “Better” has no scope. |
| Baseline | Names the comparand. | “Better” has no direction. |
| Ablation | Names the cause. | The gain is not attributed. |
| Guarantee | Names the promise and its assumptions. | The claim is empirical by default. |
It does not score a paper, verify a number, or know the true benchmark. It cannot tell whether the benchmark you selected actually measures the claim, whether the split is leak-free, or whether the guarantee is sound. It checks that the five slots are present and flags a few common mismatches between the claim text and the selected fields. The rest is reading.
Read the reading protocol · Read what the benchmarks measure · Slides