Bench evidence & honesty

Bench is designed so a report can be handed to a skeptical reviewer — an engineer, a finance partner, an auditor — and survive scrutiny. This page defines the vocabulary a report uses and the honesty rules behind it. Read it before you quote a Bench number.

Dataset cards

Freezing a suite produces one immutable, content-addressed weighted dataset per call type, described by a dataset card. The card is the provenance of everything downstream:

• Unique and cluster counts. Exact duplicates are collapsed but retain their frequency weight; near-duplicates form visible clusters rather than disappearing. The decision unit is the independent cluster, not the raw call count.
• Strata and weights. The same frozen population weights are shared by the quality and cost views, so they can never disagree about what the population is.
• Exclusions. PII redaction always runs; a low-confidence item is excluded with a visible per-item reason (unless the suite carries an audited attestation admitting it post-redaction). Test and Bench traffic never leak into production golden sets.
• A pre-registered power plan (below).

Identical captures always produce an identical dataset_hash — the card is content-addressed and independent of envelope ids, so a re-freeze is a no-op.

Power plans

The dataset card carries a power plan: a planning statement of the form "this dataset can detect about an X percentage-point quality change at 80% power", derived from a conservative-prior / A-A variance estimate. It also states the minimum detectable effect (MDE).

The MDE is a power diagnostic only — it is never the verdict. A small dataset with a wide MDE does not fail; it produces an inconclusive verdict with the MDE stated so you know why. Raw item count never establishes decision power on its own.

Paired vs. retained compare

Two things Bench deliberately does not do:

• It never trusts a historical output as a control. Your current incumbent is re-invoked and charged in the same run as each challenger, contemporaneously, from the same pinned item envelope through the same execution profile. A captured historical output is provenance, not a perfect counterfactual.
• It never re-invokes an output for scoring without charging it. Under explicit evaluation-retention consent, the exact billed output is retained (encrypted, purpose-limited, TTL-bounded) so cost and quality are proven from the same retained invocation. Without that artifact the observation is ineligible — never silently re-invoked and mislabeled as paired.

Compare mode runs your current effective config as the single arm and pairs it, by content hash, against a pinned baseline run's retained arm artifacts. A missing or expired baseline artifact makes that pair ineligible and degrades the verdict toward inconclusive — it is never silently reported as green.

Dual-interval verdicts

Every migration decision is an interval-based tri-state on the frozen population. Two paired cluster-bootstrap intervals are computed on the same items:

1. a quality non-inferiority interval, and
2. a cost-savings interval.

The verdict is:

• pass — only when both registered bars clear (the quality interval's lower bound stays above the non-inferiority margin and the cost-savings interval's lower bound clears the registered minimum-savings bar).
• fail — when a regression is established, a hard-scorer or established outcome gate breaks, or an absolute per-unit cost ceiling is breached (the ceiling fails independently of quality).
• inconclusive — otherwise. A quality pass with an uncertain cost interval is inconclusive; a cost pass with a hard-outcome regression is fail.

A keep / inconclusive outcome is first-class and always carries a named remediation (e.g. capture more traffic, register an outcome, resolve an evaluator question) — a report never pretends an unproven result is a win. The pre-registered margins state their snapshot provenance (the pinned run spec, never live config), so a mid-flight config edit can never move an in-flight verdict. Multiplicity across finalists and repeated/sequential re-runs are controlled, so a repeated-winner loop cannot inflate the false-positive rate.

Cost views and the provider-reported vs. estimate rule

A cost figure is only ever labeled by its basis, and the unit is explicit:

• $/task is used only when a true task runner fed outputs forward and closed the outcome (or a proven deterministic one-request oracle applies). Otherwise a figure is $/request.
• Token components are shown as billed units, never as a universal efficiency unit (different tokenizers are not fungible).

The word measured labels a cost figure only when its usage was provider-reported and replayable (or invoice-reconciled). It is never applied to an estimated (chars÷4) usage basis — that is labeled a discovery estimate. Per-token list or blended price is a discovery input only: it can shortlist a candidate but it can never rank, emit, or auto-apply a migration. Any savings percentage a surface shows is labeled as an estimate or a projection, never a bare headline number.

Trust classes

Every run and every metric carries an evidence-class tag and a composite trust class, so a reader always knows how the quality signal was produced:

• Class A — at least ~70% of the composite weight is programmatic scorers plus first-party outcomes (your CI assertions, ur.bench.expect/ur.bench.record). The strongest, judge-light evidence.
• Class B — a mix, still weighting programmatic and outcome evidence above the judge.
• Class C — judge-dominant. Legitimate, but disclosed as such and reported with the judge's calibration (agreement kappa).

Programmatic assertions and first-party outcomes always outweigh the LLM judge; when the deterministic tiers cover every applicable risk, the judge residue is retired entirely (a zero-judge run). A same-provider-family judge is rejected per pair. This Bench trust class (A/B/C) is distinct from the evidence-interoperability trust class used for imported external scores.

Evidence bundles

Any completed run exports an evidence bundle (?format=bundle on the report, or the SDK/CLI). A bundle is a self-contained, content-addressed record of the frozen dataset card, the pinned run spec, the immutable per-arm cost evidence, the run's pinned stats, and first-party outcomes — with every decision-gated figure linked to its lineage ids.

A bundle verifies offline: pnpm evidence:verify <bundle.json> re-checks it with no network and no ultrarouter account, so the proof survives even after a customer leaves. The report reads no realized-ledger table — source separation is enforced so a realized saving can never be inferred from a Bench report, and the incumbent renders its actual re-invoked cost evidence, never a fictitious $0 placeholder.

Where this leaves you

A Bench report proves a paired comparison under stated evidence and limitations. It is not a claim of realized cash — realized cash lives on the invoice-reconciled savings ledger after you actually migrate. Bench tells you, honestly and reproducibly, whether a change is worth making; the ledger tells you what it saved once you made it.