ultrarouter Bench

ultrarouter Bench turns the dev/test traffic you already send through the gateway into a paired benchmark you can trust — no eval team, no uploaded dataset, no leaderboard contamination. The dataset is your own captured traffic.

Bench is an optional front door to the same evidence engine that powers passive audit mode. You never need it to get value: a base-URL/key swap already gives you passive readiness. Bench is the accelerator you reach for when you want a real, reproducible answer to "is this candidate model / prompt / config at least as good — and cheaper per task — than what I run today?", and a CI gate that fails a PR when it is not.

What it does, end to end

1. Capture — tag ordinary calls with a suite (X-UR-Bench-Suite, the SDK bench option, or the ur.bench body field). The gateway persists the full, encrypted, replayable execution envelope. Test-key traffic captures; a live key is refused (a consent boundary, not a failure).
2. Freeze — snapshot the confirmed captures into an immutable, content-addressed, per-call-type weighted dataset with a dataset card (unique/cluster counts, strata, exclusions, a pre-registered power plan).
3. Run — a paired run re-invokes your current incumbent and each challenger over the frozen dataset, contemporaneously, through the same execution profile.
4. Report — an immutable, content-addressed evidence report: quality and all-in cost intervals per call type × arm, cost views labeled by basis, a trust class (A/B/C), a dual-interval keep / migrate / inconclusive verdict, and downloadable, offline-verifiable evidence bundles.
5. Gate — pin a run as the suite baseline and gate CI on the interval-based compare verdict with ur bench run --gate.

Read next

• Bench quickstart — zero to a captured suite, a frozen dataset, a paired run, a report, and a CI gate, using only shipped surfaces.
• Bench evidence & honesty — dataset cards, power plans, paired-vs-retained compare, dual-interval verdicts, trust classes, evidence bundles, and the estimate/provider-reported labeling rules.
• Bench CI gate — the ur bench run --gate exit-code contract and the GitHub Action inputs.
• Bench API reference — the generated /v1/bench/* routes.

What Bench is not

Bench never invents a second eval engine, a second cost basis, or a second savings number. Every figure it shows references the same immutable run/cost lineage as the rest of ultrarouter, and it never converts an inconclusive result into a pass. A Bench report is proof of a paired comparison, not a claim of realized cash — the savings ledger is where realized cash lives.