Bench quickstart

This walks an agent (or a human) from nothing to a paired Bench report and a CI gate using only shipped surfaces: the raw REST API, the @ultrarouter/sdk bench option, the ur CLI, and the GitHub Action. Every stateful step is a real curl you can run; the reachability checks at the end are executed against the live gateway in CI, so a broken example fails the build.

Set your gateway and key first — the invite-only hosted gateway is https://api.ultrarouter.io; running the stack locally (pnpm dev:e2e) the base URL is http://localhost:8080; or point at your own deployment.

export UR_BASE_URL="http://localhost:8080"
export UR_API_KEY="ur_sk_test_…"   # a TEST-mode key (ur_sk_test_*), not a live key

1. Capture — tag your traffic

Add one header to the calls you already make. Nothing about how the request is served changes; the gateway durably captures the execution envelope off the hot path and returns an honest ur.bench echo (capture_status: "queued" — never captured: true; only a persisted, replayable envelope is confirmed).

curl -sS "$UR_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-UR-Bench-Suite: checkout-agent" \
  -d '{
    "model": "openai/gpt-5-mini",
    "messages": [{ "role": "user", "content": "Draft a refund reply." }],
    "call_type": "refund-reply"
  }'
# → 200; the response carries "ur": { "bench": { "suite": "checkout-agent",
#   "capture_status": "queued" } }

The suite auto-creates on first use. Run whatever produces representative traffic — your integration tests, a staging click-through, or an agent harness. Every distinct call_type becomes its own benchmark. With the SDK, set it once at construction:

import UltraRouter from "@ultrarouter/sdk";
const ur = new UltraRouter({ apiKey: process.env.UR_API_KEY, bench: "checkout-agent" });
// every request — including streams and retries — now carries the bench headers.

2. Check readiness

The readiness meter reports confirmed capture counts per call type and, once frozen, the qualified dataset card. It speaks in qualified-evidence terms — there is no raw-count "n ≥ 30 so you're ready" verdict (see power plans).

curl -sS "$UR_BASE_URL/v1/bench/suites/checkout-agent/status" \
  -H "Authorization: Bearer $UR_API_KEY"   # scope: config:read

3. Freeze into an immutable dataset

Freezing snapshots the confirmed captures into an immutable, content-addressed, per-call-type weighted dataset (exact dedupe keeps frequency; near-duplicates cluster; PII is redacted and low-confidence items are excluded with a visible reason). It returns each dataset card. Re-freezing identical captures is idempotent — the same dataset_hash.

The optional max_items caps the per-call-type frozen-item count (default 500; hard ceiling 5000). Over-cap uniques drop keeping the highest frequency and are recorded as over_cap exclusions, and the dataset card's item count reflects what actually runs.

curl -sS -X POST "$UR_BASE_URL/v1/bench/suites/checkout-agent/freeze" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "max_items": 500 }'   # scope: config:write; omit the body for the default cap

4. Plan (or run) a paired evaluation

The planner estimates one paired eval per otherwise-runnable (frozen, ≥ 10 unique) call type. A mandatory max_cost_usd is one global umbrella ceiling. Start with dry_run: true: it returns the estimate without any mutation, reservation, or provider call.

Add require_power: true to turn the pre-spend power advisory into a gate: an underpowered call type (its power.mde_pp exceeds the registered margin_pp at its cluster_n) becomes not_runnable(underpowered) and is dropped from the estimate before any budget is reserved — so n=9 never spends against a margin it can't detect. Each disposition also carries a power block (cluster_n, required_clusters, mde_pp, margin_pp, underpowered) and the response surfaces judge_worst_case_usd additively (never summed into estimate_usd); an unaffordable judge phase is refused at the trigger with judge_phase_unaffordable before any arm spend.

curl -sS -X POST "$UR_BASE_URL/v1/bench/suites/checkout-agent/runs" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "max_cost_usd": 3, "dry_run": true, "require_power": true }'   # scope: bench:run
# → estimate + judge_worst_case_usd + remaining allowance + per-call-type dispositions
#   (each with its power block); require_power parks underpowered call types not_runnable.

Live paid dispatch is controlled by the bench.paid_dispatch kill switch (an operator flag, default ON). On any seeded stack (dev/e2e, or a self-hosted deployment after db:seed) the switch is ON, so omitting dry_run DISPATCHES a real paired run — it consumes a monthly run slot, reserves against your prepaid allowance under the umbrella max_cost_usd ceiling, and makes paid provider calls. Every child settles its allowance hold truthfully (the full actual spend is debited once, unspent reservation refunded).

When an operator flips the switch OFF (or on an un-seeded database, where an unknown flag fails closed), live dispatch is refused with HTTP 503 atomic_spend_guard_unavailable before any reservation, row creation, queueing, or provider call — and every dry_run disposition that would otherwise be runnable is instead labeled not_runnable with atomic_spend_guard_unavailable, so a blocked shape is never mislabeled runnable.

5. Read the report

The report is immutable and content-addressed — a re-read is byte-stable. It renders, per call type × arm, the cost views labeled by basis ($/request vs $/task, token components as billed units), the pinned trust class, and the dual-interval verdict with a first-class keep/inconclusive outcome and a named remediation.

curl -sS "$UR_BASE_URL/v1/bench/runs/{run_id}/report" \
  -H "Authorization: Bearer $UR_API_KEY"                 # JSON (default)
curl -sS "$UR_BASE_URL/v1/bench/runs/{run_id}/report?format=bundle" \
  -H "Authorization: Bearer $UR_API_KEY" > bundle.json   # an offline-verifiable evidence bundle

Verify a downloaded bundle offline — no network, no ultrarouter account — with pnpm evidence:verify bundle.json.

6. Gate CI

Pin a completed run as the suite baseline, then gate every PR on the compare verdict. See the CI gate page for the full exit-code contract and the GitHub Action.

ur bench baseline {run_id}                       # pin the baseline once
ur bench run checkout-agent --capture "pnpm test:agent" \
  --budget 3 --gate --baseline bench:main --json # exit 0 = pass, 2 = fail/inconclusive

Verify the Bench surface is live

These checks run against your gateway with your key and prove the Bench API is served and auth-gated end to end. They are executed in CI. The stateful steps above need a captured suite and a bench-scoped test key; these reachability checks need neither, so they stay hermetic.

curl -sS --fail "$UR_BASE_URL/openapi.json" \
  | grep -q '/v1/bench/suites/{slug}/freeze' \
  && echo "ok: the Bench API surface is published"

const res = await fetch(`${process.env.UR_BASE_URL}/openapi.json`);
if (!res.ok) {
  console.error("HTTP", res.status);
  process.exit(1);
}
const spec = await res.json();
const benchPaths = Object.keys(spec.paths).filter((p) => p.startsWith("/v1/bench/"));
if (benchPaths.length === 0) {
  console.error("no bench routes in the spec");
  process.exit(1);
}
console.log(`ok: ${benchPaths.length} Bench routes are published`);

The readiness meter is a real, scope-enforced route. This proves it is wired against your gateway (200 with a config:read key on a known suite, 403 when the key lacks the scope, 404 for an unknown suite — all "wired" outcomes):

code=$(curl -sS -o /dev/null -w '%{http_code}' \
  "$UR_BASE_URL/v1/bench/suites/quickstart-demo/status" \
  -H "Authorization: Bearer $UR_API_KEY")
case "$code" in
  200|403|404) echo "ok: the Bench readiness meter is wired (HTTP $code)" ;;
  *) echo "unexpected HTTP $code"; exit 1 ;;
esac

Next steps

• Understand the evidence: dataset cards, power plans, verdicts, trust classes.
• Wire the CI gate into your pipeline.
• The full Bench API reference is generated from the spec.