ultrarouter docs
API reference

Bench

ultrarouter Bench — dev/test capture suites + readiness.

ultrarouter Bench — dev/test capture suites + readiness.

This page is GENERATED from the OpenAPI spec (openapi.yaml) — it can never drift from the served API. Set UR_BASE_URL (required: your gateway’s URL) and UR_API_KEY (an ur_sk_* key); secrets are always the $UR_API_KEY placeholder.

Get a bench suite readiness meter

GET /v1/bench/suites/{slug}/status

Per-call-type confirmed capture counts (provenance) + the QUALIFIED-EVIDENCE readiness — the latest frozen dataset card (unique/cluster n, strata, preregistered power plan). There is NO raw-count n<30 verdict — readiness is judged on the frozen, qualified evidence. Triggering a run needs the bench:run scope; this readiness read needs config:read. Requires API-key scope: config:read.

Auth: API key (Authorization: Bearer ur_sk_…).

curl -sS -X GET "$UR_BASE_URL/v1/bench/suites/{slug}/status" \
  -H "Authorization: Bearer $UR_API_KEY"

Freeze a bench suite into immutable weighted datasets

POST /v1/bench/suites/{slug}/freeze

Snapshot the suite’s CONFIRMED capture envelopes into immutable, content-addressed, per-call-type WEIGHTED datasets (exact dedupe + near-duplicate clustering, retained frequency, golden-set strata, strict PII admission) and return each dataset CARD (counts, strata, exclusions, preregistered power plan). Re-freezing identical captures is idempotent (same dataset_hash). Requires scope config:write. Requires API-key scope: config:write.

Auth: API key (Authorization: Bearer ur_sk_…).

curl -sS -X POST "$UR_BASE_URL/v1/bench/suites/{slug}/freeze" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "max_items": 0
  }'

Trigger an on-demand paired Bench run

POST /v1/bench/suites/{slug}/runs

Plan one paired eval per otherwise-runnable (frozen, ≥10-unique) call type over the suite’s FROZEN dataset. max_cost_usd is one global umbrella reservation ceiling, never a per-child clamp and never the continuous eval envelope. Live paid dispatch is gated by the bench.paid_dispatch kill switch (default ON): when ON, omitting dry_run dispatches a real paired run (it reserves the prepaid allowance under the ceiling and makes paid provider calls, each child settling its hold truthfully); when OFF (or on an unseeded DB ⇒ fail-closed) live dispatch is refused before mutation or fan-out with HTTP 503 atomic_spend_guard_unavailable. dry_run: true always returns the estimate with no mutation and, when the switch is OFF, labels otherwise-valid paid shapes not_runnable with that blocker. selection mode emits ranked evidence and NEVER a recommendation. DUAL AUTH: the bench:run scope OR a session + bench.write RBAC (workspace named via X-UR-Workspace). Requires API-key scope: bench:run.

Auth: API key (Authorization: Bearer ur_sk_…).

curl -sS -X POST "$UR_BASE_URL/v1/bench/suites/{slug}/runs" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "max_cost_usd": 0
  }'

Get an on-demand Bench run

GET /v1/bench/runs/{id}

The umbrella run (status + estimate + allowance + per-call-type dispositions) plus each child eval run’s planning verdict TIER and PER-ARM evidence (incumbent + challengers: composite score, pass flag, arm cost) + the composite TRUST CLASS (A/B/C) and evidence-class tags. Read-only. DUAL AUTH: the bench:run scope OR a session + bench.write RBAC. Requires API-key scope: bench:run.

Auth: API key (Authorization: Bearer ur_sk_…).

curl -sS -X GET "$UR_BASE_URL/v1/bench/runs/req_0000000000000000" \
  -H "Authorization: Bearer $UR_API_KEY"

Get the immutable Bench evidence report

GET /v1/bench/runs/{id}/report

The IMMUTABLE, content-addressed evidence report for a completed on-demand Bench run. Renders IN ORDER: (1) the dataset card; (2) the power statement (unique n, planning MDE, the pre-registered margins with snapshot provenance); (3) the headline table per call type × arm — cost views BY BASIS with $/request vs $/task units honestly labeled, token components as billed units, inversion callouts, latency, quality ± intervals, evidence classes + the PINNED trust class; (4) the dual-interval verdict (or the recommendation-free selection table) with keep/inconclusive first-class + a named remediation — NO realized-savings inference, NO fictitious $0 incumbent; (5) the honest projection ONLY where trailing production volume exists; (6) the per-trace task view; (7) auto-generated limitations; (8) the reproducibility appendix. ?format=json (default) returns this report; ?format=md a Markdown export; ?format=bundle an offline-verifiable evidence bundle (pnpm evidence:verify-consumable). All formats are audited (bench.report.downloaded). DUAL AUTH: the bench:run scope OR a session + bench.write RBAC. Requires API-key scope: bench:run.

Auth: API key (Authorization: Bearer ur_sk_…).

curl -sS -X GET "$UR_BASE_URL/v1/bench/runs/req_0000000000000000/report" \
  -H "Authorization: Bearer $UR_API_KEY"

Pin a completed Bench run as the suite baseline

POST /v1/bench/runs/{id}/baseline

Pin a COMPLETED bench run as the suite’s baseline. The baseline anchors COMPARE mode: it captures the run’s immutable RunSpec + RETAINED arm artifacts (never a mutable config pointer). Re-pinning advances the baseline; audited (bench.baseline.pinned). Fails closed on an unknown / wrong-suite / not-complete run. DUAL AUTH: the bench:run scope OR a session + bench.write RBAC. Requires API-key scope: bench:run.

Auth: API key (Authorization: Bearer ur_sk_…).

curl -sS -X POST "$UR_BASE_URL/v1/bench/runs/req_0000000000000000/baseline" \
  -H "Authorization: Bearer $UR_API_KEY"

Get a compare run’s baseline-compare gate

GET /v1/bench/runs/{id}/gate

The interval-based tri-state verdict for a COMPARE run. Pairs the candidate’s (current-config) per-item results against the pinned BASELINE run’s RETAINED artifacts (real retained outputs, never re-invoked) BY CONTENT HASH, then yields pass | fail | inconclusive per call type + an INDEPENDENT absolute measured-cost ceiling check. pass requires BOTH the quality non-inferiority interval AND the cost-savings interval to clear their registered bars; a ceiling breach FAILS independent of quality; a missing/expired baseline artifact makes the pair ineligible and degrades toward inconclusive (never silently green). MDE is a POWER DIAGNOSTIC only. The umbrella verdict is the STRICTEST across call types. DUAL AUTH: the bench:run scope OR a session + bench.write RBAC. Requires API-key scope: bench:run.

Auth: API key (Authorization: Bearer ur_sk_…).

curl -sS -X GET "$UR_BASE_URL/v1/bench/runs/req_0000000000000000/gate" \
  -H "Authorization: Bearer $UR_API_KEY"
POST /v1/bench/runs/{id}/share

Mint an IMMUTABLE, REVOCABLE, unguessable-token PUBLIC render of a completed bench run’s report SNAPSHOT (the product’s first UNAUTHENTICATED tenant-data surface). The public payload is BUILT AT MINT TIME from the immutable evidence report and frozen — a later report/config change NEVER changes a minted share. It carries aggregate metrics, model names, methodology/limitations, trust class and the evidence-contribution footer — never workspace identity beyond the chosen display_name, never un-redacted content, never id/key material. REDACTED example previews appear ONLY when the suite carries the audited attestation AND include_previews is set (opt-in never bypasses redaction). The raw token is returned exactly once (only its SHA-256 hash is stored); the public read is by token at /share/bench/<token>. The "Benchmarked with ultrarouter" footer renders on free and is removable only on Team. SESSION + owner|admin RBAC (bench.share) only — there is deliberately no api-key scope. Audited.

Auth: Session cookie + workspace RBAC.

curl -sS -X POST "$UR_BASE_URL/v1/bench/runs/req_0000000000000000/share" \
  -H "Cookie: better-auth.session_token=$UR_SESSION" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "string"
  }'

List a bench run’s public shares

GET /v1/bench/runs/{id}/shares

The run’s public shares for the revoke list: metadata + coarse view counters, NEVER the token (a bearer secret shown once at mint). SESSION + owner|admin RBAC (bench.share).

Auth: Session cookie + workspace RBAC.

curl -sS -X GET "$UR_BASE_URL/v1/bench/runs/req_0000000000000000/shares" \
  -H "Cookie: better-auth.session_token=$UR_SESSION"

Revoke a public bench-report share

DELETE /v1/bench/runs/{id}/share/{shareId}

Revoke a minted share. A revoked share becomes a public 404 (indistinguishable from unknown), effective ≤ one public-cache TTL window. SESSION + owner|admin RBAC (bench.share). Audited (bench_share.revoked).

Auth: Session cookie + workspace RBAC.

curl -sS -X DELETE "$UR_BASE_URL/v1/bench/runs/req_0000000000000000/share/{shareId}" \
  -H "Cookie: better-auth.session_token=$UR_SESSION"

Attach customer assertions to a bench suite

POST /v1/bench/suites/{slug}/expectations

Attach (ur.bench.expect) one or more customer ASSERTIONS mapped onto the CLOSED programmatic scorer vocabulary (json_schema / exact_field / regex_match / citation_substring). Assertions are validated (an unknown kind or malformed config is rejected), versioned and append-only; at FREEZE the applicable ones are MERGED into the call type’s compiled evaluator suite as tier-1-class evidence and the merged suite re-passes mutation qualification (a vacuous/contradictory assertion is rejected with a visible reason). They NEVER mutate a frozen dataset — they attach FORWARD. DUAL AUTH: the bench:run scope OR a session + bench.write RBAC. Requires API-key scope: bench:run.

Auth: API key (Authorization: Bearer ur_sk_…).

curl -sS -X POST "$UR_BASE_URL/v1/bench/suites/{slug}/expectations" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "expectations": [
      {
        "kind": "string"
      }
    ]
  }'

Record per-trace outcomes on a bench suite

POST /v1/bench/suites/{slug}/outcomes

Record (ur.bench.record) first-party per-trace binary OUTCOMES (pass/fail + optional labels) as the strongest human/outcome evidence class (above the LLM judge). Outcomes are scored via the McNemar/binary machinery on paired arms and feed the trace-clustered decision; they are append-only (a superseding verdict wins by version) and attach FORWARD (the next run consumes them). DUAL AUTH: the bench:run scope OR a session + bench.write RBAC. Requires API-key scope: bench:run.

Auth: API key (Authorization: Bearer ur_sk_…).

curl -sS -X POST "$UR_BASE_URL/v1/bench/suites/{slug}/outcomes" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "outcomes": [
      {
        "trace": "string",
        "passed": true
      }
    ]
  }'

Record per-trace outcomes on a bench run

POST /v1/bench/runs/{id}/outcomes

Record (ur.bench.record) first-party per-trace binary OUTCOMES scoped to a specific run (its suite is resolved from the run). Same McNemar/binary + trace-cluster semantics as the suite route; append-only + append-forward. DUAL AUTH: the bench:run scope OR a session + bench.write RBAC. Requires API-key scope: bench:run.

Auth: API key (Authorization: Bearer ur_sk_…).

curl -sS -X POST "$UR_BASE_URL/v1/bench/runs/req_0000000000000000/outcomes" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "outcomes": [
      {
        "trace": "string",
        "passed": true
      }
    ]
  }'

Audit

Zero-configuration Audit Autopilot — the passive readiness report.

Billing

Billing endpoints.

On this page

Get a bench suite readiness meterFreeze a bench suite into immutable weighted datasetsTrigger an on-demand paired Bench runGet an on-demand Bench runGet the immutable Bench evidence reportPin a completed Bench run as the suite baselineGet a compare run’s baseline-compare gateMint a shareable public bench-report linkList a bench run’s public sharesRevoke a public bench-report shareAttach customer assertions to a bench suiteRecord per-trace outcomes on a bench suiteRecord per-trace outcomes on a bench run