Bench CI gate

The ur CLI (shipped in the @ultrarouter/sdk package) turns a Bench suite into a CI gate: it captures bench-tagged traffic, freezes the suite, re-runs your current prompt/pin/config as the candidate, compares it against a pinned baseline under the dual-interval verdict contract, and exits non-zero when the change is not safe.

The ur CLI and the GitHub Action live in the public TypeScript SDK repository, lucastonon/ultrarouter-sdk-typescript, and drive the shipped /v1/bench/* API. They target whatever gateway UR_BASE_URL names: the invite-only hosted gateway (https://api.ultrarouter.io), a local stack or your own deployment. Nothing on this page depends on the hosted service.

The command

ur bench run <suite> \
  --capture "<cmd>" \        # a command that drives your app/tests through the gateway
  --budget 3 \                # per-run USD cap (bounds the atomic allowance reservation)
  --gate \                    # exit non-zero on a fail/inconclusive verdict
  --baseline bench:main \     # a br_* run id, or a symbolic ref → the suite's pinned baseline
  --json                      # versioned machine output (ur.bench.gate.v1)

--capture <cmd> spawns your command with UR_BENCH=<suite> and UR_BENCH_RUN=<run-tag> exported (the run tag defaults to GITHUB_SHA, else the short git SHA), so any traffic it sends through the gateway is captured into a fresh run. UR_API_KEY (a ur_sk_test_… key) and UR_BASE_URL come from the environment, or pass --api-key / --base-url.

Two companion subcommands manage the baseline:

ur bench baseline <run-id>   # pin a completed run as the suite baseline (e.g. after merge to main)
ur bench gate <run-id>       # read a compare run's gate and exit per the verdict

Exit-code contract

The gate maps the tri-state verdict to a CI-friendly exit code, and fails closed — a broken gate is never silently green.

Exit	Meaning
`0`	`pass` — both the quality non-inferiority and cost-savings intervals cleared their bars, no ceiling breach. Also an `inconclusive` verdict when `--allow-inconclusive` is set.
`1`	A usage/argument error (bad flags). Distinct from a gate failure.
`2`	`fail` (a proven regression, a hard-scorer break, or a per-unit cost-ceiling breach), an `inconclusive` verdict (unless `--allow-inconclusive`), or any API/pipeline error (fail closed).

Key flags that shape the verdict:

--allow-inconclusive — treat an inconclusive gate as a pass (exit 0). Off by default: unproven is not green.
--cost-ceiling <usd> — an absolute per-unit cost bound that fails the gate independently of quality.
--mode request|task — the paired-runner mode. task (whole-trace, outputs fed forward) is the only mode that yields $/task evidence; request (the default) yields $/request.
--pin-baseline — on a pass, advance the suite baseline to this run.
--run-tag <tag> — override the default commit-SHA run tag.

A deduped result (an identical run within 14 days) resolves server-side to the linked prior run, so idempotent CI re-runs are free and still gate correctly.

Machine output (`--json`)

--json emits a versioned object tagged ur.bench.gate.v1 (snapshot-tested, so the schema is stable across releases): the suite, run tag, resolved baseline, the tri-state verdict, the exit_code, and the full server gate object. Parse verdict / exit_code in your pipeline rather than scraping the summary table.

GitHub Action

A published composite Action wraps the CLI. It installs the ur bin, runs the gate, and fails the job on a non-zero exit.

- uses: lucastonon/ultrarouter-sdk-typescript@v0.0.1
  with:
    suite: checkout-agent
    api-key: ${{ secrets.UR_BENCH_KEY }}   # a ur_sk_test_… key, stored as a repo secret
    base-url: https://your-gateway.example.com
    baseline: bench:main
    budget: "3"
    capture: pnpm test:agent

The Action's Bench inputs map one-to-one to the CLI flags:

Input	Default	Purpose
`mode`	`bench`	Which gate the Action runs; this page covers `bench`.
`suite`	(required)	The Bench suite slug to run + gate.
`api-key`	(required)	A test-mode Bench key (`ur_sk_test_…`), from a repo secret.
`base-url`	(required)	The gateway base URL.
`baseline`	`bench:main`	A `br_*` run id or a symbolic ref → the suite's pinned baseline.
`budget`	`3`	Per-run USD budget cap (bounds the atomic reservation).
`capture`	`""`	A shell command that drives traffic through the gateway (bench env exported).
`cost-ceiling`	`""`	An absolute per-unit provider-reported cost ceiling (USD); a breach fails the gate.
`paired-mode`	`request`	`request` or `task` (the only `$/task` evidence).
`allow-inconclusive`	`false`	When true, an inconclusive gate exits `0`.
`pin-baseline`	`false`	On a pass, advance the suite baseline to this run.
`run-tag`	`""`	A fresh run tag (default the commit SHA).
`sdk-version`	the Action's release	The `@ultrarouter/sdk` version to install for the `ur` CLI (a pinned Action runs a pinned CLI).

What a gate proves (and does not)

A green gate proves that, on the pinned baseline dataset and under the registered margins, your current config is non-inferior in quality and clears its cost-savings bar — a paired comparison, not a claim of realized cash. Pair it with the savings ledger for what a change actually saves in production, and download an evidence bundle for an offline-verifiable record.