ultrarouter docs

For agents

The machine-first quickstart — MCP server, llms.txt, markdown, OpenAPI.

This documentation is built to be discovered, ingested, and acted on by a coding agent without a human. Everything below is a stable, machine-readable surface.

Fastest path: paste the AGENTS.md snippet into your repo's AGENTS.md / CLAUDE.md and an agent can onboard ultrarouter, add the MCP server, and wire the Bench CI gate from that one block. The rest of this page is the detail behind it.

Which product? Choose when / not when

ultrarouter is five capabilities on one ledger, not five things to wire up. Each block below says when a capability fits your app and when it does not. Every claim here is verifiable from the linked pages through the docs MCP (get_doc/search_docs).

Router

Choose it when you are building any OpenAI-compatible app and want one URL across providers: bring-your-own-key, fallbacks, spend caps, streaming and tools, with a ur block on every response recording which layer decided each field. Point your existing OpenAI or OpenRouter client at the gateway base URL and keep your messages. Not when you are locked to a single provider's non-OpenAI-shaped endpoint and never plan to compare or fail over — a proxy buys you nothing there. Verify: Router · chat completions · migrate from OpenAI.

Evals

Choose it when you are any app shipping prompt/model changes and want a defect to fail CI before a user sees it: evaluators drafted from your traffic, qualified against a labelled set, then gating a candidate from PR to canary to automatic rollback. Not when you have no repeatable task to judge — a one-off script with no recurring prompt or model change has nothing for an evaluator to guard. Verify: Evals · call types API · application CI.

Scoring

Choose it when you have any bill worth a paired test: tag your own dev/test traffic as a suite, freeze an immutable weighted dataset, run your incumbent paired against challengers, and read a signed keep / migrate / inconclusive report with quality and cost intervals you can gate CI on. A keep or inconclusive verdict is the instrument working. Not when the spend is too small to justify a run, or you have no traffic to freeze into a dataset yet — capture some first. Verify: Scoring · Bench API · savings ledger.

Flywheel

Choose it when your app has a checkable shape or a downstream act — NOT engagement-only products: an answer that is later accepted, corrected, resolved or acted on, so an outcome can be joined to it, learned into rules and examples behind a standing holdout, and exported back to you. Not when the only signal is engagement (time-on-page, clicks) with no notion of a right answer — there is nothing checkable to learn from, and a proposed objective never teaches until a live confirmation proves it. Verify: The Flywheel · record outcomes · holdout and lift.

Context

Choose it when you are building apps with more than ~10 SOPs or recurring context bloat: a deterministic runtime assembles context modules under contracts and budgets, and a lazy SOP catalog loads bodies only at the gated phase boundary instead of pasting every procedure into every prompt. The exit is pure — adopt in one file, leave in one file. Not when your prompt is small and stable — a single system prompt with a couple of examples does not need a manifest. Verify: The context runtime · the five products · The Flywheel.

MCP servers

There are two stdio MCP servers, and neither one routes your agent host's own traffic through ultrarouter: the documentation MCP (ultrarouter-docs, below, no key) and the authenticated management MCP (ultrarouter: connection check, call types, learning status and controls, outcome receipts — with your key). Install, packaging for Claude Code and Codex, and uninstall for both: MCP servers for Claude Code and Codex.

The documentation MCP (@ultrarouter/docs-mcp) lets your agent query the docs and the API contract directly. It is not on npm yet; run it from an ultrarouter checkout:

pnpm --filter @ultrarouter/docs-mcp... build
claude mcp add ultrarouter-docs -- node "$PWD/packages/docs-mcp/dist/bin.js"

Or add it to any MCP client config:

{
  "mcpServers": {
    "ultrarouter-docs": {
      "command": "node",
      "args": ["/path/to/ultrarouter/packages/docs-mcp/dist/bin.js"]
    }
  }
}

Tools:

  • search_docs(query, limit?) — ranked full-text search across the docs.
  • get_doc(path) — the clean markdown of one page (by URL or slug).
  • get_openapi(format?) — the generated OpenAPI 3.0 spec (JSON or YAML).
  • example_request(endpoint) — a runnable curl for one endpoint (by operationId or METHOD /path). Secrets are always placeholders.

Text surfaces

  • /llms.txt — an index of every page with one-line descriptions and canonical URLs.
  • /llms-full.txt — the full corpus concatenated, with a structured "for agents" quickstart (env, curl, TS, Python, error taxonomy, rate limits) in the first 2,000 tokens.
  • <url>.md — every page is available as clean markdown at its own URL plus .md (e.g. /docs/quickstart.md), and via Accept: text/markdown content negotiation.

Machine contract

  • /openapi.yaml and /openapi.json — the versioned OpenAPI 3.0 spec, CORS-open. Generated from the gateway's zod route schemas and drift-gated in CI, so it always matches the running API.
  • /sitemap.xml and /robots.txt — AI crawlers are explicitly allowed.

The minimum to make a call

Set UR_BASE_URL (your gateway, e.g. http://localhost:8080) and UR_API_KEY (an ur_sk_live_… / ur_sk_test_… key sent as Authorization: Bearer $UR_API_KEY).

curl -sS "$UR_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-5-mini","messages":[{"role":"user","content":"hi"}]}'

Any OpenAI Chat Completions request body works. The response adds a ur block (request id, model + provider served, routing trail, compression, cost). See the error taxonomy for the stable failure codes.

Zero to a verdict

The exact calls to go from a first request to a defensible keep/migrate/inconclusive verdict and a CI gate — using only shipped /v1/bench/* routes. Capture needs a test-mode key (ur_sk_test_…); a live key with a bench signal is refused (a consent boundary). These are stateful, so they are plain snippets (not the executed reachability checks in the Bench quickstart).

1. First request → captured suite. Add one header to the calls you already make. The serve path is unchanged; the gateway captures the execution envelope off the hot path and echoes ur.bench.capture_status: "queued" (never captured: true).

curl -sS "$UR_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-UR-Bench-Suite: checkout-agent" \
  -d '{"model":"openai/gpt-5-mini","call_type":"refund-reply",
       "messages":[{"role":"user","content":"Draft a refund reply."}]}'

2. Freeze → an immutable dataset. Snapshot the confirmed captures into a content-addressed, per-call-type weighted dataset. max_items caps the per-call-type item count (default 500; over-cap uniques drop keeping the highest frequency).

curl -sS -X POST "$UR_BASE_URL/v1/bench/suites/checkout-agent/freeze" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"max_items": 500}'    # scope: config:write

3. Power check → don't spend on a run that can't decide. Do a dry_run with require_power: true. Read each disposition's power block — cluster_n, required_clusters, mde_pp vs margin_pp, and underpowered — and the run-total judge_worst_case_usd (surfaced additively, never folded into estimate_usd). With require_power: true an underpowered call type becomes not_runnable(underpowered) and is excluded from the estimate/reservation before any budget is reserved.

curl -sS -X POST "$UR_BASE_URL/v1/bench/suites/checkout-agent/runs" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"max_cost_usd": 3, "dry_run": true, "require_power": true}'   # scope: bench:run
# → dispositions[].power {cluster_n, required_clusters, mde_pp, margin_pp, underpowered}
#   + estimate_usd + judge_worst_case_usd (an unaffordable judge phase is refused here).

4. Run → paired dispatch. Drop dry_run to dispatch a real paired run (incumbent re-invoked against challengers) within the same max_cost_usd umbrella. Live paid dispatch is gated by the bench.paid_dispatch kill switch (default ON); when OFF it is refused with atomic_spend_guard_unavailable before any reservation or provider call.

curl -sS -X POST "$UR_BASE_URL/v1/bench/suites/checkout-agent/runs" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"max_cost_usd": 3, "require_power": true}'   # returns bench_run_id (br_*)

5. Verdict → the immutable report. The report is content-addressed (byte-stable on re-read): per call type × arm, cost views labeled by basis ($/request vs $/task), the pinned trust class, and the dual-interval keep / migrate / inconclusive verdict with a named remediation. ?format=bundle is offline-verifiable with pnpm evidence:verify.

curl -sS "$UR_BASE_URL/v1/bench/runs/$RUN_ID/report" \
  -H "Authorization: Bearer $UR_API_KEY"

6. CI gate. Pin a completed run as the suite baseline, then fail a PR when the compare verdict is not a pass (exit 0 = pass, 2 = fail/inconclusive/any API error — fail closed).

ur bench baseline $RUN_ID                       # pin once
ur bench run checkout-agent --gate --baseline bench:main --json

A keep or inconclusive verdict is the instrument working, not a failure — never retry a bench until it turns green, and treat any savings figure as an estimate or a projection until it is reconciled on the savings ledger. See the Bench quickstart and evidence & honesty for the full contract.

Quickstart

Make your first ultrarouter call in five minutes (curl, TypeScript, Python).

The AGENTS.md snippet

Paste this into your repo's AGENTS.md / CLAUDE.md and your coding agent wires ultrarouter itself.

On this page

Which product? Choose when / not whenRouterEvalsScoringFlywheelContextMCP serversText surfacesMachine contractThe minimum to make a callZero to a verdict