Evals & auto-routing

ultrarouter continuously evaluates how models perform on your real workloads and turns that into safe, reversible routing changes.

Feedback and golden sets

• Submit human feedback on any request with POST /v1/requests/:id/feedback — a thumbs up/down, tags, and an optional comment, scoped to the key's workspace.
• Sampled requests form golden sets per call type (with optional PII redaction), the basis for offline evaluation.

What the evaluator's objective grades

Every evaluator suite has one objective — the axis a candidate is graded on and the axis a screening or migration verdict reports lift on. The suite records which axis it chose so a verdict always says what it graded.

• When a call type's golden references come from your own declared corrections — a review you accepted unchanged, or one you edited to the right answer — the objective is agreement with those references: the label your corrections express. For a label-shaped call type (a classification label, a short enum, a compact single-answer object) that agreement is an exact or formatting-insensitive match through the same scorer family every suite uses. Agreement is measured on exactly the label field(s) your objective names, read from the contract or your learning plan — never guessed from the reference. The field comes from one of two places, and the axis records which (its labelFieldOrigin):
  • plan_confirmed — you confirmed the label field once in your learning plan. Discovery proposes it structurally (never by an LLM): among your contract's scalar leaf fields, the one your declared corrections change most often (edited finals vs what was served), with a field literally named label as the tie-break, and it carries its evidence (a count per candidate field). You confirm or edit that one field in the same confirm step. This is what makes the feature plug-and-play for a contract whose label field is named intent, category or decision — the product finds it and you approve it once. Changing the confirmed field starts a new evaluator epoch.
  • schema_convention — no plan field is confirmed but your contract has a property literally named label (for example the label of a {situation, label} response); that field is used.
  • null — neither is present. With more than one field in the reference the objective fails closed to comparing the whole reference (it never guesses a field); the axis records that it compared none and why. So an incidental field the objective does not name as the label — a situation or a notes a model phrases differently on every request — never counts against a right answer. An item whose reference carries no value at the declared label is treated as not scorable and excluded from the agreement rate — it is never counted as a match, so a missing label can never inflate accuracy; a label present in the reference but absent from a candidate's output is a plain disagreement.
• A declared JSON-schema or tool contract stays a hard gate on top of the objective: an output that violates the schema or calls a tool wrongly always fails, regardless of the objective. The contract is never itself the objective — a candidate that is merely well-formed does not clear a run.
• A golden set that carries no declared-correction references keeps the standard composite objective (schema, field and semantic scorers weighted per call-type family).

This is why two candidates that both emit valid JSON can still be separated: the objective grades the answer, and the schema only gates shape.

Evaluation runs

An eval run scores candidate models for a call type against its golden set using a judge cascade. The results back routing recommendations — eval-supported suggestions to migrate a call type from its current model to a cheaper or better one. You can review recommendations via GET /v1/call-types/:id/recommendations or in the dashboard.

Cost-per-task evidence

Before any savings number, ultrarouter states what evidence backs it. A recommendation's cost gate and its projected savings rank on provider-reported cost-per-task — the actual per-item token consumption each candidate used on your golden set — never on per-token list price. Token consumption varies between models on identical work: a model that is cheaper per token can consume enough extra tokens to cost more per task. Each candidate's provider-reported token consumption is priced at your net-effective rate (your rate card when one applies, never list) and compared against the incumbent's on the same items, so a "cheaper per token but hungrier" candidate never masquerades as savings.

Each recommendation carries an evidence cost basis: the provider-reported, replayable measured basis above, or the explicitly labeled blended_price per-token fallback used only when replayable usage was unavailable. The measured label is never applied to an estimate. On the replayable basis the evidence carries both sides' $/task and the token_efficiency_ratio (candidate tokens-per-task ÷ incumbent tokens-per-task); the same figures appear on the dashboard candidate leaderboard.

Projected monthly savings is an estimate, not realized cash. It is your call type's trailing-30-day call volume × the provider-reported per-task net-cost delta, shown with its confidence interval; realized savings appear on your ledger only after you migrate.

Verdicts, power, and zero-judge

A migration decision is a tri-state verdict on the frozen population, never a single score:

• pass (migrate) — both a quality non-inferiority interval and a cost-savings interval clear their pre-registered bars.
• fail (keep) — an established regression, a hard-scorer break, or a per-unit cost ceiling breached (the ceiling fails independently of quality).
• inconclusive — anything else; the evidence does not decide either way.

A keep or inconclusive outcome is first-class and carries a named remediation (e.g. capture more traffic, register an outcome). It is the instrument telling you the truth — never a failure to retry until it turns green.

Before spending, a run publishes a power advisory. underpowered is true when the planning MDE (mde_pp) exceeds the registered margin (margin_pp) at the dataset's independent-cluster count (cluster_n); MDE is a power diagnostic, never the verdict — an underpowered dataset yields an inconclusive, never a false pass.

When deterministic scorers — closed-schema exact-label checks and your own CI assertions — cover every applicable risk, the LLM judge is retired entirely: a zero-judge run, surfaced as judge_disabled: true (with judge_worst_case_usd: 0). Otherwise the composite trust_class (A / B / C) discloses how judge-dependent the quality signal was. The full vocabulary is in Bench evidence & honesty.

Canary rollouts

Approving a recommendation does not flip traffic all at once. A canary splits traffic between the current and candidate model, monitors guardrail metrics, and advances in stages. If a guardrail dips, the canary rolls back automatically; otherwise it promotes the candidate. A one-click manual rollback and a kill switch are always available.

Commit-aware routing

When you have provider spend commitments, routing is commit-aware: the marginal ranking accounts for committed capacity so recommendations and auto-routing help you burn down (not overshoot) a commitment, and renewal packs summarize utilization.

The auto-routing engine is contract-stable today (router: "auto" returns engine: "not_enabled" where selection is not yet active for a workspace) and rolls out per workspace behind the canary machinery.