Plan a model migration

A provider is retiring a model you depend on, or you want to compare a cheaper candidate before you commit. This M0 quickstart takes you from that starting point to a scoped decision you can act on — using only shipped surfaces: the raw /v1/imports REST API and the dashboard Migrations workspace.

It is bounded on purpose. The example below imports a tiny usage export, so it runs in a few minutes and makes zero paid model calls. It also tells you plainly where the current build stops short of a numeric verdict, so you know exactly which state you are looking at and why.

What you get — and what you do not yet

M0 gives you, with no annotations and no change to how your traffic is served:

• an exposure report — which models you send traffic to, your historical spend by model, and which of those models the manually maintained deprecation registry flags as retiring;
• a guided scoping flow — pick the affected workflows, set your quality and cost limits, choose two to four candidate models, preview data coverage and the estimated test spend, and authorize a bounded test;
• a decision pack — a content-addressed record of the decision contract, the exact incumbent → candidate configuration diff, and a rollout checklist.

What M0 does not give you from a usage-only import:

Prerequisites

• The stack running. Locally, pnpm dev:e2e boots the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. These endpoints are part of the learning preview, which is not on the invite-only hosted gateway (https://api.ultrarouter.io) yet — run the local stack or point at your own deployment.
• A key and base URL. Set them once:

export UR_BASE_URL="http://localhost:8080"
export UR_API_KEY="ur_sk_live_…"   # ur_sk_live_* or ur_sk_test_*

• The outcomes:write scope on that key for the import step (below). New keys are granted it by default; check yours with GET /v1/key (step 1). Keys created with an explicit scope list may omit it — mint or roll a key in the dashboard if so.
• The Migrations workspace enabled. It is behind the dashboard.migrations feature flag (an unknown flag reads off, and the /migrations route returns 404 until it is on). Enable it for your workspace before step 3.
• Time and cost. About five to ten minutes. The bounded usage-only fixture costs nothing — a usage import replays no prompts and makes no model calls.

1. Confirm your setup

GET /v1/key describes the key itself — its scopes, caps and usage this period — so a 200 proves your base URL and key work end to end. Check that the returned scopes include outcomes:write before the import step.

curl -sS --fail "$UR_BASE_URL/v1/key" \
  -H "Authorization: Bearer $UR_API_KEY"
echo
echo "ok: key introspection reachable — confirm 'scopes' lists outcomes:write above"

2. Import your usage history

Send a small usage export to POST /v1/imports with the usage_summary adapter. Each row is a per-model spend summary — model, requests, input_tokens, output_tokens and cost_usd (or cost_micro_usd). Use your real export, or this three-line fixture that includes a retiring model (openai/gpt-4-0613):

curl -sS "$UR_BASE_URL/v1/imports" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "format": "usage_summary",
    "jsonl": "{\"model\":\"openai/gpt-4-0613\",\"requests\":1200,\"input_tokens\":480000,\"output_tokens\":96000,\"cost_usd\":18.40}\n{\"model\":\"openai/gpt-4o-2024-05-13\",\"requests\":3400,\"input_tokens\":1360000,\"output_tokens\":204000,\"cost_usd\":11.05}\n{\"model\":\"anthropic/claude-3-sonnet-20240229\",\"requests\":800,\"input_tokens\":320000,\"output_tokens\":64000,\"cost_usd\":4.72}"
  }'

Expected state. A 200 with an import object whose import_class is "usage_only", status is "completed", and report carries a per-model usage_exposure array plus a total spend_micro_usd. The report.class_truth sentence states, in plain language, what this import can and cannot prove.

Usage-only vs. full-trace imports — the distinction that matters

The two import classes are visibly different, and the difference is the point:

• Usage-only (usage_summary) reveals model usage, spend and missing fields only. It can never manufacture prompts, outputs, quality results or replayable cases — so its report is forced to replayable: false, and no quality figure can ever be attached to it, regardless of what a file claims.
• Full traces (openai_chat_jsonl, an intact-trace adapter) additionally carry request/response bodies. Those are stored only under corrections_consent, redacted on ingest — and only then can candidate models be tested against your real cases. Without consent, an intact-trace commit is refused; a preview still returns the capability report without storing a body.

You cannot get a quality comparison out of a usage-only export. If that is all you have, M0 still gives you exposure, spend and the retirement picture — enough to scope which workflows need attention — but the verdict will stay "Needs more evidence" because there is nothing to replay.

3. Read your exposure report

Open the Migrations workspace in the dashboard and select your import:

http://localhost:3000/migrations?import=<import id>

Expected state. The Model exposure card renders:

• a class-truth banner — for the fixture, the usage-only sentence ("it shows model usage, spend and missing fields; it cannot manufacture prompts, quality results or replayable cases");
• an explicit "No quality results are available from this import" line (usage-only);
• a per-model table: model id, requests, spend, and a Retiring badge with the shutdown date for any model matched in the deprecation registry — openai/gpt-4-0613 is flagged retiring in the fixture;
• your total spend, and any raw fields the import could not consume under Missing fields.

The Deprecation registry card below lists the curated retirements with their exact model id, shutdown date, retrieved date and provider source link. It is a manually maintained, dated snapshot — not a live monitoring service — so verify against the linked source before you rely on a date. Aliases you may still be sending are labeled verified or unverified.

4. Scope a decision

Use the Plan a migration wizard on the same screen. It walks five steps:

1. Workflows — the affected call types (one is enough to start; at most three per offer).
2. Limits — your quality tolerance (in pp), an optional cost ceiling, the test budget, and your goal (save money, required migration, or improve quality).
3. Candidates — two to four candidate models to compare.
4. Coverage — a preview of replayable cases per workflow and the estimated test spend (labeled Estimate).
5. Authorize — reserve the bounded test budget and create the assessment.

Expected state. Authorizing reserves the budget (a budget-engine hold) and dispatches the paired run. While the run is in flight the assessment shows a plain in-progress state and no verdict area — a verdict is only shown once a run has actually completed, never as an empty placeholder. Cancelling releases the unused reservation; resuming never double-charges settled work.

5. Read the verdict

Open the assessment to see its state and, per workflow, the verdict. Every migration resolves to exactly one of four plain-language headlines:

• Ready for a controlled rollout — a candidate cleared your quality and cost terms.
• Keep the current model — no candidate beats your incumbent on the terms you set.
• Needs more evidence — the evidence is not yet conclusive.
• No acceptable replacement found — every candidate established a quality regression (a first-class refusal, not an error).

If your incumbent is retiring, "Keep the current model" explains that a replacement is still needed before the shutdown date.

How deep the evidence goes

Once the paired run reaches a verdict, that verdict carries an evidence-depth label so you can tell which engine produced it. There are two depths, and the same scoring family the continuous evals product uses backs both:

• Judged — the candidate and the incumbent were re-run on your real cases, scored by the deterministic scorers and the position-swapped two-tier judge cascade (the same cascade the evals funnel runs, gated on a current judge calibration). This is the deeper reading.
• Scorer-only — the deterministic scorers ran, but the judge was skipped because no calibrated judge model was available for the run. The verdict still reflects your quality and cost terms, but it rests on the programmatic signals alone; the label names why the judge was skipped.

A scorer-only verdict is never presented with the same authority as a judged one — the depth is part of the record, so "ready for a controlled rollout" from the scorers alone can never read like the same headline backed by the judge.

What you see, before / during / after a run. Authorizing dispatches the paired run. Before it finishes the page shows a plain in-progress state and no verdict area — never a "Needs more evidence" placeholder that reads like a verdict it can never reach. After the run completes the verdict area appears, led by one of the four headlines, and the headline's treatment depends on the evidence depth above: a judged verdict gets the design-intent headline; a scorer-only verdict gets a visibly weaker treatment that names the scorers that ran and why the judge was skipped. A usage-only import completes straight to "Needs more evidence" — it has nothing to replay — so it never shows a quality figure. Either way you still get the scope, the incumbent → candidate configuration diff (classified as an in-surface model substitution), the content-addressed decision pack, and the rollout checklist. No numeric quality figure is shown before qualified evidence exists — that is the honesty invariant, enforced in the UI.

Applying a decision uses your existing config versions, canary rollout and rollback, and requires an explicit authorized action — never an assessment alone.

Troubleshooting

What this proves — and what it does not

Proves. You can go from a raw usage export to a scoped, honest picture: which models you depend on, what they cost you, which are retiring, and a bounded plan to test replacements — with the usage-only-vs-full-replay boundary made explicit at every step, and zero paid model runs for the bounded fixture.

Does not prove. It does not prove any candidate is better or cheaper. A usage-only import carries no quality signal by construction, so this bounded fixture never reaches a numeric quality or cost-per-task claim — it completes to "Needs more evidence". A full-replay assessment (full traces, with consent) runs the paired dispatch and resolves to one of the four headlines with the supporting sample, interval and inspectable failures; its headline is labeled with the evidence depth so a scorer-only reading is never presented with a judged verdict's authority.

Next steps

• Bring full traces (with consent) to unlock replay — see providers & BYOK for registering the keys a test run uses.
• Understand call types, which define the workflows a migration is scoped to.
• The reviewer-correction and application-CI quickstarts (M1, M2) cover the Flywheel entry point and are published as they ship.