Capture your first reviewer correction

You have an app that produces model output a human reviews before it ships — a support reply, a drafted email, an extraction someone signs off. Every time a reviewer edits that output, they are telling you what "good" looks like for your workflow. This M1 quickstart captures that signal: you send a request, record the reviewer's correction, and see the correction joined to the original output in the outcome ledger — over the raw REST API and the TypeScript SDK, side by side.

It is bounded on purpose. The correction, artifact and import steps below make zero paid model calls — they move metadata, not prompts. The only request that costs anything is the one output you choose to have reviewed, and against a local pnpm dev:e2e stack even that is free.

What you get — and what you do not yet

M1 gives you a durable, joined record of human corrections:

• A request joined to an artifact — the model output and the reviewer's verdict hang off the same content-addressed unit of value, so a correction is never an orphaned thumbs-down.
• The correction itself, channel 2 — the human-approved final, recorded with a character-level diff (edit ratio, changed spans, lengths, a content hash). Without corrections_consent the text never leaves your process; only those stats and hashes are stored.
• Signal health and costs — the reviewer integration's named states and the four separated Flywheel usage quantities, so you can see what is being captured and what it costs before anything is billed.
• History import — bring eligible past corrections in one call, with a repairable receipt for any row that does not parse.

Prerequisites

• The stack running. Locally, pnpm dev:e2e boots the gateway on http://localhost:8080. These endpoints are part of the learning preview, which is not on the invite-only hosted gateway (https://api.ultrarouter.io) yet — run the local stack or point at your own deployment.
• A key and base URL. Set them once:

export UR_BASE_URL="http://localhost:8080"
export UR_API_KEY="ur_sk_live_…"   # ur_sk_live_* or ur_sk_test_*

• The outcomes:write scope on that key — every artifact, correction and import step needs it. New keys are granted it by default; confirm yours in step 1. Use a ur_sk_test_* key to rehearse: test-mode data is captured but never enters learning, invoicing or your allowance (doc 17 §4).
• Consent, only where a body is stored. Recording a correction as diff stats needs no consent. Storing the corrected text (or importing intact request/response traces) is gated on corrections_consent — see Consent and cost controls.
• Time and cost. About five to ten minutes. No per-correction fee, and the bounded fixtures below cost nothing — a metadata-only correction and a usage-only import make no model calls.

1. Confirm your setup

GET /v1/key describes the key itself — scopes, caps and usage this period — so a 200 proves your base URL and key work end to end. Check that the returned scopes include outcomes:write before you go on.

curl -sS --fail "$UR_BASE_URL/v1/key" \
  -H "Authorization: Bearer $UR_API_KEY" > /dev/null
echo "ok: key introspection reachable — confirm 'scopes' lists outcomes:write"

2. Send a request, then join it to an artifact

First send the request whose output a reviewer will correct, and capture its ur.request_id. Then create the artifact — the durable unit of value — and declare that this request assembled it. The correction in step 3 attaches to this artifact, so the request and the reviewer's verdict end up on the same row.

# 1. Send the request. The `ur` block on the response carries the request id.
curl -sS "$UR_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5-mini",
    "messages": [{ "role": "user", "content": "Draft a reply to support ticket 4817." }]
  }'
# → response.ur.request_id, e.g. "req_01J…"; response.choices[0].message.content is the draft.

# 2. Join that request to an artifact keyed to your workflow.
curl -sS "$UR_BASE_URL/v1/artifacts" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "support-reply:ticket-4817",
    "archetype": "draft",
    "request_ids": ["req_01J…"]
  }'
# → { "artifact": { "id": "art_01J…", "current_version": 1, "request_ids": ["req_01J…"], … } }

import UltraRouter, { diffStats } from "@ultrarouter/sdk";

const ur = new UltraRouter({
  apiKey: process.env.UR_API_KEY,
  baseURL: process.env.UR_BASE_URL, // http://localhost:8080 locally
});

// 1. Send the request whose output a human will review.
const draft = await ur.chat.completions.create({
  model: "openai/gpt-5-mini",
  messages: [{ role: "user", content: "Draft a reply to support ticket 4817." }],
});
const draftText = draft.choices[0]?.message?.content ?? "";

// 2. Join that request to an artifact — the unit of value the correction attaches to.
const artifact = await ur.ur.artifacts.create({
  key: "support-reply:ticket-4817",
  archetype: "draft",
  request_ids: [draft.ur.request_id],
});

Expected state. A 201 with an artifact object: an id, current_version 1 (the initial draft version — no correction recorded yet), and your request_ids echoed back. An artifact belongs to at most one request-group — re-declaring a request that already belongs elsewhere is a 409, not a silent move.

3. Record the reviewer's correction

The reviewer edits the draft. Record the approved output as the artifact's final (channel 2 — the human's verdict). Two paths, and the page is explicit about which one stores what:

• Metadata-only (no consent required). Send client-computed diff_stats + final_hash. The corrected text never leaves your process — only the edit ratio, changed-span count, lengths and content hash are stored. This is the default.
• Body stored (needs corrections_consent). Send draft + final; the gateway stores the corrected text in the consent-gated blob store and computes the diff server-side. You need this only when you later want to learn from the exact text (M2).

# Metadata-only correction: never sends the text, only the diff stats + hash.
curl -sS "$UR_BASE_URL/v1/artifacts/art_01J…/final" \
  -H "Authorization: Bearer $UR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "final_hash": "3b1f…e9",
    "diff_stats": {
      "edit_ratio": 0.08,
      "changed_spans": 1,
      "lengths": { "draft": 220, "final": 214 }
    }
  }'
# → { "object": "artifact_version", "version": 2, "stored": "metadata_only", "client_computed": true, … }

// The reviewer's edit. `diffStats` is pure and local — it computes the edit ratio,
// changed spans, lengths and a SHA-256 of the final without the text leaving the process.
const finalText = draftText.replace("Regards,", "Warm regards,");
const stats = diffStats(draftText, finalText);

// Metadata-only path (no consent): send the stats + hash, never the body.
const finalVersion = await ur.ur.artifacts.final(artifact.id, {
  final_hash: stats.final_hash,
  diff_stats: stats,
});
// finalVersion.stored === "metadata_only"; finalVersion.client_computed === true.

// Consent-on path instead: pass the bodies and let the gateway store + diff them.
//   await ur.ur.artifacts.final(artifact.id, { draft: draftText, final: finalText });

Expected state. A 200 with object: "artifact_version", an incremented version, and stored reading metadata_only (no consent) or blob (consent on, body sent). The provenance — source, producer, learning_use — is assigned by the server from your credential; a caller-supplied value is a 400, never a silent strip.

4. See the correction joined to the original output

Read the artifact back. The request you sent in step 2 and the correction from step 3 are on the same row: request_ids still lists the original request, and current_version has moved to the corrected version.

curl -sS "$UR_BASE_URL/v1/artifacts/art_01J…" \
  -H "Authorization: Bearer $UR_API_KEY"
# → { "artifact": { "id": "art_01J…", "current_version": 2,
#       "request_ids": ["req_01J…"], "archetype": "draft", … } }

const joined = await ur.ur.artifacts.get(artifact.id);
console.log(joined.request_ids);     // includes draft.ur.request_id — the original request
console.log(joined.current_version); // the corrected version from step 3

Run the whole loop against your dev stack in one shot — this creates an artifact, records a correction on it, and reads back the joined version:

# A bounded, self-contained correction: create an artifact, record a reviewer's
# correction as its final version, and read the JOINED result. Zero model calls.
key="correction-quickstart:$(date +%s)-$RANDOM"
id=$(curl -sS --fail "$UR_BASE_URL/v1/artifacts" \
  -H "Authorization: Bearer $UR_API_KEY" -H "Content-Type: application/json" \
  -d "{\"key\":\"$key\",\"archetype\":\"draft\"}" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["artifact"]["id"])')

ver=$(curl -sS --fail "$UR_BASE_URL/v1/artifacts/$id/final" \
  -H "Authorization: Bearer $UR_API_KEY" -H "Content-Type: application/json" \
  -d '{"final_hash":"3b1fe9","diff_stats":{"edit_ratio":0.08,"changed_spans":1,"lengths":{"draft":220,"final":214}}}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["version"])')

curl -sS --fail "$UR_BASE_URL/v1/artifacts/$id" \
  -H "Authorization: Bearer $UR_API_KEY" \
  | python3 -c "import sys,json; a=json.load(sys.stdin)['artifact']; assert a['current_version']==$ver, a; print('ok: correction joined at version', a['current_version'])"

The same loop over fetch — the TypeScript equivalent, executed against your stack in CI, so the create → final → joined-read path never rots silently:

// The same bounded correction loop as above, over the raw HTTP the SDK wraps: create an
// artifact, record the reviewer's correction as its final version, and read back the
// JOINED result. Only fetch + process, zero model calls — it runs hermetically.
const base = process.env.UR_BASE_URL;
const auth = {
  Authorization: `Bearer ${process.env.UR_API_KEY}`,
  "Content-Type": "application/json",
};

const created = await fetch(`${base}/v1/artifacts`, {
  method: "POST",
  headers: auth,
  body: JSON.stringify({
    key: `correction-quickstart-ts:${Date.now()}-${Math.random()}`,
    archetype: "draft",
  }),
});
if (!created.ok) {
  console.error("create HTTP", created.status);
  process.exit(1);
}
const artifactId = (await created.json()).artifact.id;

const recorded = await fetch(`${base}/v1/artifacts/${artifactId}/final`, {
  method: "POST",
  headers: auth,
  body: JSON.stringify({
    final_hash: "3b1fe9",
    diff_stats: { edit_ratio: 0.08, changed_spans: 1, lengths: { draft: 220, final: 214 } },
  }),
});
if (!recorded.ok) {
  console.error("final HTTP", recorded.status);
  process.exit(1);
}
const version = (await recorded.json()).version;

const read = await fetch(`${base}/v1/artifacts/${artifactId}`, { headers: auth });
const joined = (await read.json()).artifact;
if (joined.current_version !== version) {
  console.error("version mismatch", joined);
  process.exit(1);
}
console.log(`ok: correction joined at version ${joined.current_version}`);

Automating it: the reviewer adapter

Wiring a review queue by hand is fine for one correction; for a stream of them, the reviewer adapter is the one supported integration. GET /v1/flywheel/reviewer/setup returns the decision → (channel, kind) mapping and the HMAC signature recipe, and POST /v1/flywheel/reviewer/test runs a guided, joined test end to end without touching your data. In TypeScript the client signs and durably delivers each decision for you:

await ur.ur.reviewer.setup();            // mapping preview + signature recipe (no secret echoed)
await ur.ur.reviewer.test("happy");      // a guided, joined test pair
await ur.ur.reviewer.record({            // one reviewer decision, durably queued + signed
  event_id: "rev-4817",
  artifact_key: "support-reply:ticket-4817",
  artifact_version: 1,
  decision: "approved_with_edits",
  effective_at: new Date().toISOString(),
  final: { hash: "3b1fe9" },
});
const health = await ur.ur.reviewer.health(); // the six named integration states

Only approved_with_edits / approved accept a final; rejected and regenerated are recorded as decisions without one. Every reviewer decision maps to the correction channel — the adapter reuses the same ledger service as the API above, so there is no second ingestion path.

Tool-calling agents: correct the next tool call

If your workflow is an agent — the request declares tools and the model answers with a tool call rather than text — the reviewer's correction is the right next tool call, not a rewritten paragraph. Send that corrected tool call as the artifact's final under corrections_consent, either as a bare { "name": …, "arguments": … } object or as a full OpenAI completion body carrying one tool_calls entry:

# The agent turn declared `tools`; the served answer was a tool call. The reviewer's
# corrected NEXT TOOL CALL becomes the artifact's final (needs corrections_consent to store).
curl -sS "$UR_BASE_URL/v1/artifacts/art_01J…/final" \
  -H "Authorization: Bearer $UR_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "draft":  "{\"name\":\"get_order_status\",\"arguments\":{\"order_id\":\"WRONG\"}}",
    "final":  "{\"name\":\"get_order_status\",\"arguments\":{\"order_id\":\"ORD-4817\",\"include_history\":true}}"
  }'
# → { "object": "artifact_version", "version": 2, "stored": "blob", … }

An accepted_unchanged correction (the served tool call was already right) records the served call as the reference; an edited correction records your corrected call. Once enough distinct corrected tool calls are captured for a call type, the product compiles a tool-call evaluator for it: a deterministic scorer that grades whether a candidate calls the right tool with the right arguments (the tool name must match; arguments are compared on the fields your correction sets), with the declared tool contract as a hard gate. This is what lets an agent workflow qualify an evaluator at all — a tool-calling turn has no text label for the text-shaped path to grade.

Corrections that are neither a clean single tool call nor clean text (a hybrid answer, or more than one tool call in a turn) are excluded and counted, never guessed at.

5. Bring history, and repair a failed import row

You do not have to wait for new traffic — import eligible history in one call. A usage-only import (usage_summary) gives per-model exposure and spend and needs no consent; intact traces (with request/response bodies and prior review outcomes) are gated on corrections_consent and redacted on ingest. Ingest is idempotent per record content hash, and a malformed row is rejected with a named reason — never guessed.

# Import a two-row usage history where ONE row is malformed (no `model`). The good row is
# accepted; the bad one is not — and the receipt names why. Nothing is guessed.
imp=$(curl -sS --fail "$UR_BASE_URL/v1/imports" \
  -H "Authorization: Bearer $UR_API_KEY" -H "Content-Type: application/json" \
  -d '{"format":"usage_summary","records":[
        {"model":"openai/gpt-4o","requests":12,"input_tokens":4800,"output_tokens":960,"cost_usd":0.21},
        {"requests":3,"input_tokens":900,"output_tokens":120,"cost_usd":0.04}]}')
id=$(printf '%s' "$imp" | python3 -c 'import sys,json; d=json.load(sys.stdin)["import"]; assert d["accepted_count"]==1 and d["record_count"]==2, d; print(d["id"])')
echo "ok: import $id accepted 1/2 rows; the malformed row is named in report.reasons"

# REPAIR: fix the row and re-send with the SAME import_id. Idempotent per record content
# hash, so the good row dedupes and only the repaired row lands — never double-counted.
curl -sS --fail "$UR_BASE_URL/v1/imports" \
  -H "Authorization: Bearer $UR_API_KEY" -H "Content-Type: application/json" \
  -d "{\"format\":\"usage_summary\",\"import_id\":\"$id\",\"records\":[
        {\"model\":\"openai/gpt-4o\",\"requests\":12,\"input_tokens\":4800,\"output_tokens\":960,\"cost_usd\":0.21},
        {\"model\":\"openai/gpt-4o-mini\",\"requests\":3,\"input_tokens\":900,\"output_tokens\":120,\"cost_usd\":0.04}]}" \
  | python3 -c 'import sys,json; d=json.load(sys.stdin)["import"]; assert d["accepted_count"]>=1 and d["duplicate_count"]>=1, d; print("ok: repaired row accepted; the already-good row deduped")'

Expected state. The first call returns accepted_count: 1, the other row counted under rejected_count / unsupported_count, and report.reasons maps the reason (a usage row requires a model) to a count. The repair call, resumed with the same import_id, returns the repaired row as accepted and the unchanged row as duplicate. The dashboard Flywheel screen shows the same receipt: each rejected row's state is named, and a reviewer event that could not be joined (a mismatched draft version) is a first-class dead letter you re-drive with POST /v1/flywheel/reviewer/repair — a repair re-runs the stored event, it never rewrites a recorded consequence.

Consent and cost controls

Consent is a gate on stored bodies, not on capture. Recording corrections as diff stats, and usage-only imports, work with no consent at all. corrections_consent (owner-only, audited) is required only to store the corrected text or to import intact traces; revoking it stops new derivations and marks existing exports — it never rewrites what is already recorded. A separate training_export_consent governs the customer-owned export; neither defaults on.

Capture is not billed per event. Artifact and outcome counts are operational counters, not charges — there is no per-correction fee. What is billable is bounded analysis and approved evaluation compute (an M2 concern), reserved and settled through the existing budget engine, with failed attempts counted in actual cost and retries never double-charged. Test-mode keys are excluded from all of it. GET /v1/flywheel/status shows the four separated quantities — tracked artifacts, accepted outcomes, analysis units, evaluation compute — plus storage usage against your quota (which pauses overage rather than surprising you) and the honest learning block:

curl -sS "$UR_BASE_URL/v1/flywheel/status" \
  -H "Authorization: Bearer $UR_API_KEY"
# → { …, "learning": { "flywheel_class": "unavailable", "flywheel_class_reason": "collecting",
#       "active_levers": [], "calibration_available": false } }

That unavailable / collecting is the honest M1 state: signal is being captured; no lift is claimed, because none has been established.

Troubleshooting

What this proves — and what it does not

Proves. You can go from a served request to a durable, joined record of a human reviewer's correction — over REST or the SDK — with the corrected text kept in your process unless you consent to store it, past history imported and any failed row repairable, and the capture counted separately from anything billable. No model call is needed for the correction itself, and there is no per-event fee.

Does not prove. It does not prove any correction has improved your app. A captured correction is a joined observation, not a learned consequence: at M1 nothing is trained, no lever is active, and the ledger's learning block says so. Turning these corrections into learned context, testing them through your real application, and confirming a change with a controlled live comparison is M2 — the application-CI and live-confirmation quickstart — and until that evidence exists, no quality or cost-per-task claim is made here.

Next steps

• Register what "it worked" means for a type of artifact — the one thing the product refuses to infer.
• Grant corrections_consent and set retention in the dashboard Settings if you want to store the corrected text for later learning.
• The application-CI and live-confirmation quickstart (M2) covers turning these corrections into a controlled comparison, and is published as it ships.