Get started

Set up ultrarouter learning in this project. You are Claude Code.

ultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.

This setup
- Recipe: Add learning to an existing app (existing-app, revision 4, schema ur.setup-recipes/1). Machine-readable copy: http://localhost:3002/setup-recipes.json
- Project language: TypeScript. If the project keeps agent instructions in CLAUDE.md, add a short note there on where the gateway is configured and where outcomes are reported.
- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.
- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: http://localhost:3002/docs/evidence-status.md

Rules
- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.
- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.
- Claude Code may ask me before it runs a shell command. When you request one that calls the gateway, say why it is needed.
- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.
- Never claim that learning improved anything. Report what the receipt and the learning status say.

Steps
1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Find where the client's base URL, key and model are configured, and how a finished task is judged today: that judgment is the outcome signal to reuse.
2. Read the current ultrarouter docs before you change anything:
   - Get started: http://localhost:3002/docs/get-started.md
   - Record outcomes: http://localhost:3002/docs/guides/outcomes.md
   - Evidence status: http://localhost:3002/docs/evidence-status.md
   - Set up with Claude Code or Codex: http://localhost:3002/docs/guides/agent-integrations.md
   - Setup recipes (JSON): http://localhost:3002/setup-recipes.json
3. Check that the local gateway answers.
   Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.
   Commands:
     curl -sS http://localhost:8080/readyz
     pnpm dev:e2e   (in the ultrarouter checkout)
   Expected output (illustrative; ids differ):
     {"status":"ready","service":"gateway","checks":{…}}
4. Create or sign in to the account.
   Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=<email>. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces/<workspace id>/self-learning with {"enabled":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces/<workspace id>/billing/funding with {"mode":"byok","allowPlatformFallback":false,"requestKey":"<16 to 128 letters, digits, - or _>"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.
   This step is mine: ask me to do it, then wait until I confirm it is done.
   Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: http://localhost:3002/docs/guides/providers-byok.md | Security and data handling: http://localhost:3002/docs/security.md
5. Put the endpoint and the key in the environment.
   read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.
   This step is mine: ask me to do it, then wait until I confirm it is done.
   Commands:
     export UR_BASE_URL=http://localhost:8080
     read -rs UR_API_KEY && export UR_API_KEY
     claude --continue
     test -n "$UR_API_KEY" && echo "UR_API_KEY is set"
   Expected output (illustrative; ids differ):
     UR_API_KEY is set
   Docs: Set up with Claude Code or Codex: http://localhost:3002/docs/guides/agent-integrations.md
6. Install the TypeScript SDK from a local build (optional).
   Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. @ultrarouter/sdk is not on the npm registry, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the tarball with the project's package manager (pnpm add and yarn add take the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.
   Commands:
     git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-typescript ../ultrarouter-sdk-typescript
     npm ci && npm run build && npm pack   (in ../ultrarouter-sdk-typescript)
     npm install ../ultrarouter-sdk-typescript/ultrarouter-sdk-0.1.0.tgz
   Expected output (illustrative; ids differ):
     ultrarouter-sdk-0.1.0.tgz   (the last line npm pack prints)
   Docs: https://github.com/lucastonon/ultrarouter-sdk-typescript | TypeScript SDK: http://localhost:3002/docs/guides/sdk-typescript.md
7. Point the existing client at the gateway.
   Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes "$UR_BASE_URL/v1", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, .withResponse() exposes the headers.
   your existing client module:
   ```ts
   import OpenAI from "openai";

   // Only the configuration changes: base URL, key and the call-type header.
   const client = new OpenAI({
     baseURL: `${process.env.UR_BASE_URL}/v1`,
     apiKey: process.env.UR_API_KEY,
     defaultHeaders: { "X-UR-Call-Type": "my-agent" },
   });

   // Every model call of one task carries the same task id; keep the last request id.
   const { data: completion, response } = await client.chat.completions
     .create({ model, messages }, { headers: { "X-UR-Artifact": taskId } })
     .withResponse();
   const lastRequestId = response.headers.get("x-ur-request-id");

   // When the task is finished, report your verifier's verdict once.
   await fetch(`${process.env.UR_BASE_URL}/v1/requests/${lastRequestId}/outcomes`, {
     method: "POST",
     headers: {
       Authorization: `Bearer ${process.env.UR_API_KEY}`,
       "Content-Type": "application/json",
     },
     body: JSON.stringify({
       channel: "computable",
       kind: passed ? "oracle_pass" : "oracle_fail",
       source_event_id: taskId,
     }),
   });
   ```
   Docs: Quickstart: http://localhost:3002/docs/quickstart.md
8. Report a trusted outcome for each finished task.
   Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.
   Commands:
     curl -sS -X POST "$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes" -H "Authorization: Bearer $UR_API_KEY" -H "Content-Type: application/json" -d '{"channel":"computable","kind":"oracle_pass","source_event_id":"<task id>"}'
   Expected output (illustrative; ids differ):
     HTTP 202 with "status" accepted, duplicate or pending, and a "receipt" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.
   Docs: Record outcomes: http://localhost:3002/docs/guides/outcomes.md
9. Install the ultrarouter MCP servers (optional).
   Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.
   Ask me before you run it.
   Commands:
     pnpm --filter @ultrarouter/mcp build   (in the ultrarouter checkout)
     (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz)   (in the ultrarouter checkout)
     pnpm --filter "@ultrarouter/docs-mcp..." build   (in the ultrarouter checkout)
     ultrarouter-mcp install --host claude --with-docs --docs-repo "$PWD"   (in the ultrarouter checkout)
     claude mcp get ultrarouter
     ultrarouter-mcp check
   Expected output (illustrative; ids differ):
     Connected to ultrarouter at http://localhost:8080.
     Workspace id: …   (from ultrarouter-mcp check)
   Docs: MCP servers for Claude Code and Codex: http://localhost:3002/docs/guides/mcp-servers.md
10. Run one verified episode.
   Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.
11. Confirm the outcome receipt and the learning state.
   The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.
   Commands:
     curl -sS "$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts" -H "Authorization: Bearer $UR_API_KEY"
     curl -sS "$UR_BASE_URL/v1/call-types" -H "Authorization: Bearer $UR_API_KEY" | jq -r '.data[] | select(.slug == "my-agent") | .id'
     curl -sS "$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning" -H "Authorization: Bearer $UR_API_KEY"
   Expected output (illustrative; ids differ):
     {"object":"list","data":[{"object":"outcome_receipt","id":"ocr_…","status":"applied",…}]}
     ct_…
     {"object":"learning","call_type_id":"ct_…","state":"off",…}
   Docs: Evidence status: http://localhost:3002/docs/evidence-status.md
12. Turn on learning for the call type.
   Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces/<workspace id>/self-learning with {"enabled":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode "evaluate", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.
   Ask me before you run it.
   Commands:
     curl -sS -X POST "$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable" -H "Authorization: Bearer $UR_API_KEY" -H "Content-Type: application/json" -d '{"reflection_model":"<a model the workspace owns>","promotion":{"mode":"evaluate"}}'
   Expected output (illustrative; ids differ):
     {"object":"learning_control","action":"enable","changed":true,…,"learning":{"state":"collecting",…}}
   Docs: Provider keys: http://localhost:3002/docs/guides/providers-byok.md
13. Pause, disable or roll back.
   These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.
   Ask me before you run it.
   Commands:
     curl -sS -X POST "$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause" -H "Authorization: Bearer $UR_API_KEY"
     curl -sS -X POST "$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume" -H "Authorization: Bearer $UR_API_KEY"
     curl -sS -X POST "$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable" -H "Authorization: Bearer $UR_API_KEY"
     curl -sS "$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions" -H "Authorization: Bearer $UR_API_KEY"
     curl -sS -X POST "$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback" -H "Authorization: Bearer $UR_API_KEY" -H "Content-Type: application/json" -d '{"version_id":"<an earlier version id>"}'
14. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host claude --with-docs (or the plugin with claude plugin uninstall ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.