{ "schema_version": "ur.setup-recipes/1", "site_url": "https://docs.ultrarouter.io", "docs": { "get_started": "https://docs.ultrarouter.io/docs/get-started", "evidence_status": "https://docs.ultrarouter.io/docs/evidence-status", "agent_integrations": "https://docs.ultrarouter.io/docs/guides/agent-integrations" }, "availability": { "hosted_gateway": true, "hosted_gateway_url": "https://api.ultrarouter.io", "hosted_access": "invite_only", "hosted_learning": false, "sdk_on_registries": false, "learning": "preview", "learning_available_to_public": false, "mcp_servers_available_to_public": false, "note": "The learning preview and the MCP servers currently run only from the private ultrarouter source on a local stack, so they are not yet available to outside users. The invite-only hosted gateway offers routing and compression; the public SDKs are available. Learning is a preview: whether it improves a given agent is not yet demonstrated; see docs.evidence_status." }, "local_stack": { "hosted": false, "bootCommand": "pnpm dev:e2e", "stopCommand": "pnpm dev:e2e:down", "gatewayUrl": "http://localhost:8080", "dashboardUrl": "http://localhost:3000", "signUpUrl": "http://localhost:3000/signup", "signInUrl": "http://localhost:3000/login", "outboxUrl": "http://localhost:8080/auth/test/outbox?to=", "docsMcpBuild": "pnpm --filter \"@ultrarouter/docs-mcp...\" build", "docsMcpBin": "packages/docs-mcp/dist/bin.js", "docsMcpName": "ultrarouter-docs" }, "sdk_release": { "typescript": { "packageName": "@ultrarouter/sdk", "version": "0.1.0", "repository": "https://github.com/lucastonon/ultrarouter-sdk-typescript", "checkoutDir": "ultrarouter-sdk-typescript", "sourceRef": "main", "learningRef": "feat/learning-loop", "learningRefPublic": true, "published": false, "learningHelpers": true, "artifact": "ultrarouter-sdk-0.1.0.tgz" }, "python": { "distribution": "ultrarouter", "version": "0.1.0", "repository": "https://github.com/lucastonon/ultrarouter-sdk-python", "checkoutDir": "ultrarouter-sdk-python", "sourceRef": "main", "learningRef": "feat/learning-loop", "learningRefPublic": true, "published": false, "learningHelpers": true, "wheel": "ultrarouter-0.1.0-py3-none-any.whl", "sdist": "ultrarouter-0.1.0.tar.gz" } }, "hosts": [ { "id": "claude", "label": "Claude Code", "instructionsFile": "CLAUDE.md", "shellNote": "Claude Code may ask me before it runs a shell command. When you request one that calls the gateway, say why it is needed.", "envNote": "Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues.", "startCommand": "claude", "resumeCommand": "claude --continue", "mcpInstall": "ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\"", "mcpCheck": "claude mcp get ultrarouter", "mcpRemove": "ultrarouter-mcp uninstall --host claude --with-docs", "mcpConfigLocation": "~/.claude.json (user scope, every project)", "pluginInstall": [ "claude plugin marketplace add ./integrations/claude-code", "claude plugin install ultrarouter@ultrarouter-local" ], "pluginRemove": "claude plugin uninstall ultrarouter@ultrarouter-local", "pluginNote": "Use the plugin or the installer, not both: both register an ultrarouter server. The plugin keeps its key in Claude Code secure storage; set it with /plugin configure." }, { "id": "codex", "label": "Codex", "instructionsFile": "AGENTS.md", "shellNote": "Codex runs commands in a sandbox that may block network access, including to localhost. If a call to the gateway fails with a network or sandbox error, ask me to approve it or to allow network access for this session. Never turn the sandbox off.", "envNote": "Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs.", "startCommand": "codex -c shell_environment_policy.ignore_default_excludes=true", "resumeCommand": "codex resume --last -c shell_environment_policy.ignore_default_excludes=true", "mcpInstall": "ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\"", "mcpCheck": "codex mcp get ultrarouter", "mcpRemove": "ultrarouter-mcp uninstall --host codex --with-docs", "mcpConfigLocation": "~/.codex/config.toml", "pluginInstall": [ "codex plugin marketplace add ./integrations/codex", "codex plugin add ultrarouter@ultrarouter-local", "ultrarouter-mcp install --host codex" ], "pluginRemove": "codex plugin remove ultrarouter@ultrarouter-local", "pluginNote": "A Codex plugin cannot carry a secret, so the installer runs too: Codex then uses the installer's entry, which forwards ULTRAROUTER_API_KEY." } ], "languages": [ "typescript", "python" ], "paths": [ "existing-client", "native-sdk", "mcp-plugin" ], "recipes": [ { "id": "existing-app", "revision": 4, "path": "existing-client", "title": "Add learning to an existing app", "summary": "Keep your agent and its OpenAI-compatible client. Point the client at the gateway, report a trusted outcome per finished task, then turn learning on.", "availability": "preview", "languages": [ "typescript", "python" ], "hosts": [ "claude", "codex" ], "inspect": "Find where the client's base URL, key and model are configured, and how a finished task is judged today: that judgment is the outcome signal to reuse.", "env": [ { "name": "UR_BASE_URL", "required": true, "description": "The gateway base URL. On the local stack: http://localhost:8080." }, { "name": "UR_API_KEY", "required": true, "description": "An ultrarouter API key minted in the dashboard (Settings, API keys). Keep it in the environment or a secret manager, never in source control." }, { "name": "UR_CALL_TYPE_ID", "required": false, "description": "The call type's id (ct_…) the learning routes take. The first request that names the call type creates it; then look the id up by slug (the receipt step shows how)." }, { "name": "ULTRAROUTER_API_KEY", "required": false, "description": "The key the management MCP server uses: a separate key with config:read, plus config:write only if the agent may pause, disable or roll back learning. Read from the environment the coding agent was started in; the installer never writes it." }, { "name": "ULTRAROUTER_BASE_URL", "required": false, "description": "The gateway the management MCP server talks to. Defaults to http://localhost:8080." } ], "steps": [ { "id": "check-stack", "kind": "stack", "title": "Check that the local gateway answers", "detail": "Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS http://localhost:8080/readyz" }, { "run": "pnpm dev:e2e", "cwd": "the ultrarouter checkout" } ], "env": [], "links": [ "/docs/get-started" ], "expectedOutput": "{\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}" }, { "id": "account", "kind": "account", "title": "Create or sign in to the account", "detail": "Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.", "actor": "user", "optional": false, "confirmFirst": false, "commands": [], "env": [], "links": [ "http://localhost:3000/signup", "http://localhost:3000/login", "/docs/guides/providers-byok", "/docs/security" ] }, { "id": "environment-claude", "kind": "environment", "title": "Put the endpoint and the key in the environment", "detail": "read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.", "actor": "user", "optional": false, "confirmFirst": false, "hosts": [ "claude" ], "commands": [ { "run": "export UR_BASE_URL=http://localhost:8080" }, { "run": "read -rs UR_API_KEY && export UR_API_KEY" }, { "run": "claude --continue" }, { "run": "test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/agent-integrations" ], "expectedOutput": "UR_API_KEY is set" }, { "id": "environment-codex", "kind": "environment", "title": "Put the endpoint and the key in the environment", "detail": "read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.", "actor": "user", "optional": false, "confirmFirst": false, "hosts": [ "codex" ], "commands": [ { "run": "export UR_BASE_URL=http://localhost:8080" }, { "run": "read -rs UR_API_KEY && export UR_API_KEY" }, { "run": "codex resume --last -c shell_environment_policy.ignore_default_excludes=true" }, { "run": "test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/agent-integrations" ], "expectedOutput": "UR_API_KEY is set" }, { "id": "install-sdk-typescript", "kind": "install-sdk", "title": "Install the TypeScript SDK from a local build", "detail": "Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. @ultrarouter/sdk is not on the npm registry, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the tarball with the project's package manager (pnpm add and yarn add take the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.", "actor": "agent", "optional": true, "confirmFirst": false, "languages": [ "typescript" ], "commands": [ { "run": "git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-typescript ../ultrarouter-sdk-typescript" }, { "run": "npm ci && npm run build && npm pack", "cwd": "../ultrarouter-sdk-typescript" }, { "run": "npm install ../ultrarouter-sdk-typescript/ultrarouter-sdk-0.1.0.tgz" } ], "env": [], "links": [ "https://github.com/lucastonon/ultrarouter-sdk-typescript", "/docs/guides/sdk-typescript" ], "expectedOutput": "ultrarouter-sdk-0.1.0.tgz (the last line npm pack prints)" }, { "id": "install-sdk-python", "kind": "install-sdk", "title": "Install the Python SDK from a local build", "detail": "Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. ultrarouter is not on PyPI, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the wheel into the project's environment (uv pip install takes the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.", "actor": "agent", "optional": true, "confirmFirst": false, "languages": [ "python" ], "commands": [ { "run": "git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-python ../ultrarouter-sdk-python" }, { "run": "uv build", "cwd": "../ultrarouter-sdk-python" }, { "run": "pip install ../ultrarouter-sdk-python/dist/ultrarouter-0.1.0-py3-none-any.whl" } ], "env": [], "links": [ "https://github.com/lucastonon/ultrarouter-sdk-python", "/docs/guides/sdk-python" ], "expectedOutput": "Successfully built dist/ultrarouter-0.1.0.tar.gz\nSuccessfully built dist/ultrarouter-0.1.0-py3-none-any.whl" }, { "id": "connect-typescript", "kind": "connect", "title": "Point the existing client at the gateway", "detail": "Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, .withResponse() exposes the headers.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "typescript" ], "commands": [], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/get-started", "/docs/quickstart" ] }, { "id": "connect-python", "kind": "connect", "title": "Point the existing client at the gateway", "detail": "Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, with_raw_response exposes the headers.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "python" ], "commands": [], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/get-started", "/docs/quickstart" ] }, { "id": "outcome", "kind": "outcome", "title": "Report a trusted outcome for each finished task", "detail": "Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/outcomes", "/docs/get-started" ], "expectedOutput": "HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied." }, { "id": "mcp-servers", "kind": "mcp", "title": "Install the ultrarouter MCP servers", "detail": "Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.", "actor": "agent", "optional": true, "confirmFirst": true, "commands": [ { "run": "pnpm --filter @ultrarouter/mcp build", "cwd": "the ultrarouter checkout" }, { "run": "(cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz)", "cwd": "the ultrarouter checkout" }, { "run": "pnpm --filter \"@ultrarouter/docs-mcp...\" build", "cwd": "the ultrarouter checkout" }, { "run": "ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\"", "cwd": "the ultrarouter checkout", "hosts": [ "claude" ] }, { "run": "claude mcp get ultrarouter", "hosts": [ "claude" ] }, { "run": "ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\"", "cwd": "the ultrarouter checkout", "hosts": [ "codex" ] }, { "run": "codex mcp get ultrarouter", "hosts": [ "codex" ] }, { "run": "ultrarouter-mcp check" } ], "env": [ "ULTRAROUTER_API_KEY", "ULTRAROUTER_BASE_URL" ], "links": [ "/docs/guides/mcp-servers" ], "expectedOutput": "Connected to ultrarouter at http://localhost:8080.\nWorkspace id: … (from ultrarouter-mcp check)" }, { "id": "verify-episode", "kind": "verify", "title": "Run one verified episode", "detail": "Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/get-started" ] }, { "id": "receipt", "kind": "receipt", "title": "Confirm the outcome receipt and the learning state", "detail": "The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"my-agent\") | .id'" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started", "/docs/evidence-status" ], "expectedOutput": "{\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\nct_…\n{\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}" }, { "id": "enable-learning", "kind": "enable", "title": "Turn on learning for the call type", "detail": "Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.", "actor": "agent", "optional": false, "confirmFirst": true, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started", "/docs/guides/providers-byok" ], "expectedOutput": "{\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}" }, { "id": "controls", "kind": "controls", "title": "Pause, disable or roll back", "detail": "These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.", "actor": "agent", "optional": false, "confirmFirst": true, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started" ] } ], "examples": { "typescript": { "kind": "snippet", "filename": "your existing client module", "code": "import OpenAI from \"openai\";\n\n// Only the configuration changes: base URL, key and the call-type header.\nconst client = new OpenAI({\n baseURL: `${process.env.UR_BASE_URL}/v1`,\n apiKey: process.env.UR_API_KEY,\n defaultHeaders: { \"X-UR-Call-Type\": \"my-agent\" },\n});\n\n// Every model call of one task carries the same task id; keep the last request id.\nconst { data: completion, response } = await client.chat.completions\n .create({ model, messages }, { headers: { \"X-UR-Artifact\": taskId } })\n .withResponse();\nconst lastRequestId = response.headers.get(\"x-ur-request-id\");\n\n// When the task is finished, report your verifier's verdict once.\nawait fetch(`${process.env.UR_BASE_URL}/v1/requests/${lastRequestId}/outcomes`, {\n method: \"POST\",\n headers: {\n Authorization: `Bearer ${process.env.UR_API_KEY}`,\n \"Content-Type\": \"application/json\",\n },\n body: JSON.stringify({\n channel: \"computable\",\n kind: passed ? \"oracle_pass\" : \"oracle_fail\",\n source_event_id: taskId,\n }),\n});\n", "run": [], "note": "The lines to change in an existing project. model, messages, taskId and passed come from your own code; passed is your verifier, not a model grading itself." }, "python": { "kind": "snippet", "filename": "your existing client module", "code": "import os\n\nimport httpx\nfrom openai import OpenAI\n\n# Only the configuration changes: base URL, key and the call-type header.\nclient = OpenAI(\n base_url=f\"{os.environ['UR_BASE_URL']}/v1\",\n api_key=os.environ[\"UR_API_KEY\"],\n default_headers={\"X-UR-Call-Type\": \"my-agent\"},\n)\n\n# Every model call of one task carries the same task id; keep the last request id.\nraw = client.chat.completions.with_raw_response.create(\n model=model, messages=messages, extra_headers={\"X-UR-Artifact\": task_id}\n)\nlast_request_id = raw.headers.get(\"x-ur-request-id\")\ncompletion = raw.parse()\n\n# When the task is finished, report your verifier's verdict once.\nhttpx.post(\n f\"{os.environ['UR_BASE_URL']}/v1/requests/{last_request_id}/outcomes\",\n headers={\"Authorization\": f\"Bearer {os.environ['UR_API_KEY']}\"},\n json={\n \"channel\": \"computable\",\n \"kind\": \"oracle_pass\" if passed else \"oracle_fail\",\n \"source_event_id\": task_id,\n },\n).raise_for_status()\n", "run": [], "note": "The lines to change in an existing project. model, messages, taskId and passed come from your own code; passed is your verifier, not a model grading itself." } }, "links": [ "/docs/get-started", "/docs/guides/outcomes", "/docs/evidence-status", "/docs/guides/agent-integrations" ] }, { "id": "fresh-typescript", "revision": 4, "path": "native-sdk", "title": "New TypeScript example", "summary": "A new Node.js project with the native SDK installed from a local build and one runnable toy episode: a request, a verifier, an outcome, its receipt and the learning state.", "availability": "preview", "languages": [ "typescript" ], "hosts": [ "claude", "codex" ], "inspect": "Check that Node.js 20 or newer, npm and git are available. Whatever you find, leave an existing project unchanged: the example goes in a new ultrarouter-learning-demo directory beside it.", "env": [ { "name": "UR_BASE_URL", "required": true, "description": "The gateway base URL. On the local stack: http://localhost:8080." }, { "name": "UR_API_KEY", "required": true, "description": "An ultrarouter API key minted in the dashboard (Settings, API keys). Keep it in the environment or a secret manager, never in source control." }, { "name": "UR_MODEL", "required": false, "description": "The model id the demo calls. A provider key in your workspace must serve it. Defaults to openai/gpt-5-mini." }, { "name": "UR_CALL_TYPE_ID", "required": false, "description": "The call type's id (ct_…) the learning routes take. The first request that names the call type creates it; then look the id up by slug (the receipt step shows how)." }, { "name": "ULTRAROUTER_API_KEY", "required": false, "description": "The key the management MCP server uses: a separate key with config:read, plus config:write only if the agent may pause, disable or roll back learning. Read from the environment the coding agent was started in; the installer never writes it." }, { "name": "ULTRAROUTER_BASE_URL", "required": false, "description": "The gateway the management MCP server talks to. Defaults to http://localhost:8080." } ], "steps": [ { "id": "check-stack", "kind": "stack", "title": "Check that the local gateway answers", "detail": "Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS http://localhost:8080/readyz" }, { "run": "pnpm dev:e2e", "cwd": "the ultrarouter checkout" } ], "env": [], "links": [ "/docs/get-started" ], "expectedOutput": "{\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}" }, { "id": "account", "kind": "account", "title": "Create or sign in to the account", "detail": "Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.", "actor": "user", "optional": false, "confirmFirst": false, "commands": [], "env": [], "links": [ "http://localhost:3000/signup", "http://localhost:3000/login", "/docs/guides/providers-byok", "/docs/security" ] }, { "id": "environment-claude", "kind": "environment", "title": "Put the endpoint and the key in the environment", "detail": "read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.", "actor": "user", "optional": false, "confirmFirst": false, "hosts": [ "claude" ], "commands": [ { "run": "export UR_BASE_URL=http://localhost:8080" }, { "run": "read -rs UR_API_KEY && export UR_API_KEY" }, { "run": "claude --continue" }, { "run": "test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/agent-integrations" ], "expectedOutput": "UR_API_KEY is set" }, { "id": "environment-codex", "kind": "environment", "title": "Put the endpoint and the key in the environment", "detail": "read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.", "actor": "user", "optional": false, "confirmFirst": false, "hosts": [ "codex" ], "commands": [ { "run": "export UR_BASE_URL=http://localhost:8080" }, { "run": "read -rs UR_API_KEY && export UR_API_KEY" }, { "run": "codex resume --last -c shell_environment_policy.ignore_default_excludes=true" }, { "run": "test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/agent-integrations" ], "expectedOutput": "UR_API_KEY is set" }, { "id": "scaffold", "kind": "example", "title": "Create the example project", "detail": "A new, empty ultrarouter-learning-demo directory holds the example, and the SDK checkout is cloned next to it. Every later command runs in ultrarouter-learning-demo unless it names another directory.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "typescript" ], "commands": [ { "run": "mkdir ultrarouter-learning-demo", "cwd": "the directory that will hold the example" }, { "run": "npm init -y && npm pkg set type=module", "cwd": "ultrarouter-learning-demo" } ], "env": [], "links": [ "/docs/get-started" ] }, { "id": "install-sdk-typescript", "kind": "install-sdk", "title": "Install the TypeScript SDK from a local build", "detail": "@ultrarouter/sdk is not on the npm registry, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to the ultrarouter-learning-demo directory. Install the tarball with the project's package manager (pnpm add and yarn add take the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "typescript" ], "commands": [ { "run": "git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-typescript ../ultrarouter-sdk-typescript", "cwd": "ultrarouter-learning-demo" }, { "run": "npm ci && npm run build && npm pack", "cwd": "ultrarouter-sdk-typescript" }, { "run": "npm install ../ultrarouter-sdk-typescript/ultrarouter-sdk-0.1.0.tgz", "cwd": "ultrarouter-learning-demo" }, { "run": "npm install --save-dev tsx", "cwd": "ultrarouter-learning-demo" } ], "env": [], "links": [ "https://github.com/lucastonon/ultrarouter-sdk-typescript", "/docs/guides/sdk-typescript" ], "expectedOutput": "ultrarouter-sdk-0.1.0.tgz (the last line npm pack prints)" }, { "id": "connect", "kind": "connect", "title": "Connect the SDK to the gateway", "detail": "The example passes UR_BASE_URL and UR_API_KEY to the SDK client and tags its request with the learning-demo call type. Write verify-episode.ts with the code of this recipe's example, unchanged.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "typescript" ], "commands": [], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_MODEL" ], "links": [ "/docs/guides/sdk-typescript" ] }, { "id": "outcome", "kind": "outcome", "title": "Report a trusted outcome for each finished task", "detail": "The example reports its exact-match verifier's verdict this way; a real app follows the same rules. Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/outcomes", "/docs/get-started" ], "expectedOutput": "HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied." }, { "id": "run-example", "kind": "verify", "title": "Run one verified episode", "detail": "Run the example. It sends one request, applies the exact-match verifier, reports the outcome and waits for its receipt. The output below is illustrative of this toy demo, not evidence that learning helps.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "typescript" ], "commands": [ { "run": "npx tsx verify-episode.ts", "cwd": "ultrarouter-learning-demo" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_MODEL" ], "links": [ "/docs/evidence-status" ], "expectedOutput": "request: req_… (model openai/gpt-5-mini)\nverifier: pass\noutcome: pending, receipt ocr_… pending\nreceipt: applied\ncall type: ct_…, learning: off" }, { "id": "receipt", "kind": "receipt", "title": "Confirm the outcome receipt and the learning state", "detail": "The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug learning-demo (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"learning-demo\") | .id'" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started", "/docs/evidence-status" ], "expectedOutput": "{\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\nct_…\n{\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}" }, { "id": "mcp-servers", "kind": "mcp", "title": "Install the ultrarouter MCP servers", "detail": "Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.", "actor": "agent", "optional": true, "confirmFirst": true, "commands": [ { "run": "pnpm --filter @ultrarouter/mcp build", "cwd": "the ultrarouter checkout" }, { "run": "(cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz)", "cwd": "the ultrarouter checkout" }, { "run": "pnpm --filter \"@ultrarouter/docs-mcp...\" build", "cwd": "the ultrarouter checkout" }, { "run": "ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\"", "cwd": "the ultrarouter checkout", "hosts": [ "claude" ] }, { "run": "claude mcp get ultrarouter", "hosts": [ "claude" ] }, { "run": "ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\"", "cwd": "the ultrarouter checkout", "hosts": [ "codex" ] }, { "run": "codex mcp get ultrarouter", "hosts": [ "codex" ] }, { "run": "ultrarouter-mcp check" } ], "env": [ "ULTRAROUTER_API_KEY", "ULTRAROUTER_BASE_URL" ], "links": [ "/docs/guides/mcp-servers" ], "expectedOutput": "Connected to ultrarouter at http://localhost:8080.\nWorkspace id: … (from ultrarouter-mcp check)" }, { "id": "enable-learning", "kind": "enable", "title": "Turn on learning for the call type", "detail": "Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.", "actor": "agent", "optional": false, "confirmFirst": true, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started", "/docs/guides/providers-byok" ], "expectedOutput": "{\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}" }, { "id": "controls", "kind": "controls", "title": "Pause, disable or roll back", "detail": "These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.", "actor": "agent", "optional": false, "confirmFirst": true, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started" ] } ], "examples": { "typescript": { "kind": "runnable", "filename": "verify-episode.ts", "code": "// A toy episode: one question with a checkable answer, a real verifier, one outcome\n// report, its receipt and the learning state. It demonstrates the plumbing only.\nimport UltraRouter from \"@ultrarouter/sdk\";\n\nconst baseUrl = process.env.UR_BASE_URL;\nconst apiKey = process.env.UR_API_KEY;\nif (!baseUrl || !apiKey) throw new Error(\"Set UR_BASE_URL and UR_API_KEY first.\");\nconst model = process.env.UR_MODEL ?? \"openai/gpt-5-mini\";\nconst headers = { Authorization: `Bearer ${apiKey}`, \"Content-Type\": \"application/json\" };\nconst ur = new UltraRouter({ apiKey, baseURL: baseUrl });\n\n// 1. One task with a checkable answer. Naming the call type creates it on first use.\nconst [a, b] = [17, 25];\nconst res = await ur.chat.completions.create({\n model,\n call_type: \"learning-demo\",\n messages: [{ role: \"user\", content: `What is ${a} + ${b}? Reply with the number only.` }],\n});\nconst requestId = res.ur.request_id;\nconsole.log(`request: ${requestId} (model ${res.ur.model_served})`);\n\n// 2. The verifier: an exact match against the computed answer.\nconst passed = (res.choices[0]?.message.content ?? \"\").trim() === String(a + b);\nconsole.log(`verifier: ${passed ? \"pass\" : \"fail\"}`);\n\n// 3. Report the outcome once; a retry with the same source_event_id never counts twice.\nconst post = await fetch(`${baseUrl}/v1/requests/${requestId}/outcomes`, {\n method: \"POST\",\n headers,\n body: JSON.stringify({\n channel: \"computable\",\n kind: passed ? \"oracle_pass\" : \"oracle_fail\",\n source_event_id: `learning-demo-${requestId}`,\n }),\n});\nif (!post.ok) throw new Error(`outcome refused: HTTP ${post.status}`);\nconst outcome = await post.json();\nconsole.log(`outcome: ${outcome.status}, receipt ${outcome.receipt.id} ${outcome.receipt.status}`);\n\n// 4. The receipt settles once the gateway has recorded the request.\nlet receipt = outcome.receipt;\nfor (let i = 0; i < 10 && receipt.status === \"pending\"; i += 1) {\n await new Promise((resolve) => setTimeout(resolve, 1000));\n receipt = await (await fetch(`${baseUrl}/v1/outcome-receipts/${receipt.id}`, { headers })).json();\n}\nconsole.log(`receipt: ${receipt.status}`);\n\n// 5. Learning routes take the call type's id (ct_…), so look it up by slug.\nconst { data: callTypes } = await (await fetch(`${baseUrl}/v1/call-types`, { headers })).json();\nconst callType = callTypes.find((t: { slug: string }) => t.slug === \"learning-demo\");\nconst learning = await (\n await fetch(`${baseUrl}/v1/call-types/${callType.id}/learning`, { headers })\n).json();\nconsole.log(`call type: ${callType.id}, learning: ${learning.state}`);\n", "run": [ "npx tsx verify-episode.ts" ], "expectedOutput": "request: req_… (model openai/gpt-5-mini)\nverifier: pass\noutcome: pending, receipt ocr_… pending\nreceipt: applied\ncall type: ct_…, learning: off", "note": "A toy episode: one arithmetic question, an exact-match verifier, one outcome, its receipt and the learning state. The output is illustrative and shows the plumbing works; it says nothing about whether learning improves an agent. The outcome line reads accepted instead of pending when the gateway has already recorded the request." } }, "links": [ "/docs/get-started", "/docs/guides/sdk-typescript", "/docs/evidence-status" ] }, { "id": "fresh-python", "revision": 4, "path": "native-sdk", "title": "New Python example", "summary": "A new Python project with the native SDK installed from a local build and one runnable toy episode: a request, a verifier, an outcome, its receipt and the learning state.", "availability": "preview", "languages": [ "python" ], "hosts": [ "claude", "codex" ], "inspect": "Check that Python 3.10 or newer, uv and git are available. Whatever you find, leave an existing project unchanged: the example goes in a new ultrarouter-learning-demo directory beside it.", "env": [ { "name": "UR_BASE_URL", "required": true, "description": "The gateway base URL. On the local stack: http://localhost:8080." }, { "name": "UR_API_KEY", "required": true, "description": "An ultrarouter API key minted in the dashboard (Settings, API keys). Keep it in the environment or a secret manager, never in source control." }, { "name": "UR_MODEL", "required": false, "description": "The model id the demo calls. A provider key in your workspace must serve it. Defaults to openai/gpt-5-mini." }, { "name": "UR_CALL_TYPE_ID", "required": false, "description": "The call type's id (ct_…) the learning routes take. The first request that names the call type creates it; then look the id up by slug (the receipt step shows how)." }, { "name": "ULTRAROUTER_API_KEY", "required": false, "description": "The key the management MCP server uses: a separate key with config:read, plus config:write only if the agent may pause, disable or roll back learning. Read from the environment the coding agent was started in; the installer never writes it." }, { "name": "ULTRAROUTER_BASE_URL", "required": false, "description": "The gateway the management MCP server talks to. Defaults to http://localhost:8080." } ], "steps": [ { "id": "check-stack", "kind": "stack", "title": "Check that the local gateway answers", "detail": "Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS http://localhost:8080/readyz" }, { "run": "pnpm dev:e2e", "cwd": "the ultrarouter checkout" } ], "env": [], "links": [ "/docs/get-started" ], "expectedOutput": "{\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}" }, { "id": "account", "kind": "account", "title": "Create or sign in to the account", "detail": "Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.", "actor": "user", "optional": false, "confirmFirst": false, "commands": [], "env": [], "links": [ "http://localhost:3000/signup", "http://localhost:3000/login", "/docs/guides/providers-byok", "/docs/security" ] }, { "id": "environment-claude", "kind": "environment", "title": "Put the endpoint and the key in the environment", "detail": "read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.", "actor": "user", "optional": false, "confirmFirst": false, "hosts": [ "claude" ], "commands": [ { "run": "export UR_BASE_URL=http://localhost:8080" }, { "run": "read -rs UR_API_KEY && export UR_API_KEY" }, { "run": "claude --continue" }, { "run": "test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/agent-integrations" ], "expectedOutput": "UR_API_KEY is set" }, { "id": "environment-codex", "kind": "environment", "title": "Put the endpoint and the key in the environment", "detail": "read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.", "actor": "user", "optional": false, "confirmFirst": false, "hosts": [ "codex" ], "commands": [ { "run": "export UR_BASE_URL=http://localhost:8080" }, { "run": "read -rs UR_API_KEY && export UR_API_KEY" }, { "run": "codex resume --last -c shell_environment_policy.ignore_default_excludes=true" }, { "run": "test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/agent-integrations" ], "expectedOutput": "UR_API_KEY is set" }, { "id": "scaffold", "kind": "example", "title": "Create the example project", "detail": "A new, empty ultrarouter-learning-demo directory holds the example, and the SDK checkout is cloned next to it. Every later command runs in ultrarouter-learning-demo unless it names another directory.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "python" ], "commands": [ { "run": "mkdir ultrarouter-learning-demo", "cwd": "the directory that will hold the example" }, { "run": "python3 -m venv .venv", "cwd": "ultrarouter-learning-demo" } ], "env": [], "links": [ "/docs/get-started" ] }, { "id": "install-sdk-python", "kind": "install-sdk", "title": "Install the Python SDK from a local build", "detail": "ultrarouter is not on PyPI, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to the ultrarouter-learning-demo directory. Install the wheel into the project's environment (uv pip install takes the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "python" ], "commands": [ { "run": "git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-python ../ultrarouter-sdk-python", "cwd": "ultrarouter-learning-demo" }, { "run": "uv build", "cwd": "ultrarouter-sdk-python" }, { "run": ".venv/bin/pip install ../ultrarouter-sdk-python/dist/ultrarouter-0.1.0-py3-none-any.whl", "cwd": "ultrarouter-learning-demo" } ], "env": [], "links": [ "https://github.com/lucastonon/ultrarouter-sdk-python", "/docs/guides/sdk-python" ], "expectedOutput": "Successfully built dist/ultrarouter-0.1.0.tar.gz\nSuccessfully built dist/ultrarouter-0.1.0-py3-none-any.whl" }, { "id": "connect", "kind": "connect", "title": "Connect the SDK to the gateway", "detail": "The example passes UR_BASE_URL and UR_API_KEY to the SDK client and tags its request with the learning-demo call type. Write verify_episode.py with the code of this recipe's example, unchanged.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "python" ], "commands": [], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_MODEL" ], "links": [ "/docs/guides/sdk-python" ] }, { "id": "outcome", "kind": "outcome", "title": "Report a trusted outcome for each finished task", "detail": "The example reports its exact-match verifier's verdict this way; a real app follows the same rules. Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/outcomes", "/docs/get-started" ], "expectedOutput": "HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied." }, { "id": "run-example", "kind": "verify", "title": "Run one verified episode", "detail": "Run the example. It sends one request, applies the exact-match verifier, reports the outcome and waits for its receipt. The output below is illustrative of this toy demo, not evidence that learning helps.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "python" ], "commands": [ { "run": ".venv/bin/python verify_episode.py", "cwd": "ultrarouter-learning-demo" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_MODEL" ], "links": [ "/docs/evidence-status" ], "expectedOutput": "request: req_… (model openai/gpt-5-mini)\nverifier: pass\noutcome: pending, receipt ocr_… pending\nreceipt: applied\ncall type: ct_…, learning: off" }, { "id": "receipt", "kind": "receipt", "title": "Confirm the outcome receipt and the learning state", "detail": "The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug learning-demo (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"learning-demo\") | .id'" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started", "/docs/evidence-status" ], "expectedOutput": "{\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\nct_…\n{\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}" }, { "id": "mcp-servers", "kind": "mcp", "title": "Install the ultrarouter MCP servers", "detail": "Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.", "actor": "agent", "optional": true, "confirmFirst": true, "commands": [ { "run": "pnpm --filter @ultrarouter/mcp build", "cwd": "the ultrarouter checkout" }, { "run": "(cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz)", "cwd": "the ultrarouter checkout" }, { "run": "pnpm --filter \"@ultrarouter/docs-mcp...\" build", "cwd": "the ultrarouter checkout" }, { "run": "ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\"", "cwd": "the ultrarouter checkout", "hosts": [ "claude" ] }, { "run": "claude mcp get ultrarouter", "hosts": [ "claude" ] }, { "run": "ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\"", "cwd": "the ultrarouter checkout", "hosts": [ "codex" ] }, { "run": "codex mcp get ultrarouter", "hosts": [ "codex" ] }, { "run": "ultrarouter-mcp check" } ], "env": [ "ULTRAROUTER_API_KEY", "ULTRAROUTER_BASE_URL" ], "links": [ "/docs/guides/mcp-servers" ], "expectedOutput": "Connected to ultrarouter at http://localhost:8080.\nWorkspace id: … (from ultrarouter-mcp check)" }, { "id": "enable-learning", "kind": "enable", "title": "Turn on learning for the call type", "detail": "Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.", "actor": "agent", "optional": false, "confirmFirst": true, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started", "/docs/guides/providers-byok" ], "expectedOutput": "{\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}" }, { "id": "controls", "kind": "controls", "title": "Pause, disable or roll back", "detail": "These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.", "actor": "agent", "optional": false, "confirmFirst": true, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started" ] } ], "examples": { "python": { "kind": "runnable", "filename": "verify_episode.py", "code": "\"\"\"A toy episode: one question with a checkable answer, a real verifier, one outcome\nreport, its receipt and the learning state. It demonstrates the plumbing only.\"\"\"\nimport os\nimport time\n\nimport httpx\nfrom ultrarouter import UltraRouter\n\nbase_url = os.environ[\"UR_BASE_URL\"]\napi_key = os.environ[\"UR_API_KEY\"]\nmodel = os.environ.get(\"UR_MODEL\", \"openai/gpt-5-mini\")\nheaders = {\"Authorization\": f\"Bearer {api_key}\"}\nclient = UltraRouter(api_key=api_key, base_url=base_url)\n\n# 1. One task with a checkable answer. Naming the call type creates it on first use.\na, b = 17, 25\nreply = client.chat.completions.create(\n model=model,\n call_type=\"learning-demo\",\n messages=[{\"role\": \"user\", \"content\": f\"What is {a} + {b}? Reply with the number only.\"}],\n)\nrequest_id = reply.ur.request_id\nprint(f\"request: {request_id} (model {reply.ur.model_served})\")\n\n# 2. The verifier: an exact match against the computed answer.\npassed = (reply.choices[0].message.content or \"\").strip() == str(a + b)\nprint(f\"verifier: {'pass' if passed else 'fail'}\")\n\n# 3. Report the outcome once; a retry with the same source_event_id never counts twice.\npost = httpx.post(\n f\"{base_url}/v1/requests/{request_id}/outcomes\",\n headers=headers,\n json={\n \"channel\": \"computable\",\n \"kind\": \"oracle_pass\" if passed else \"oracle_fail\",\n \"source_event_id\": f\"learning-demo-{request_id}\",\n },\n)\npost.raise_for_status()\noutcome = post.json()\nreceipt = outcome[\"receipt\"]\nprint(f\"outcome: {outcome['status']}, receipt {receipt['id']} {receipt['status']}\")\n\n# 4. The receipt settles once the gateway has recorded the request.\nfor _ in range(10):\n if receipt[\"status\"] != \"pending\":\n break\n time.sleep(1)\n receipt = httpx.get(f\"{base_url}/v1/outcome-receipts/{receipt['id']}\", headers=headers).json()\nprint(f\"receipt: {receipt['status']}\")\n\n# 5. Learning routes take the call type's id (ct_…), so look it up by slug.\ncall_types = httpx.get(f\"{base_url}/v1/call-types\", headers=headers).json()[\"data\"]\ncall_type = next(t for t in call_types if t[\"slug\"] == \"learning-demo\")\nlearning = httpx.get(f\"{base_url}/v1/call-types/{call_type['id']}/learning\", headers=headers).json()\nprint(f\"call type: {call_type['id']}, learning: {learning['state']}\")\n", "run": [ ".venv/bin/python verify_episode.py" ], "expectedOutput": "request: req_… (model openai/gpt-5-mini)\nverifier: pass\noutcome: pending, receipt ocr_… pending\nreceipt: applied\ncall type: ct_…, learning: off", "note": "A toy episode: one arithmetic question, an exact-match verifier, one outcome, its receipt and the learning state. The output is illustrative and shows the plumbing works; it says nothing about whether learning improves an agent. The outcome line reads accepted instead of pending when the gateway has already recorded the request." } }, "links": [ "/docs/get-started", "/docs/guides/sdk-python", "/docs/evidence-status" ] }, { "id": "mcp-plugin", "revision": 4, "path": "mcp-plugin", "title": "Install the MCP servers, then connect the project", "summary": "Give the coding agent the management MCP (learning status, receipts, pause and rollback) and the documentation MCP first, built from source and registered by the installer or the plugin, then connect the project and report outcomes.", "availability": "preview", "languages": [ "typescript", "python" ], "hosts": [ "claude", "codex" ], "inspect": "Find where any client is configured and how a finished task is judged today, and check which MCP servers this agent already has, so you do not add a duplicate.", "env": [ { "name": "UR_BASE_URL", "required": true, "description": "The gateway base URL. On the local stack: http://localhost:8080." }, { "name": "UR_API_KEY", "required": true, "description": "An ultrarouter API key minted in the dashboard (Settings, API keys). Keep it in the environment or a secret manager, never in source control." }, { "name": "UR_CALL_TYPE_ID", "required": false, "description": "The call type's id (ct_…) the learning routes take. The first request that names the call type creates it; then look the id up by slug (the receipt step shows how)." }, { "name": "ULTRAROUTER_API_KEY", "required": true, "description": "The key the management MCP server uses: a separate key with config:read, plus config:write only if the agent may pause, disable or roll back learning. Read from the environment the coding agent was started in; the installer never writes it." }, { "name": "ULTRAROUTER_BASE_URL", "required": false, "description": "The gateway the management MCP server talks to. Defaults to http://localhost:8080." } ], "steps": [ { "id": "check-stack", "kind": "stack", "title": "Check that the local gateway answers", "detail": "Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS http://localhost:8080/readyz" }, { "run": "pnpm dev:e2e", "cwd": "the ultrarouter checkout" } ], "env": [], "links": [ "/docs/get-started" ], "expectedOutput": "{\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}" }, { "id": "account", "kind": "account", "title": "Create or sign in to the account", "detail": "Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.", "actor": "user", "optional": false, "confirmFirst": false, "commands": [], "env": [], "links": [ "http://localhost:3000/signup", "http://localhost:3000/login", "/docs/guides/providers-byok", "/docs/security" ] }, { "id": "environment-claude", "kind": "environment", "title": "Put the endpoint and the key in the environment", "detail": "read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.", "actor": "user", "optional": false, "confirmFirst": false, "hosts": [ "claude" ], "commands": [ { "run": "export UR_BASE_URL=http://localhost:8080" }, { "run": "read -rs UR_API_KEY && export UR_API_KEY" }, { "run": "claude --continue" }, { "run": "test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/agent-integrations" ], "expectedOutput": "UR_API_KEY is set" }, { "id": "environment-codex", "kind": "environment", "title": "Put the endpoint and the key in the environment", "detail": "read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.", "actor": "user", "optional": false, "confirmFirst": false, "hosts": [ "codex" ], "commands": [ { "run": "export UR_BASE_URL=http://localhost:8080" }, { "run": "read -rs UR_API_KEY && export UR_API_KEY" }, { "run": "codex resume --last -c shell_environment_policy.ignore_default_excludes=true" }, { "run": "test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/agent-integrations" ], "expectedOutput": "UR_API_KEY is set" }, { "id": "mcp-servers", "kind": "mcp", "title": "Install the ultrarouter MCP servers", "detail": "Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "pnpm --filter @ultrarouter/mcp build", "cwd": "the ultrarouter checkout" }, { "run": "(cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz)", "cwd": "the ultrarouter checkout" }, { "run": "pnpm --filter \"@ultrarouter/docs-mcp...\" build", "cwd": "the ultrarouter checkout" }, { "run": "ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\"", "cwd": "the ultrarouter checkout", "hosts": [ "claude" ] }, { "run": "claude mcp get ultrarouter", "hosts": [ "claude" ] }, { "run": "ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\"", "cwd": "the ultrarouter checkout", "hosts": [ "codex" ] }, { "run": "codex mcp get ultrarouter", "hosts": [ "codex" ] }, { "run": "ultrarouter-mcp check" } ], "env": [ "ULTRAROUTER_API_KEY", "ULTRAROUTER_BASE_URL" ], "links": [ "/docs/guides/mcp-servers" ], "expectedOutput": "Connected to ultrarouter at http://localhost:8080.\nWorkspace id: … (from ultrarouter-mcp check)" }, { "id": "plugin-claude", "kind": "mcp", "title": "Or use the ultrarouter plugin", "detail": "The plugin bundles the same management server and an integrate-ultrarouter skill, from the local marketplace in the ultrarouter checkout (not a public listing). Use the plugin or the installer, not both: both register an ultrarouter server. The plugin keeps its key in Claude Code secure storage; set it with /plugin configure.", "actor": "agent", "optional": true, "confirmFirst": true, "hosts": [ "claude" ], "commands": [ { "run": "claude plugin marketplace add ./integrations/claude-code", "cwd": "the ultrarouter checkout" }, { "run": "claude plugin install ultrarouter@ultrarouter-local", "cwd": "the ultrarouter checkout" } ], "env": [ "ULTRAROUTER_API_KEY" ], "links": [ "/docs/guides/mcp-servers" ] }, { "id": "plugin-codex", "kind": "mcp", "title": "Or use the ultrarouter plugin", "detail": "The plugin bundles the same management server and an integrate-ultrarouter skill, from the local marketplace in the ultrarouter checkout (not a public listing). A Codex plugin cannot carry a secret, so the installer runs too: Codex then uses the installer's entry, which forwards ULTRAROUTER_API_KEY.", "actor": "agent", "optional": true, "confirmFirst": true, "hosts": [ "codex" ], "commands": [ { "run": "codex plugin marketplace add ./integrations/codex", "cwd": "the ultrarouter checkout" }, { "run": "codex plugin add ultrarouter@ultrarouter-local", "cwd": "the ultrarouter checkout" }, { "run": "ultrarouter-mcp install --host codex", "cwd": "the ultrarouter checkout" } ], "env": [ "ULTRAROUTER_API_KEY" ], "links": [ "/docs/guides/mcp-servers" ] }, { "id": "install-sdk-typescript", "kind": "install-sdk", "title": "Install the TypeScript SDK from a local build", "detail": "Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. @ultrarouter/sdk is not on the npm registry, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the tarball with the project's package manager (pnpm add and yarn add take the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.", "actor": "agent", "optional": true, "confirmFirst": false, "languages": [ "typescript" ], "commands": [ { "run": "git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-typescript ../ultrarouter-sdk-typescript" }, { "run": "npm ci && npm run build && npm pack", "cwd": "../ultrarouter-sdk-typescript" }, { "run": "npm install ../ultrarouter-sdk-typescript/ultrarouter-sdk-0.1.0.tgz" } ], "env": [], "links": [ "https://github.com/lucastonon/ultrarouter-sdk-typescript", "/docs/guides/sdk-typescript" ], "expectedOutput": "ultrarouter-sdk-0.1.0.tgz (the last line npm pack prints)" }, { "id": "install-sdk-python", "kind": "install-sdk", "title": "Install the Python SDK from a local build", "detail": "Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. ultrarouter is not on PyPI, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the wheel into the project's environment (uv pip install takes the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.", "actor": "agent", "optional": true, "confirmFirst": false, "languages": [ "python" ], "commands": [ { "run": "git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-python ../ultrarouter-sdk-python" }, { "run": "uv build", "cwd": "../ultrarouter-sdk-python" }, { "run": "pip install ../ultrarouter-sdk-python/dist/ultrarouter-0.1.0-py3-none-any.whl" } ], "env": [], "links": [ "https://github.com/lucastonon/ultrarouter-sdk-python", "/docs/guides/sdk-python" ], "expectedOutput": "Successfully built dist/ultrarouter-0.1.0.tar.gz\nSuccessfully built dist/ultrarouter-0.1.0-py3-none-any.whl" }, { "id": "connect-typescript", "kind": "connect", "title": "Point the existing client at the gateway", "detail": "Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, .withResponse() exposes the headers.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "typescript" ], "commands": [], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/get-started", "/docs/quickstart" ] }, { "id": "connect-python", "kind": "connect", "title": "Point the existing client at the gateway", "detail": "Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, with_raw_response exposes the headers.", "actor": "agent", "optional": false, "confirmFirst": false, "languages": [ "python" ], "commands": [], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/get-started", "/docs/quickstart" ] }, { "id": "outcome", "kind": "outcome", "title": "Report a trusted outcome for each finished task", "detail": "Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/guides/outcomes", "/docs/get-started" ], "expectedOutput": "HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied." }, { "id": "verify-episode", "kind": "verify", "title": "Run one verified episode", "detail": "Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [], "env": [ "UR_BASE_URL", "UR_API_KEY" ], "links": [ "/docs/get-started" ] }, { "id": "receipt", "kind": "receipt", "title": "Confirm the outcome receipt and the learning state", "detail": "The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.", "actor": "agent", "optional": false, "confirmFirst": false, "commands": [ { "run": "curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"my-agent\") | .id'" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started", "/docs/evidence-status" ], "expectedOutput": "{\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\nct_…\n{\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}" }, { "id": "enable-learning", "kind": "enable", "title": "Turn on learning for the call type", "detail": "Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.", "actor": "agent", "optional": false, "confirmFirst": true, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started", "/docs/guides/providers-byok" ], "expectedOutput": "{\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}" }, { "id": "controls", "kind": "controls", "title": "Pause, disable or roll back", "detail": "These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.", "actor": "agent", "optional": false, "confirmFirst": true, "commands": [ { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"" }, { "run": "curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'" } ], "env": [ "UR_BASE_URL", "UR_API_KEY", "UR_CALL_TYPE_ID" ], "links": [ "/docs/get-started" ] } ], "examples": {}, "links": [ "/docs/guides/mcp-servers", "/docs/guides/agent-integrations", "/docs/get-started", "/docs/evidence-status" ] } ], "prompts": [ { "recipe": "existing-app", "host": "claude", "language": "typescript", "text": "Set up ultrarouter learning in this project. You are Claude Code.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: Add learning to an existing app (existing-app, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: TypeScript. If the project keeps agent instructions in CLAUDE.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Claude Code may ask me before it runs a shell command. When you request one that calls the gateway, say why it is needed.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Find where the client's base URL, key and model are configured, and how a finished task is judged today: that judgment is the outcome signal to reuse.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n claude --continue\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Install the TypeScript SDK from a local build (optional).\n Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. @ultrarouter/sdk is not on the npm registry, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the tarball with the project's package manager (pnpm add and yarn add take the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-typescript ../ultrarouter-sdk-typescript\n npm ci && npm run build && npm pack (in ../ultrarouter-sdk-typescript)\n npm install ../ultrarouter-sdk-typescript/ultrarouter-sdk-0.1.0.tgz\n Expected output (illustrative; ids differ):\n ultrarouter-sdk-0.1.0.tgz (the last line npm pack prints)\n Docs: https://github.com/lucastonon/ultrarouter-sdk-typescript | TypeScript SDK: https://docs.ultrarouter.io/docs/guides/sdk-typescript.md\n7. Point the existing client at the gateway.\n Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, .withResponse() exposes the headers.\n your existing client module:\n ```ts\n import OpenAI from \"openai\";\n\n // Only the configuration changes: base URL, key and the call-type header.\n const client = new OpenAI({\n baseURL: `${process.env.UR_BASE_URL}/v1`,\n apiKey: process.env.UR_API_KEY,\n defaultHeaders: { \"X-UR-Call-Type\": \"my-agent\" },\n });\n\n // Every model call of one task carries the same task id; keep the last request id.\n const { data: completion, response } = await client.chat.completions\n .create({ model, messages }, { headers: { \"X-UR-Artifact\": taskId } })\n .withResponse();\n const lastRequestId = response.headers.get(\"x-ur-request-id\");\n\n // When the task is finished, report your verifier's verdict once.\n await fetch(`${process.env.UR_BASE_URL}/v1/requests/${lastRequestId}/outcomes`, {\n method: \"POST\",\n headers: {\n Authorization: `Bearer ${process.env.UR_API_KEY}`,\n \"Content-Type\": \"application/json\",\n },\n body: JSON.stringify({\n channel: \"computable\",\n kind: passed ? \"oracle_pass\" : \"oracle_fail\",\n source_event_id: taskId,\n }),\n });\n ```\n Docs: Quickstart: https://docs.ultrarouter.io/docs/quickstart.md\n8. Report a trusted outcome for each finished task.\n Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n9. Install the ultrarouter MCP servers (optional).\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Ask me before you run it.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n claude mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n10. Run one verified episode.\n Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.\n11. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"my-agent\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n12. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n13. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n14. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host claude --with-docs (or the plugin with claude plugin uninstall ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "existing-app", "host": "claude", "language": "python", "text": "Set up ultrarouter learning in this project. You are Claude Code.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: Add learning to an existing app (existing-app, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: Python. If the project keeps agent instructions in CLAUDE.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Claude Code may ask me before it runs a shell command. When you request one that calls the gateway, say why it is needed.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Find where the client's base URL, key and model are configured, and how a finished task is judged today: that judgment is the outcome signal to reuse.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n claude --continue\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Install the Python SDK from a local build (optional).\n Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. ultrarouter is not on PyPI, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the wheel into the project's environment (uv pip install takes the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-python ../ultrarouter-sdk-python\n uv build (in ../ultrarouter-sdk-python)\n pip install ../ultrarouter-sdk-python/dist/ultrarouter-0.1.0-py3-none-any.whl\n Expected output (illustrative; ids differ):\n Successfully built dist/ultrarouter-0.1.0.tar.gz\n Successfully built dist/ultrarouter-0.1.0-py3-none-any.whl\n Docs: https://github.com/lucastonon/ultrarouter-sdk-python | Python SDK: https://docs.ultrarouter.io/docs/guides/sdk-python.md\n7. Point the existing client at the gateway.\n Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, with_raw_response exposes the headers.\n your existing client module:\n ```python\n import os\n\n import httpx\n from openai import OpenAI\n\n # Only the configuration changes: base URL, key and the call-type header.\n client = OpenAI(\n base_url=f\"{os.environ['UR_BASE_URL']}/v1\",\n api_key=os.environ[\"UR_API_KEY\"],\n default_headers={\"X-UR-Call-Type\": \"my-agent\"},\n )\n\n # Every model call of one task carries the same task id; keep the last request id.\n raw = client.chat.completions.with_raw_response.create(\n model=model, messages=messages, extra_headers={\"X-UR-Artifact\": task_id}\n )\n last_request_id = raw.headers.get(\"x-ur-request-id\")\n completion = raw.parse()\n\n # When the task is finished, report your verifier's verdict once.\n httpx.post(\n f\"{os.environ['UR_BASE_URL']}/v1/requests/{last_request_id}/outcomes\",\n headers={\"Authorization\": f\"Bearer {os.environ['UR_API_KEY']}\"},\n json={\n \"channel\": \"computable\",\n \"kind\": \"oracle_pass\" if passed else \"oracle_fail\",\n \"source_event_id\": task_id,\n },\n ).raise_for_status()\n ```\n Docs: Quickstart: https://docs.ultrarouter.io/docs/quickstart.md\n8. Report a trusted outcome for each finished task.\n Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n9. Install the ultrarouter MCP servers (optional).\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Ask me before you run it.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n claude mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n10. Run one verified episode.\n Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.\n11. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"my-agent\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n12. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n13. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n14. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host claude --with-docs (or the plugin with claude plugin uninstall ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "existing-app", "host": "codex", "language": "typescript", "text": "Set up ultrarouter learning in this project. You are Codex.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: Add learning to an existing app (existing-app, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: TypeScript. If the project keeps agent instructions in AGENTS.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Codex runs commands in a sandbox that may block network access, including to localhost. If a call to the gateway fails with a network or sandbox error, ask me to approve it or to allow network access for this session. Never turn the sandbox off.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Find where the client's base URL, key and model are configured, and how a finished task is judged today: that judgment is the outcome signal to reuse.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n codex resume --last -c shell_environment_policy.ignore_default_excludes=true\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Install the TypeScript SDK from a local build (optional).\n Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. @ultrarouter/sdk is not on the npm registry, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the tarball with the project's package manager (pnpm add and yarn add take the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-typescript ../ultrarouter-sdk-typescript\n npm ci && npm run build && npm pack (in ../ultrarouter-sdk-typescript)\n npm install ../ultrarouter-sdk-typescript/ultrarouter-sdk-0.1.0.tgz\n Expected output (illustrative; ids differ):\n ultrarouter-sdk-0.1.0.tgz (the last line npm pack prints)\n Docs: https://github.com/lucastonon/ultrarouter-sdk-typescript | TypeScript SDK: https://docs.ultrarouter.io/docs/guides/sdk-typescript.md\n7. Point the existing client at the gateway.\n Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, .withResponse() exposes the headers.\n your existing client module:\n ```ts\n import OpenAI from \"openai\";\n\n // Only the configuration changes: base URL, key and the call-type header.\n const client = new OpenAI({\n baseURL: `${process.env.UR_BASE_URL}/v1`,\n apiKey: process.env.UR_API_KEY,\n defaultHeaders: { \"X-UR-Call-Type\": \"my-agent\" },\n });\n\n // Every model call of one task carries the same task id; keep the last request id.\n const { data: completion, response } = await client.chat.completions\n .create({ model, messages }, { headers: { \"X-UR-Artifact\": taskId } })\n .withResponse();\n const lastRequestId = response.headers.get(\"x-ur-request-id\");\n\n // When the task is finished, report your verifier's verdict once.\n await fetch(`${process.env.UR_BASE_URL}/v1/requests/${lastRequestId}/outcomes`, {\n method: \"POST\",\n headers: {\n Authorization: `Bearer ${process.env.UR_API_KEY}`,\n \"Content-Type\": \"application/json\",\n },\n body: JSON.stringify({\n channel: \"computable\",\n kind: passed ? \"oracle_pass\" : \"oracle_fail\",\n source_event_id: taskId,\n }),\n });\n ```\n Docs: Quickstart: https://docs.ultrarouter.io/docs/quickstart.md\n8. Report a trusted outcome for each finished task.\n Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n9. Install the ultrarouter MCP servers (optional).\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Ask me before you run it.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n codex mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n10. Run one verified episode.\n Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.\n11. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"my-agent\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n12. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n13. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n14. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host codex --with-docs (or the plugin with codex plugin remove ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "existing-app", "host": "codex", "language": "python", "text": "Set up ultrarouter learning in this project. You are Codex.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: Add learning to an existing app (existing-app, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: Python. If the project keeps agent instructions in AGENTS.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Codex runs commands in a sandbox that may block network access, including to localhost. If a call to the gateway fails with a network or sandbox error, ask me to approve it or to allow network access for this session. Never turn the sandbox off.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Find where the client's base URL, key and model are configured, and how a finished task is judged today: that judgment is the outcome signal to reuse.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n codex resume --last -c shell_environment_policy.ignore_default_excludes=true\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Install the Python SDK from a local build (optional).\n Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. ultrarouter is not on PyPI, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the wheel into the project's environment (uv pip install takes the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-python ../ultrarouter-sdk-python\n uv build (in ../ultrarouter-sdk-python)\n pip install ../ultrarouter-sdk-python/dist/ultrarouter-0.1.0-py3-none-any.whl\n Expected output (illustrative; ids differ):\n Successfully built dist/ultrarouter-0.1.0.tar.gz\n Successfully built dist/ultrarouter-0.1.0-py3-none-any.whl\n Docs: https://github.com/lucastonon/ultrarouter-sdk-python | Python SDK: https://docs.ultrarouter.io/docs/guides/sdk-python.md\n7. Point the existing client at the gateway.\n Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, with_raw_response exposes the headers.\n your existing client module:\n ```python\n import os\n\n import httpx\n from openai import OpenAI\n\n # Only the configuration changes: base URL, key and the call-type header.\n client = OpenAI(\n base_url=f\"{os.environ['UR_BASE_URL']}/v1\",\n api_key=os.environ[\"UR_API_KEY\"],\n default_headers={\"X-UR-Call-Type\": \"my-agent\"},\n )\n\n # Every model call of one task carries the same task id; keep the last request id.\n raw = client.chat.completions.with_raw_response.create(\n model=model, messages=messages, extra_headers={\"X-UR-Artifact\": task_id}\n )\n last_request_id = raw.headers.get(\"x-ur-request-id\")\n completion = raw.parse()\n\n # When the task is finished, report your verifier's verdict once.\n httpx.post(\n f\"{os.environ['UR_BASE_URL']}/v1/requests/{last_request_id}/outcomes\",\n headers={\"Authorization\": f\"Bearer {os.environ['UR_API_KEY']}\"},\n json={\n \"channel\": \"computable\",\n \"kind\": \"oracle_pass\" if passed else \"oracle_fail\",\n \"source_event_id\": task_id,\n },\n ).raise_for_status()\n ```\n Docs: Quickstart: https://docs.ultrarouter.io/docs/quickstart.md\n8. Report a trusted outcome for each finished task.\n Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n9. Install the ultrarouter MCP servers (optional).\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Ask me before you run it.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n codex mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n10. Run one verified episode.\n Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.\n11. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"my-agent\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n12. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n13. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n14. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host codex --with-docs (or the plugin with codex plugin remove ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "fresh-typescript", "host": "claude", "language": "typescript", "text": "Set up ultrarouter learning in this project. You are Claude Code.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: New TypeScript example (fresh-typescript, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: TypeScript. If the project keeps agent instructions in CLAUDE.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Claude Code may ask me before it runs a shell command. When you request one that calls the gateway, say why it is needed.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Check that Node.js 20 or newer, npm and git are available. Whatever you find, leave an existing project unchanged: the example goes in a new ultrarouter-learning-demo directory beside it.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - TypeScript SDK: https://docs.ultrarouter.io/docs/guides/sdk-typescript.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n claude --continue\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Create the example project.\n A new, empty ultrarouter-learning-demo directory holds the example, and the SDK checkout is cloned next to it. Every later command runs in ultrarouter-learning-demo unless it names another directory.\n Commands:\n mkdir ultrarouter-learning-demo (in the directory that will hold the example)\n npm init -y && npm pkg set type=module (in ultrarouter-learning-demo)\n7. Install the TypeScript SDK from a local build.\n @ultrarouter/sdk is not on the npm registry, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to the ultrarouter-learning-demo directory. Install the tarball with the project's package manager (pnpm add and yarn add take the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-typescript ../ultrarouter-sdk-typescript (in ultrarouter-learning-demo)\n npm ci && npm run build && npm pack (in ultrarouter-sdk-typescript)\n npm install ../ultrarouter-sdk-typescript/ultrarouter-sdk-0.1.0.tgz (in ultrarouter-learning-demo)\n npm install --save-dev tsx (in ultrarouter-learning-demo)\n Expected output (illustrative; ids differ):\n ultrarouter-sdk-0.1.0.tgz (the last line npm pack prints)\n Docs: https://github.com/lucastonon/ultrarouter-sdk-typescript | TypeScript SDK: https://docs.ultrarouter.io/docs/guides/sdk-typescript.md\n8. Connect the SDK to the gateway.\n The example passes UR_BASE_URL and UR_API_KEY to the SDK client and tags its request with the learning-demo call type. Write verify-episode.ts with the code of this recipe's example, unchanged.\n verify-episode.ts:\n ```ts\n // A toy episode: one question with a checkable answer, a real verifier, one outcome\n // report, its receipt and the learning state. It demonstrates the plumbing only.\n import UltraRouter from \"@ultrarouter/sdk\";\n\n const baseUrl = process.env.UR_BASE_URL;\n const apiKey = process.env.UR_API_KEY;\n if (!baseUrl || !apiKey) throw new Error(\"Set UR_BASE_URL and UR_API_KEY first.\");\n const model = process.env.UR_MODEL ?? \"openai/gpt-5-mini\";\n const headers = { Authorization: `Bearer ${apiKey}`, \"Content-Type\": \"application/json\" };\n const ur = new UltraRouter({ apiKey, baseURL: baseUrl });\n\n // 1. One task with a checkable answer. Naming the call type creates it on first use.\n const [a, b] = [17, 25];\n const res = await ur.chat.completions.create({\n model,\n call_type: \"learning-demo\",\n messages: [{ role: \"user\", content: `What is ${a} + ${b}? Reply with the number only.` }],\n });\n const requestId = res.ur.request_id;\n console.log(`request: ${requestId} (model ${res.ur.model_served})`);\n\n // 2. The verifier: an exact match against the computed answer.\n const passed = (res.choices[0]?.message.content ?? \"\").trim() === String(a + b);\n console.log(`verifier: ${passed ? \"pass\" : \"fail\"}`);\n\n // 3. Report the outcome once; a retry with the same source_event_id never counts twice.\n const post = await fetch(`${baseUrl}/v1/requests/${requestId}/outcomes`, {\n method: \"POST\",\n headers,\n body: JSON.stringify({\n channel: \"computable\",\n kind: passed ? \"oracle_pass\" : \"oracle_fail\",\n source_event_id: `learning-demo-${requestId}`,\n }),\n });\n if (!post.ok) throw new Error(`outcome refused: HTTP ${post.status}`);\n const outcome = await post.json();\n console.log(`outcome: ${outcome.status}, receipt ${outcome.receipt.id} ${outcome.receipt.status}`);\n\n // 4. The receipt settles once the gateway has recorded the request.\n let receipt = outcome.receipt;\n for (let i = 0; i < 10 && receipt.status === \"pending\"; i += 1) {\n await new Promise((resolve) => setTimeout(resolve, 1000));\n receipt = await (await fetch(`${baseUrl}/v1/outcome-receipts/${receipt.id}`, { headers })).json();\n }\n console.log(`receipt: ${receipt.status}`);\n\n // 5. Learning routes take the call type's id (ct_…), so look it up by slug.\n const { data: callTypes } = await (await fetch(`${baseUrl}/v1/call-types`, { headers })).json();\n const callType = callTypes.find((t: { slug: string }) => t.slug === \"learning-demo\");\n const learning = await (\n await fetch(`${baseUrl}/v1/call-types/${callType.id}/learning`, { headers })\n ).json();\n console.log(`call type: ${callType.id}, learning: ${learning.state}`);\n ```\n Docs: TypeScript SDK: https://docs.ultrarouter.io/docs/guides/sdk-typescript.md\n9. Report a trusted outcome for each finished task.\n The example reports its exact-match verifier's verdict this way; a real app follows the same rules. Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n10. Run one verified episode.\n Run the example. It sends one request, applies the exact-match verifier, reports the outcome and waits for its receipt. The output below is illustrative of this toy demo, not evidence that learning helps.\n Commands:\n npx tsx verify-episode.ts (in ultrarouter-learning-demo)\n Expected output (illustrative; ids differ):\n request: req_… (model openai/gpt-5-mini)\n verifier: pass\n outcome: pending, receipt ocr_… pending\n receipt: applied\n call type: ct_…, learning: off\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n11. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug learning-demo (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"learning-demo\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n12. Install the ultrarouter MCP servers (optional).\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Ask me before you run it.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n claude mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n13. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n14. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n15. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host claude --with-docs (or the plugin with claude plugin uninstall ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "fresh-typescript", "host": "codex", "language": "typescript", "text": "Set up ultrarouter learning in this project. You are Codex.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: New TypeScript example (fresh-typescript, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: TypeScript. If the project keeps agent instructions in AGENTS.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Codex runs commands in a sandbox that may block network access, including to localhost. If a call to the gateway fails with a network or sandbox error, ask me to approve it or to allow network access for this session. Never turn the sandbox off.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Check that Node.js 20 or newer, npm and git are available. Whatever you find, leave an existing project unchanged: the example goes in a new ultrarouter-learning-demo directory beside it.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - TypeScript SDK: https://docs.ultrarouter.io/docs/guides/sdk-typescript.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n codex resume --last -c shell_environment_policy.ignore_default_excludes=true\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Create the example project.\n A new, empty ultrarouter-learning-demo directory holds the example, and the SDK checkout is cloned next to it. Every later command runs in ultrarouter-learning-demo unless it names another directory.\n Commands:\n mkdir ultrarouter-learning-demo (in the directory that will hold the example)\n npm init -y && npm pkg set type=module (in ultrarouter-learning-demo)\n7. Install the TypeScript SDK from a local build.\n @ultrarouter/sdk is not on the npm registry, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to the ultrarouter-learning-demo directory. Install the tarball with the project's package manager (pnpm add and yarn add take the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-typescript ../ultrarouter-sdk-typescript (in ultrarouter-learning-demo)\n npm ci && npm run build && npm pack (in ultrarouter-sdk-typescript)\n npm install ../ultrarouter-sdk-typescript/ultrarouter-sdk-0.1.0.tgz (in ultrarouter-learning-demo)\n npm install --save-dev tsx (in ultrarouter-learning-demo)\n Expected output (illustrative; ids differ):\n ultrarouter-sdk-0.1.0.tgz (the last line npm pack prints)\n Docs: https://github.com/lucastonon/ultrarouter-sdk-typescript | TypeScript SDK: https://docs.ultrarouter.io/docs/guides/sdk-typescript.md\n8. Connect the SDK to the gateway.\n The example passes UR_BASE_URL and UR_API_KEY to the SDK client and tags its request with the learning-demo call type. Write verify-episode.ts with the code of this recipe's example, unchanged.\n verify-episode.ts:\n ```ts\n // A toy episode: one question with a checkable answer, a real verifier, one outcome\n // report, its receipt and the learning state. It demonstrates the plumbing only.\n import UltraRouter from \"@ultrarouter/sdk\";\n\n const baseUrl = process.env.UR_BASE_URL;\n const apiKey = process.env.UR_API_KEY;\n if (!baseUrl || !apiKey) throw new Error(\"Set UR_BASE_URL and UR_API_KEY first.\");\n const model = process.env.UR_MODEL ?? \"openai/gpt-5-mini\";\n const headers = { Authorization: `Bearer ${apiKey}`, \"Content-Type\": \"application/json\" };\n const ur = new UltraRouter({ apiKey, baseURL: baseUrl });\n\n // 1. One task with a checkable answer. Naming the call type creates it on first use.\n const [a, b] = [17, 25];\n const res = await ur.chat.completions.create({\n model,\n call_type: \"learning-demo\",\n messages: [{ role: \"user\", content: `What is ${a} + ${b}? Reply with the number only.` }],\n });\n const requestId = res.ur.request_id;\n console.log(`request: ${requestId} (model ${res.ur.model_served})`);\n\n // 2. The verifier: an exact match against the computed answer.\n const passed = (res.choices[0]?.message.content ?? \"\").trim() === String(a + b);\n console.log(`verifier: ${passed ? \"pass\" : \"fail\"}`);\n\n // 3. Report the outcome once; a retry with the same source_event_id never counts twice.\n const post = await fetch(`${baseUrl}/v1/requests/${requestId}/outcomes`, {\n method: \"POST\",\n headers,\n body: JSON.stringify({\n channel: \"computable\",\n kind: passed ? \"oracle_pass\" : \"oracle_fail\",\n source_event_id: `learning-demo-${requestId}`,\n }),\n });\n if (!post.ok) throw new Error(`outcome refused: HTTP ${post.status}`);\n const outcome = await post.json();\n console.log(`outcome: ${outcome.status}, receipt ${outcome.receipt.id} ${outcome.receipt.status}`);\n\n // 4. The receipt settles once the gateway has recorded the request.\n let receipt = outcome.receipt;\n for (let i = 0; i < 10 && receipt.status === \"pending\"; i += 1) {\n await new Promise((resolve) => setTimeout(resolve, 1000));\n receipt = await (await fetch(`${baseUrl}/v1/outcome-receipts/${receipt.id}`, { headers })).json();\n }\n console.log(`receipt: ${receipt.status}`);\n\n // 5. Learning routes take the call type's id (ct_…), so look it up by slug.\n const { data: callTypes } = await (await fetch(`${baseUrl}/v1/call-types`, { headers })).json();\n const callType = callTypes.find((t: { slug: string }) => t.slug === \"learning-demo\");\n const learning = await (\n await fetch(`${baseUrl}/v1/call-types/${callType.id}/learning`, { headers })\n ).json();\n console.log(`call type: ${callType.id}, learning: ${learning.state}`);\n ```\n Docs: TypeScript SDK: https://docs.ultrarouter.io/docs/guides/sdk-typescript.md\n9. Report a trusted outcome for each finished task.\n The example reports its exact-match verifier's verdict this way; a real app follows the same rules. Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n10. Run one verified episode.\n Run the example. It sends one request, applies the exact-match verifier, reports the outcome and waits for its receipt. The output below is illustrative of this toy demo, not evidence that learning helps.\n Commands:\n npx tsx verify-episode.ts (in ultrarouter-learning-demo)\n Expected output (illustrative; ids differ):\n request: req_… (model openai/gpt-5-mini)\n verifier: pass\n outcome: pending, receipt ocr_… pending\n receipt: applied\n call type: ct_…, learning: off\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n11. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug learning-demo (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"learning-demo\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n12. Install the ultrarouter MCP servers (optional).\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Ask me before you run it.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n codex mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n13. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n14. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n15. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host codex --with-docs (or the plugin with codex plugin remove ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "fresh-python", "host": "claude", "language": "python", "text": "Set up ultrarouter learning in this project. You are Claude Code.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: New Python example (fresh-python, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: Python. If the project keeps agent instructions in CLAUDE.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Claude Code may ask me before it runs a shell command. When you request one that calls the gateway, say why it is needed.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Check that Python 3.10 or newer, uv and git are available. Whatever you find, leave an existing project unchanged: the example goes in a new ultrarouter-learning-demo directory beside it.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - Python SDK: https://docs.ultrarouter.io/docs/guides/sdk-python.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n claude --continue\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Create the example project.\n A new, empty ultrarouter-learning-demo directory holds the example, and the SDK checkout is cloned next to it. Every later command runs in ultrarouter-learning-demo unless it names another directory.\n Commands:\n mkdir ultrarouter-learning-demo (in the directory that will hold the example)\n python3 -m venv .venv (in ultrarouter-learning-demo)\n7. Install the Python SDK from a local build.\n ultrarouter is not on PyPI, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to the ultrarouter-learning-demo directory. Install the wheel into the project's environment (uv pip install takes the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-python ../ultrarouter-sdk-python (in ultrarouter-learning-demo)\n uv build (in ultrarouter-sdk-python)\n .venv/bin/pip install ../ultrarouter-sdk-python/dist/ultrarouter-0.1.0-py3-none-any.whl (in ultrarouter-learning-demo)\n Expected output (illustrative; ids differ):\n Successfully built dist/ultrarouter-0.1.0.tar.gz\n Successfully built dist/ultrarouter-0.1.0-py3-none-any.whl\n Docs: https://github.com/lucastonon/ultrarouter-sdk-python | Python SDK: https://docs.ultrarouter.io/docs/guides/sdk-python.md\n8. Connect the SDK to the gateway.\n The example passes UR_BASE_URL and UR_API_KEY to the SDK client and tags its request with the learning-demo call type. Write verify_episode.py with the code of this recipe's example, unchanged.\n verify_episode.py:\n ```python\n \"\"\"A toy episode: one question with a checkable answer, a real verifier, one outcome\n report, its receipt and the learning state. It demonstrates the plumbing only.\"\"\"\n import os\n import time\n\n import httpx\n from ultrarouter import UltraRouter\n\n base_url = os.environ[\"UR_BASE_URL\"]\n api_key = os.environ[\"UR_API_KEY\"]\n model = os.environ.get(\"UR_MODEL\", \"openai/gpt-5-mini\")\n headers = {\"Authorization\": f\"Bearer {api_key}\"}\n client = UltraRouter(api_key=api_key, base_url=base_url)\n\n # 1. One task with a checkable answer. Naming the call type creates it on first use.\n a, b = 17, 25\n reply = client.chat.completions.create(\n model=model,\n call_type=\"learning-demo\",\n messages=[{\"role\": \"user\", \"content\": f\"What is {a} + {b}? Reply with the number only.\"}],\n )\n request_id = reply.ur.request_id\n print(f\"request: {request_id} (model {reply.ur.model_served})\")\n\n # 2. The verifier: an exact match against the computed answer.\n passed = (reply.choices[0].message.content or \"\").strip() == str(a + b)\n print(f\"verifier: {'pass' if passed else 'fail'}\")\n\n # 3. Report the outcome once; a retry with the same source_event_id never counts twice.\n post = httpx.post(\n f\"{base_url}/v1/requests/{request_id}/outcomes\",\n headers=headers,\n json={\n \"channel\": \"computable\",\n \"kind\": \"oracle_pass\" if passed else \"oracle_fail\",\n \"source_event_id\": f\"learning-demo-{request_id}\",\n },\n )\n post.raise_for_status()\n outcome = post.json()\n receipt = outcome[\"receipt\"]\n print(f\"outcome: {outcome['status']}, receipt {receipt['id']} {receipt['status']}\")\n\n # 4. The receipt settles once the gateway has recorded the request.\n for _ in range(10):\n if receipt[\"status\"] != \"pending\":\n break\n time.sleep(1)\n receipt = httpx.get(f\"{base_url}/v1/outcome-receipts/{receipt['id']}\", headers=headers).json()\n print(f\"receipt: {receipt['status']}\")\n\n # 5. Learning routes take the call type's id (ct_…), so look it up by slug.\n call_types = httpx.get(f\"{base_url}/v1/call-types\", headers=headers).json()[\"data\"]\n call_type = next(t for t in call_types if t[\"slug\"] == \"learning-demo\")\n learning = httpx.get(f\"{base_url}/v1/call-types/{call_type['id']}/learning\", headers=headers).json()\n print(f\"call type: {call_type['id']}, learning: {learning['state']}\")\n ```\n Docs: Python SDK: https://docs.ultrarouter.io/docs/guides/sdk-python.md\n9. Report a trusted outcome for each finished task.\n The example reports its exact-match verifier's verdict this way; a real app follows the same rules. Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n10. Run one verified episode.\n Run the example. It sends one request, applies the exact-match verifier, reports the outcome and waits for its receipt. The output below is illustrative of this toy demo, not evidence that learning helps.\n Commands:\n .venv/bin/python verify_episode.py (in ultrarouter-learning-demo)\n Expected output (illustrative; ids differ):\n request: req_… (model openai/gpt-5-mini)\n verifier: pass\n outcome: pending, receipt ocr_… pending\n receipt: applied\n call type: ct_…, learning: off\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n11. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug learning-demo (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"learning-demo\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n12. Install the ultrarouter MCP servers (optional).\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Ask me before you run it.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n claude mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n13. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n14. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n15. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host claude --with-docs (or the plugin with claude plugin uninstall ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "fresh-python", "host": "codex", "language": "python", "text": "Set up ultrarouter learning in this project. You are Codex.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: New Python example (fresh-python, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: Python. If the project keeps agent instructions in AGENTS.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Codex runs commands in a sandbox that may block network access, including to localhost. If a call to the gateway fails with a network or sandbox error, ask me to approve it or to allow network access for this session. Never turn the sandbox off.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Check that Python 3.10 or newer, uv and git are available. Whatever you find, leave an existing project unchanged: the example goes in a new ultrarouter-learning-demo directory beside it.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - Python SDK: https://docs.ultrarouter.io/docs/guides/sdk-python.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n codex resume --last -c shell_environment_policy.ignore_default_excludes=true\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Create the example project.\n A new, empty ultrarouter-learning-demo directory holds the example, and the SDK checkout is cloned next to it. Every later command runs in ultrarouter-learning-demo unless it names another directory.\n Commands:\n mkdir ultrarouter-learning-demo (in the directory that will hold the example)\n python3 -m venv .venv (in ultrarouter-learning-demo)\n7. Install the Python SDK from a local build.\n ultrarouter is not on PyPI, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to the ultrarouter-learning-demo directory. Install the wheel into the project's environment (uv pip install takes the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-python ../ultrarouter-sdk-python (in ultrarouter-learning-demo)\n uv build (in ultrarouter-sdk-python)\n .venv/bin/pip install ../ultrarouter-sdk-python/dist/ultrarouter-0.1.0-py3-none-any.whl (in ultrarouter-learning-demo)\n Expected output (illustrative; ids differ):\n Successfully built dist/ultrarouter-0.1.0.tar.gz\n Successfully built dist/ultrarouter-0.1.0-py3-none-any.whl\n Docs: https://github.com/lucastonon/ultrarouter-sdk-python | Python SDK: https://docs.ultrarouter.io/docs/guides/sdk-python.md\n8. Connect the SDK to the gateway.\n The example passes UR_BASE_URL and UR_API_KEY to the SDK client and tags its request with the learning-demo call type. Write verify_episode.py with the code of this recipe's example, unchanged.\n verify_episode.py:\n ```python\n \"\"\"A toy episode: one question with a checkable answer, a real verifier, one outcome\n report, its receipt and the learning state. It demonstrates the plumbing only.\"\"\"\n import os\n import time\n\n import httpx\n from ultrarouter import UltraRouter\n\n base_url = os.environ[\"UR_BASE_URL\"]\n api_key = os.environ[\"UR_API_KEY\"]\n model = os.environ.get(\"UR_MODEL\", \"openai/gpt-5-mini\")\n headers = {\"Authorization\": f\"Bearer {api_key}\"}\n client = UltraRouter(api_key=api_key, base_url=base_url)\n\n # 1. One task with a checkable answer. Naming the call type creates it on first use.\n a, b = 17, 25\n reply = client.chat.completions.create(\n model=model,\n call_type=\"learning-demo\",\n messages=[{\"role\": \"user\", \"content\": f\"What is {a} + {b}? Reply with the number only.\"}],\n )\n request_id = reply.ur.request_id\n print(f\"request: {request_id} (model {reply.ur.model_served})\")\n\n # 2. The verifier: an exact match against the computed answer.\n passed = (reply.choices[0].message.content or \"\").strip() == str(a + b)\n print(f\"verifier: {'pass' if passed else 'fail'}\")\n\n # 3. Report the outcome once; a retry with the same source_event_id never counts twice.\n post = httpx.post(\n f\"{base_url}/v1/requests/{request_id}/outcomes\",\n headers=headers,\n json={\n \"channel\": \"computable\",\n \"kind\": \"oracle_pass\" if passed else \"oracle_fail\",\n \"source_event_id\": f\"learning-demo-{request_id}\",\n },\n )\n post.raise_for_status()\n outcome = post.json()\n receipt = outcome[\"receipt\"]\n print(f\"outcome: {outcome['status']}, receipt {receipt['id']} {receipt['status']}\")\n\n # 4. The receipt settles once the gateway has recorded the request.\n for _ in range(10):\n if receipt[\"status\"] != \"pending\":\n break\n time.sleep(1)\n receipt = httpx.get(f\"{base_url}/v1/outcome-receipts/{receipt['id']}\", headers=headers).json()\n print(f\"receipt: {receipt['status']}\")\n\n # 5. Learning routes take the call type's id (ct_…), so look it up by slug.\n call_types = httpx.get(f\"{base_url}/v1/call-types\", headers=headers).json()[\"data\"]\n call_type = next(t for t in call_types if t[\"slug\"] == \"learning-demo\")\n learning = httpx.get(f\"{base_url}/v1/call-types/{call_type['id']}/learning\", headers=headers).json()\n print(f\"call type: {call_type['id']}, learning: {learning['state']}\")\n ```\n Docs: Python SDK: https://docs.ultrarouter.io/docs/guides/sdk-python.md\n9. Report a trusted outcome for each finished task.\n The example reports its exact-match verifier's verdict this way; a real app follows the same rules. Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n10. Run one verified episode.\n Run the example. It sends one request, applies the exact-match verifier, reports the outcome and waits for its receipt. The output below is illustrative of this toy demo, not evidence that learning helps.\n Commands:\n .venv/bin/python verify_episode.py (in ultrarouter-learning-demo)\n Expected output (illustrative; ids differ):\n request: req_… (model openai/gpt-5-mini)\n verifier: pass\n outcome: pending, receipt ocr_… pending\n receipt: applied\n call type: ct_…, learning: off\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n11. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug learning-demo (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"learning-demo\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n12. Install the ultrarouter MCP servers (optional).\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Ask me before you run it.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n codex mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n13. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n14. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n15. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host codex --with-docs (or the plugin with codex plugin remove ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "mcp-plugin", "host": "claude", "language": "typescript", "text": "Set up ultrarouter learning in this project. You are Claude Code.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: Install the MCP servers, then connect the project (mcp-plugin, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: TypeScript. If the project keeps agent instructions in CLAUDE.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Claude Code may ask me before it runs a shell command. When you request one that calls the gateway, say why it is needed.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Find where any client is configured and how a finished task is judged today, and check which MCP servers this agent already has, so you do not add a duplicate.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n - Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n claude --continue\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Install the ultrarouter MCP servers.\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n claude mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n7. Or use the ultrarouter plugin (optional).\n The plugin bundles the same management server and an integrate-ultrarouter skill, from the local marketplace in the ultrarouter checkout (not a public listing). Use the plugin or the installer, not both: both register an ultrarouter server. The plugin keeps its key in Claude Code secure storage; set it with /plugin configure.\n Ask me before you run it.\n Commands:\n claude plugin marketplace add ./integrations/claude-code (in the ultrarouter checkout)\n claude plugin install ultrarouter@ultrarouter-local (in the ultrarouter checkout)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n8. Install the TypeScript SDK from a local build (optional).\n Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. @ultrarouter/sdk is not on the npm registry, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the tarball with the project's package manager (pnpm add and yarn add take the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-typescript ../ultrarouter-sdk-typescript\n npm ci && npm run build && npm pack (in ../ultrarouter-sdk-typescript)\n npm install ../ultrarouter-sdk-typescript/ultrarouter-sdk-0.1.0.tgz\n Expected output (illustrative; ids differ):\n ultrarouter-sdk-0.1.0.tgz (the last line npm pack prints)\n Docs: https://github.com/lucastonon/ultrarouter-sdk-typescript | TypeScript SDK: https://docs.ultrarouter.io/docs/guides/sdk-typescript.md\n9. Point the existing client at the gateway.\n Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, .withResponse() exposes the headers.\n Docs: Quickstart: https://docs.ultrarouter.io/docs/quickstart.md\n10. Report a trusted outcome for each finished task.\n Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n11. Run one verified episode.\n Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.\n12. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"my-agent\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n13. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n14. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n15. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host claude --with-docs (or the plugin with claude plugin uninstall ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "mcp-plugin", "host": "claude", "language": "python", "text": "Set up ultrarouter learning in this project. You are Claude Code.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: Install the MCP servers, then connect the project (mcp-plugin, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: Python. If the project keeps agent instructions in CLAUDE.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Claude Code may ask me before it runs a shell command. When you request one that calls the gateway, say why it is needed.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Find where any client is configured and how a finished task is judged today, and check which MCP servers this agent already has, so you do not add a duplicate.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n - Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Claude Code passes the environment it was started from to its shell. Export UR_BASE_URL and UR_API_KEY before starting it; if it is already running, restart it with the resume command so this conversation continues. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n claude --continue\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Install the ultrarouter MCP servers.\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host claude --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n claude mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n7. Or use the ultrarouter plugin (optional).\n The plugin bundles the same management server and an integrate-ultrarouter skill, from the local marketplace in the ultrarouter checkout (not a public listing). Use the plugin or the installer, not both: both register an ultrarouter server. The plugin keeps its key in Claude Code secure storage; set it with /plugin configure.\n Ask me before you run it.\n Commands:\n claude plugin marketplace add ./integrations/claude-code (in the ultrarouter checkout)\n claude plugin install ultrarouter@ultrarouter-local (in the ultrarouter checkout)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n8. Install the Python SDK from a local build (optional).\n Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. ultrarouter is not on PyPI, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the wheel into the project's environment (uv pip install takes the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-python ../ultrarouter-sdk-python\n uv build (in ../ultrarouter-sdk-python)\n pip install ../ultrarouter-sdk-python/dist/ultrarouter-0.1.0-py3-none-any.whl\n Expected output (illustrative; ids differ):\n Successfully built dist/ultrarouter-0.1.0.tar.gz\n Successfully built dist/ultrarouter-0.1.0-py3-none-any.whl\n Docs: https://github.com/lucastonon/ultrarouter-sdk-python | Python SDK: https://docs.ultrarouter.io/docs/guides/sdk-python.md\n9. Point the existing client at the gateway.\n Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, with_raw_response exposes the headers.\n Docs: Quickstart: https://docs.ultrarouter.io/docs/quickstart.md\n10. Report a trusted outcome for each finished task.\n Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n11. Run one verified episode.\n Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.\n12. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"my-agent\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n13. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n14. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n15. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host claude --with-docs (or the plugin with claude plugin uninstall ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "mcp-plugin", "host": "codex", "language": "typescript", "text": "Set up ultrarouter learning in this project. You are Codex.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: Install the MCP servers, then connect the project (mcp-plugin, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: TypeScript. If the project keeps agent instructions in AGENTS.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Codex runs commands in a sandbox that may block network access, including to localhost. If a call to the gateway fails with a network or sandbox error, ask me to approve it or to allow network access for this session. Never turn the sandbox off.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Find where any client is configured and how a finished task is judged today, and check which MCP servers this agent already has, so you do not add a duplicate.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n - Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n codex resume --last -c shell_environment_policy.ignore_default_excludes=true\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Install the ultrarouter MCP servers.\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n codex mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n7. Or use the ultrarouter plugin (optional).\n The plugin bundles the same management server and an integrate-ultrarouter skill, from the local marketplace in the ultrarouter checkout (not a public listing). A Codex plugin cannot carry a secret, so the installer runs too: Codex then uses the installer's entry, which forwards ULTRAROUTER_API_KEY.\n Ask me before you run it.\n Commands:\n codex plugin marketplace add ./integrations/codex (in the ultrarouter checkout)\n codex plugin add ultrarouter@ultrarouter-local (in the ultrarouter checkout)\n ultrarouter-mcp install --host codex (in the ultrarouter checkout)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n8. Install the TypeScript SDK from a local build (optional).\n Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. @ultrarouter/sdk is not on the npm registry, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the tarball with the project's package manager (pnpm add and yarn add take the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-typescript ../ultrarouter-sdk-typescript\n npm ci && npm run build && npm pack (in ../ultrarouter-sdk-typescript)\n npm install ../ultrarouter-sdk-typescript/ultrarouter-sdk-0.1.0.tgz\n Expected output (illustrative; ids differ):\n ultrarouter-sdk-0.1.0.tgz (the last line npm pack prints)\n Docs: https://github.com/lucastonon/ultrarouter-sdk-typescript | TypeScript SDK: https://docs.ultrarouter.io/docs/guides/sdk-typescript.md\n9. Point the existing client at the gateway.\n Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, .withResponse() exposes the headers.\n Docs: Quickstart: https://docs.ultrarouter.io/docs/quickstart.md\n10. Report a trusted outcome for each finished task.\n Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n11. Run one verified episode.\n Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.\n12. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"my-agent\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n13. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n14. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n15. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host codex --with-docs (or the plugin with codex plugin remove ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" }, { "recipe": "mcp-plugin", "host": "codex", "language": "python", "text": "Set up ultrarouter learning in this project. You are Codex.\n\nultrarouter is an OpenAI-compatible gateway. The project keeps its agent; it reports a trusted outcome for each finished task, and ultrarouter turns those outcomes into candidate adaptations, evaluates each one against the version running today, deploys it only if it wins, and keeps learning from the new outcomes. Learning changes the context ultrarouter serves the agent, not the model's weights.\n\nThis setup\n- Recipe: Install the MCP servers, then connect the project (mcp-plugin, revision 4, schema ur.setup-recipes/1). Machine-readable copy: https://docs.ultrarouter.io/setup-recipes.json\n- Project language: Python. If the project keeps agent instructions in AGENTS.md, add a short note there on where the gateway is configured and where outcomes are reported.\n- Learning is not on the hosted service yet (the hosted gateway at https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup uses the local stack: the gateway at http://localhost:8080 and the dashboard at http://localhost:3000.\n- Learning is a preview. The loop runs on the local stack; if an outcome-receipt or learning endpoint returns 404, the checkout is older than it, so tell me and stop at that step. A working setup proves the plumbing, not that learning helps: https://docs.ultrarouter.io/docs/evidence-status.md\n\nRules\n- Never write a key into source files, tests, fixtures, logs, commit messages or your replies. Read it from UR_API_KEY in the environment. Before adding it to a .env file, confirm .env is git-ignored, and never print that file.\n- Ask me only for what needs my identity, my consent or credentials you cannot get: signing up or in, email verification, two-factor enrolment, accepting terms, the workspace Learning opt-in (corrections and knowledge consent), provider keys, minting the ultrarouter API key, the path of my ultrarouter checkout, and a yes before you turn learning on or use its controls. Do everything else yourself. Never create an account, accept terms or change consent for me.\n- Codex runs commands in a sandbox that may block network access, including to localhost. If a call to the gateway fails with a network or sandbox error, ask me to approve it or to allow network access for this session. Never turn the sandbox off.\n- Reuse the project's existing configuration: read UR_BASE_URL and UR_API_KEY where the project already reads its settings, and keep its client, its retry and timeout settings, and its tests.\n- Never claim that learning improved anything. Report what the receipt and the learning status say.\n\nSteps\n1. Inspect the project. Identify its language and package manager, the existing LLM client (the openai package, another OpenAI-compatible SDK, or raw HTTP calls) and its tests. Find where any client is configured and how a finished task is judged today, and check which MCP servers this agent already has, so you do not add a duplicate.\n2. Read the current ultrarouter docs before you change anything:\n - Get started: https://docs.ultrarouter.io/docs/get-started.md\n - MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n - Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n - Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n - Setup recipes (JSON): https://docs.ultrarouter.io/setup-recipes.json\n3. Check that the local gateway answers.\n Learning is not on the hosted service yet (https://api.ultrarouter.io is invite-only and serves routing and compression), so this setup runs on the local stack from the ultrarouter repository: the gateway on http://localhost:8080 and the dashboard on http://localhost:3000. If the readiness check does not answer, the stack has to be started in the ultrarouter checkout, which the coding agent may not have access to.\n Commands:\n curl -sS http://localhost:8080/readyz\n pnpm dev:e2e (in the ultrarouter checkout)\n Expected output (illustrative; ids differ):\n {\"status\":\"ready\",\"service\":\"gateway\",\"checks\":{…}}\n4. Create or sign in to the account.\n Sign up at http://localhost:3000/signup (or sign in at http://localhost:3000/login) and create the workspace. Email is not delivered on the local stack: the verification link is in the test outbox at http://localhost:8080/auth/test/outbox?to=. Enrol two-factor authentication when asked; an owner needs it to mint keys and to change learning. As the workspace owner, choose Allow learning on the dashboard Learning page for the call type, or in the Learn onboarding flow. This workspace opt-in grants both corrections consent (store task content) and knowledge consent (use what it learns); corrections consent alone is insufficient. The equivalent owner-session request is PUT /v1/workspaces//self-learning with {\"enabled\":true}; a bearer API key cannot grant these permissions. Under Providers & BYOK, add a key (Add key) for the provider whose models the agent and the learning reflection will use, or add a custom provider with its own key: learning only ever uses a model the workspace owns, never a platform key. A new workspace starts on prepaid UR credits, which never pay the workspace's own keys, so choose BYOK funding once from the signed-in session: POST /v1/workspaces//billing/funding with {\"mode\":\"byok\",\"allowPlatformFallback\":false,\"requestKey\":\"<16 to 128 letters, digits, - or _>\"}. Until then a request only your own key can serve is refused with HTTP 403 permission. Then mint an API key in Settings, API keys, with the gateway:invoke, outcomes:write and config:read scopes; add config:write only if the agent should turn learning on, pause it or roll it back through the API.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Docs: http://localhost:3000/signup | http://localhost:3000/login | Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md | Security and data handling: https://docs.ultrarouter.io/docs/security.md\n5. Put the endpoint and the key in the environment.\n read -rs reads the key without echoing it, so it stays out of the terminal and the shell history. Codex removes environment variables whose names contain KEY, SECRET or TOKEN from the commands it runs, so UR_API_KEY is invisible to them by default. Export UR_BASE_URL and UR_API_KEY, then start Codex (or resume this conversation) with the override below; it lets every such variable through, so start it from a shell that holds only the secrets this project needs. If the project loads a .env file, add UR_API_KEY there only after confirming .env is git-ignored. To use the management MCP, export ULTRAROUTER_API_KEY (a separate key) the same way before starting the agent. The last command checks the key is visible without printing it.\n This step is mine: ask me to do it, then wait until I confirm it is done.\n Commands:\n export UR_BASE_URL=http://localhost:8080\n read -rs UR_API_KEY && export UR_API_KEY\n codex resume --last -c shell_environment_policy.ignore_default_excludes=true\n test -n \"$UR_API_KEY\" && echo \"UR_API_KEY is set\"\n Expected output (illustrative; ids differ):\n UR_API_KEY is set\n Docs: Set up with Claude Code or Codex: https://docs.ultrarouter.io/docs/guides/agent-integrations.md\n6. Install the ultrarouter MCP servers.\n Neither MCP server is on a package registry yet. In the ultrarouter checkout, build and pack the management MCP and install that tarball, build the documentation MCP, then let the installer register both for this agent; it keeps every other host setting and never writes a key. The management server reads ULTRAROUTER_API_KEY (a separate key with config:read; config:write only if the agent may pause, disable or roll back learning) and ULTRAROUTER_BASE_URL from the environment the agent was started in. It reads learning status, evaluations and outcome receipts and can pause, disable or roll back; it cannot report an outcome, so it can never mark a task as succeeded, and it does not route this agent's own traffic through ultrarouter.\n Commands:\n pnpm --filter @ultrarouter/mcp build (in the ultrarouter checkout)\n (cd packages/mcp && pnpm pack && npm install -g ./ultrarouter-mcp-0.1.0.tgz) (in the ultrarouter checkout)\n pnpm --filter \"@ultrarouter/docs-mcp...\" build (in the ultrarouter checkout)\n ultrarouter-mcp install --host codex --with-docs --docs-repo \"$PWD\" (in the ultrarouter checkout)\n codex mcp get ultrarouter\n ultrarouter-mcp check\n Expected output (illustrative; ids differ):\n Connected to ultrarouter at http://localhost:8080.\n Workspace id: … (from ultrarouter-mcp check)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n7. Or use the ultrarouter plugin (optional).\n The plugin bundles the same management server and an integrate-ultrarouter skill, from the local marketplace in the ultrarouter checkout (not a public listing). A Codex plugin cannot carry a secret, so the installer runs too: Codex then uses the installer's entry, which forwards ULTRAROUTER_API_KEY.\n Ask me before you run it.\n Commands:\n codex plugin marketplace add ./integrations/codex (in the ultrarouter checkout)\n codex plugin add ultrarouter@ultrarouter-local (in the ultrarouter checkout)\n ultrarouter-mcp install --host codex (in the ultrarouter checkout)\n Docs: MCP servers for Claude Code and Codex: https://docs.ultrarouter.io/docs/guides/mcp-servers.md\n8. Install the Python SDK from a local build (optional).\n Only when the project has no LLM client yet, or I ask for the native SDK: the existing OpenAI-compatible client is enough for the connection. ultrarouter is not on PyPI, so never install it from there. Build it from the feat/learning-loop branch of the public source repository, cloned next to this project. Install the wheel into the project's environment (uv pip install takes the same path). Version 0.1.0 includes the learning and outcome-receipt helpers (see the SDK guide); the commands in this recipe use the same REST calls, so they also work without them.\n Commands:\n git clone --branch feat/learning-loop --single-branch https://github.com/lucastonon/ultrarouter-sdk-python ../ultrarouter-sdk-python\n uv build (in ../ultrarouter-sdk-python)\n pip install ../ultrarouter-sdk-python/dist/ultrarouter-0.1.0-py3-none-any.whl\n Expected output (illustrative; ids differ):\n Successfully built dist/ultrarouter-0.1.0.tar.gz\n Successfully built dist/ultrarouter-0.1.0-py3-none-any.whl\n Docs: https://github.com/lucastonon/ultrarouter-sdk-python | Python SDK: https://docs.ultrarouter.io/docs/guides/sdk-python.md\n9. Point the existing client at the gateway.\n Keep the project's OpenAI-compatible client and change only its configuration: the base URL becomes \"$UR_BASE_URL/v1\", the key comes from UR_API_KEY, and an X-UR-Call-Type header names this agent, because learning is per call type. If one task makes several model calls, send the same X-UR-Artifact header (the task or run id) on each of them. Keep the x-ur-request-id response header of the task's last call: the outcome is reported on it. With the openai package, with_raw_response exposes the headers.\n Docs: Quickstart: https://docs.ultrarouter.io/docs/quickstart.md\n10. Report a trusted outcome for each finished task.\n Use the most trustworthy signal the project already has that a finished task succeeded, and report it once per task on the task's last request id. A verifier (tests, a schema or exact-match check) or a business rule sends channel computable with kind oracle_pass or oracle_fail. A delayed webhook sends channel resolution with kind resolved_true or resolved_false. A human review sends channel correction with kind accepted_unchanged or rejected. Self-rating (the agent or a model grading its own answer, or a thumbs-up) is weaker: it is recorded as a proxy and never teaches. An HTTP 200 or a tool call that returned is not task success. If the outcome is unknown, send nothing. A retry reuses the same source_event_id, so it never counts twice.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcomes\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"channel\":\"computable\",\"kind\":\"oracle_pass\",\"source_event_id\":\"\"}'\n Expected output (illustrative; ids differ):\n HTTP 202 with \"status\" accepted, duplicate or pending, and a \"receipt\" (ocr_…) that stays pending until the gateway has recorded the request, then becomes applied.\n Docs: Record outcomes: https://docs.ultrarouter.io/docs/guides/outcomes.md\n11. Run one verified episode.\n Add or extend a test or script that runs one real task end to end through the gateway, applies the real verifier and reports the outcome. Run it with the project's own test command and keep its request id.\n12. Confirm the outcome receipt and the learning state.\n The outcome's receipt (ocr_…) must reach applied. Pending is normal until the gateway has recorded the request; rejected, failed or expired carries a reason to fix. The learning routes take the call type's id (ct_…), not its slug: the gateway created the call type on the first request that named it, so look the id up by its slug my-agent (with jq, or read it from the JSON) and use it as UR_CALL_TYPE_ID below. Then read the learning state and report it as returned; it is off until learning is turned on.\n Commands:\n curl -sS \"$UR_BASE_URL/v1/requests/$REQUEST_ID/outcome-receipts\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types\" -H \"Authorization: Bearer $UR_API_KEY\" | jq -r '.data[] | select(.slug == \"my-agent\") | .id'\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning\" -H \"Authorization: Bearer $UR_API_KEY\"\n Expected output (illustrative; ids differ):\n {\"object\":\"list\",\"data\":[{\"object\":\"outcome_receipt\",\"id\":\"ocr_…\",\"status\":\"applied\",…}]}\n ct_…\n {\"object\":\"learning\",\"call_type_id\":\"ct_…\",\"state\":\"off\",…}\n Docs: Evidence status: https://docs.ultrarouter.io/docs/evidence-status.md\n13. Turn on learning for the call type.\n Learning stays off until it is turned on for the call type, by its id. Three things must hold first, and the route names the one that is missing. Both corrections consent and knowledge consent are on through the workspace Allow learning opt-in (409 corrections_consent_required if either is missing; the owner uses the dashboard Learning page or PUT /v1/workspaces//self-learning with {\"enabled\":true} from a signed-in owner session, never a bearer API key). reflection_model is a model the workspace owns, through a provider key it stored or one of its custom providers (400 reflection_model_not_workspace_owned with the reason; there is no platform or paid fallback, and a workspace on prepaid billing must fund with its own keys, byok). The caller is an owner or admin, or a key with config:write. With promotion mode \"evaluate\", the default, a learned candidate is evaluated before replacing the active version. During the comparison it serves only its randomly assigned share of eligible episodes; negative or inconclusive results keep the current active version. Afterwards, reflection.provider in the learning status says whether the reflection model is usable and, if not, why.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/enable\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"reflection_model\":\"\",\"promotion\":{\"mode\":\"evaluate\"}}'\n Expected output (illustrative; ids differ):\n {\"object\":\"learning_control\",\"action\":\"enable\",\"changed\":true,…,\"learning\":{\"state\":\"collecting\",…}}\n Docs: Provider keys: https://docs.ultrarouter.io/docs/guides/providers-byok.md\n14. Pause, disable or roll back.\n These are the controls for later and the core of the rollback instructions; do not run them during setup. Pause stops reflecting and evaluating but keeps serving the active version; resume undoes it. Disable stops serving learned lessons and stops a running evaluation. Rollback points the call type at an earlier version from the version history (version_id, epb_…; omitted, the version that was active before the current one). Every control is audited and needs an owner or admin, or a key with config:write.\n Ask me before you run it.\n Commands:\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/pause\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/resume\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/disable\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/versions\" -H \"Authorization: Bearer $UR_API_KEY\"\n curl -sS -X POST \"$UR_BASE_URL/v1/call-types/$UR_CALL_TYPE_ID/learning/rollback\" -H \"Authorization: Bearer $UR_API_KEY\" -H \"Content-Type: application/json\" -d '{\"version_id\":\"\"}'\n15. Summarize. List every file you changed and why, the commands you ran with their results, the request id, the receipt status and the learning state. Then give exact rollback instructions: revert those files, point the client back at its previous base URL, disable learning or roll back to an earlier version with the calls above, remove the MCP servers with ultrarouter-mcp uninstall --host codex --with-docs (or the plugin with codex plugin remove ultrarouter@ultrarouter-local) if you added them, and remind me to revoke the API keys in the dashboard Settings.\n" } ] }