Connect a check (oracles)

Some of your systems cannot produce the right answer, but they can check one: a SQL sandbox that runs the generated query, an address validator, a booking dry-run, a reconciliation job that learns the truth a week later, your own CI harness. ultrarouter calls such a check an oracle. (A system that can produce the right answer should send a declared correction instead.)

This page is the API-level contract. The dashboard card and the SDK helpers come later; everything below works today over plain HTTP.

Kinds, transports and roles

• platform — evaluated inside ultrarouter, no network call.
• push — ultrarouter POSTs each job to your HTTPS endpoint.
• pull — your worker or CI claims jobs and posts signed verdicts.

A check never declares a role; it earns one by qualification:

• Detector — it catches at least 80 % of planted failures and passes at least 90 % of valid rewrites. A detector may flag failures.
• Certifier — the detector bars plus zero false passes over at least 73 planted failures (the Wilson 95 % upper bound on its false-pass rate is at most 5 %). Only a certifier can back a verified fix. A label set can only ever be a detector: a wrong label that is still in the set passes it.

Register

curl -s "$UR_BASE_URL/v1/flywheel/oracles" \
  -H "Cookie: $UR_SESSION" -H "X-UR-Workspace: $UR_WORKSPACE" \
  -H 'Content-Type: application/json' \
  -d '{ "kind": "test_endpoint",
        "spec": { "url": "https://checks.example.com/verify", "contract_version": "v1" } }'

The response carries the oracle id (orc_…), the version (the sha256 of the canonical definition — a changed spec is a new version whose qualification starts from nothing) and, for a new push or pull oracle, its signing secret:

{ "object": "oracle_registration", "oracle_id": "orc_…", "version_hash": "…",
  "transport": "push", "secret": "orsec_…", "secret_last4": "…" }

The secret is shown once. Store it now; ultrarouter keeps it encrypted and shows only its last 4 characters afterwards. Rotate it with POST /v1/flywheel/oracles/{id}/rotate-secret — the old secret stops verifying immediately.

Who may register:

• a platform check — an API key with config:write, or a session with the config.write permission;
• a push or pull check — a session with the owner/admin flywheel.oracle permission. An API key cannot register an endpoint we will call or mint a signing secret.

A push endpoint must pass the egress guard at registration and on every call: HTTPS only, and never a private, link-local or cloud-metadata address (a refused endpoint is a 400 egress_blocked).

The job and the verdict

Every job — a qualification probe, a canary, a connectivity test, a real candidate — has the same shape. It is the body we POST to a push endpoint and each entry a pull claim returns:

{ "object": "oracle_job", "schema": "ur.oracle.v1",
  "job_id": "orj_…", "oracle_id": "orc_…", "oracle_version": "…",
  "input": "the task input", "output": "the candidate to judge",
  "candidate_sha256": "…", "due_at": "…", "issued_at": "…" }

Your check answers with one verdict per job:

{ "job_id": "orj_…", "oracle_version": "…", "candidate_sha256": "…",
  "verdict": "fail", "reasons": ["wrong_value"] }

• verdict is pass or fail. There is no third answer: a timeout, an error or a malformed reply is recorded as an error and never counts as a pass.
• The verdict must echo the job_id, the oracle_version the job was issued under and the candidate_sha256. A mismatch is refused by name (version_mismatch, candidate_mismatch, job_mismatch) and never recorded as an answer.
• reasons is optional, only on a fail, at most 8, from a closed list: wrong_value, missing_field, extra_field, format_invalid, schema_invalid, rule_failed, execution_failed, truncated, off_task, mismatch, other. An unknown key or free text makes the whole reply malformed_reply.

Signing

Every delivery in both directions is signed with the oracle's secret, the same scheme as our webhooks:

x-ur-oracle-timestamp: <unix seconds>
x-ur-oracle-signature: sha256=<hex HMAC-SHA256(secret, "<timestamp>.<raw body>")>

The timestamp must be within 15 minutes of our clock. A missing, stale or wrong signature is refused with a 403 that names the reason (missing_signature, stale_signature, invalid_signature).

Push: we call your endpoint

For each job we POST the job JSON to your URL, signed as above (plus the informational x-ur-oracle-job and x-ur-oracle-version headers). Your endpoint must:

1. verify our signature;
2. reply 200 with Content-Type: application/json and one verdict, signed back with the same secret — an unsigned reply is refused, so a hijacked host cannot answer for you.

Limits: the connection is pinned to the address we validated, redirects are followed at most twice (each re-validated), the reply is capped at 64 KB, and each call is cut at 10 seconds.

Pull: your worker claims jobs

BODY='{"max":50,"lease_seconds":900}'
TS=$(date +%s)
SIG="sha256=$(printf '%s' "$TS.$BODY" | openssl dgst -sha256 -hmac "$ORACLE_SECRET" -hex | sed 's/^.* //')"
curl -s "$UR_BASE_URL/v1/flywheel/oracles/$ORACLE_ID/jobs/claim" \
  -H "Authorization: Bearer $UR_API_KEY" -H 'Content-Type: application/json' \
  -H "x-ur-oracle-timestamp: $TS" -H "x-ur-oracle-signature: $SIG" -d "$BODY"

• The API key needs outcomes:write and logs:read (a claim returns stored task content). Posting verdicts needs outcomes:write.
• Each job's payload is returned once — in the claim response — and then deleted. Keep it until you have posted the verdict.
• A claim leases its jobs (30 seconds to 1 hour, default 15 minutes). A verdict after the lease is refused lease_expired.

Post verdicts to POST /v1/flywheel/oracles/{id}/verdicts with {"verdicts":[…]}, signed the same way. Per verdict you get accepted, duplicate (the same verdict again — a visible no-op, forever), conflict (a different answer for an answered job — never overwritten) or refused with a named reason. When nothing in a post is accepted or duplicate, the whole post is a 409 naming the first refusal.

Qualify

curl -s "$UR_BASE_URL/v1/flywheel/oracles/$ORACLE_ID/qualify" \
  -H "Cookie: $UR_SESSION" -H "X-UR-Workspace: $UR_WORKSPACE" \
  -H 'Content-Type: application/json' \
  -d '{ "examples": [ { "input": "…", "output": "a known-good answer" }, … ] }'

Send at least 10 distinct known-good examples (up to 50 are used; the certifier bar needs at least 37). From each, ultrarouter plants two failures — an omitted field, a wrong value, a truncated answer, an answer that ignores the instruction — and adds one valid rewrite, then shuffles them. The probes travel as ordinary jobs: your check cannot tell them apart from real work.

• A platform check is graded at once.
• A push check receives every probe immediately and is graded when the last one is answered. If any call fails, the result is incomplete and nothing is recorded — fix the endpoint and qualify again.
• A pull check returns pending; it is graded when your worker answers the last probe (the verdicts response then lists the graded qualification).

A push or pull qualification stores its probes and your examples, so the workspace needs corrections consent and full retention (409 no_consent otherwise).

A false pass is never forgotten: a version that ever passed a planted failure can never become a certifier again. A fixed check is a new version.

Canaries

About one in twenty real jobs your check receives is a planted failure it cannot tell apart from the rest. If your check passes one, that version immediately loses its qualification (check_failed), and every fix it vouched for stops counting. Canaries are planted from the examples you qualified with; if those are deleted (retention or a consent change), new jobs are refused no_canary_pool until you qualify again.

Traffic in your holdout is never sent to a check. A job that fixes a request outside the arms the flywheel learns from (the holdout, or a request with no arm) is refused holdout_excluded, together with the rest of its batch.

Test, read, revoke

• POST /v1/flywheel/oracles/{id}/test — a push check is called at once with a synthetic probe (no customer content) and the response shows the exact job we sent and the verdict or error; a pull check gets a pending test job to claim. Tests are never graded.
• GET /v1/flywheel/oracles/{id} — the current version, every version, the role it earned, and for push and pull checks the secret's last 4, the canary pool, job counts and the latest qualification's progress.
• POST /v1/flywheel/oracles/{id}/revoke — terminal. Every version stops certifying, open jobs are cancelled and their content deleted.

Let a certified check repair failed answers

Once your check is a certifier, you can let ultrarouter use it to verify and repair the answers of a workflow. You grant this when you confirm the workflow's learning plan (POST /v1/flywheel/plan/{id}/confirm, a session action — an API key can never turn it on), with a repair_policy block:

{ "repair_policy": {
    "oracle_id": "orc_…",
    "monthly_cap_micro_usd": 5000000,
    "teacher_model": "openai/gpt-5.2",
    "sample_rate": 1 } }

• oracle_id — the check. Its current version is pinned; a new version of the check needs a new confirmation.
• monthly_cap_micro_usd — required, in integer micro-USD. There is no default: repairs spend nothing until you set a budget, and 0 means "spend nothing".
• teacher_model — optional: a stronger model allowed to propose fixes. Without it, that step is skipped.
• sample_rate — optional, in (0, 1] (default 1): the fixed share of eligible answers that are looked at.

The confirm response lists the grant it wrote in repair_policies (the call type, the pinned check version, the budget). A malformed grant is a 400 invalid_repair_policy, an unknown check a 404, a revoked one a 409. A pull check is a 409 oracle_transport_unsupported: its verdicts arrive whenever your worker claims its jobs, after the repair pass that would have to wait for them, so a grant naming it could never run (see below). A refused grant confirms nothing — the plan stays open. The grant lives on the workflow's final call type; changing that call type's config does not change it, and a config rollback keeps the grant you confirmed last.

What ultrarouter then does on its own, for answers served on the arms the flywheel learns from (never holdout traffic — and never an answer whose record carries no arm: an unknown assignment is skipped before your check is called, because a fix for it could never be used for learning):

1. Your served answer. It asks your check about the answer you already served. A pass is recorded as "accepted unchanged". No model is called.
2. Your application's own retry. If your application retried the same request in the same trace and got a different answer (for an agent: the same tool, called again with repaired arguments), your check judges that retry. No model is called.
3. Resamples. Up to 3 new samples from the model that served the request.
4. The teacher. Up to 2 samples from the teacher you confirmed.

It stops at the first answer your check passes. Agent workflows (tools, or multi-turn conversations) get step 2 only: a changed earlier action would make the rest of the recorded conversation meaningless. A fix is recorded as a correction backed by a verification receipt; it is used for learning only through that receipt, and only while your check is still a certifier.

Your check decides what is right; ultrarouter never uses its own guess, the model's answer or a label of its own as the truth. A pass or a fail is never turned into a label: what is kept is the answer your check passed (your served answer, your application's retry, a resample or the teacher's answer), recorded as an inferred correction backed by the receipt — never as one you declared.

Spend is held before any model call and settled after. A call that succeeds is charged its real cost. A call that fails — an error, a timeout, a dropped connection — may still have been billed by the provider, so it is charged the usage the provider reported for it, or, when the provider reported none, the most that call could have cost (the amount held for it). A failed call counts toward your budget and toward the paid-repair limit below like any other, and that answer is not repaired again under the same grant (never a free second attempt). Calls on your own provider keys count against your budget too. A repair pass verifies up to 25 answers and runs at most 10 paid repairs (steps 3–4); answers your check passes at step 1 or 2 cost nothing and do not count toward that limit. Repairs pause on their own — with a named reason — when the budget is used up, when paid repairs stop working (after 30 of them, if the verified share is too low to be worth it), when your check keeps erroring, or when a canary shows your check passing a planted failure.

Push and platform checks can repair today. Pull checks cannot yet: their verdicts arrive after the repair pass, and a paid candidate would have to be kept until your worker answers. Until then, register a push endpoint (or a platform check) for repairs; a pull check still qualifies and still answers canaries and tests.

If your system knows the right answer itself — for example the queue a ticket was finally routed to — do not use a check for it: report it as a declared correction with its final body. That is a declared reference, used for learning while corrections consent is on and deleted when you revoke it.

What this does not do yet

The dashboard card for connecting a check and choosing the budget, a view of the repairs made, and the SDK helpers (ur.oracles.register, ur oracle serve) come later.