Evidence status

A working API call says nothing about whether learning helps. This page keeps three questions apart, and answers each one only as far as the evidence goes.

1. Is it available? The feature exists and you can call it on the local stack.
2. Is it operationally verified? An ordinary client connects, reports outcomes and takes part in the whole loop through public interfaces only, proven on the real HTTP path against a live gateway, database and workers.
3. Is an improvement demonstrated? Two separate claims:
   • useful adaptation: a learned configuration beats the right no-learning baseline on outcomes or efficiency;
   • continued improvement: a later learned configuration beats an earlier learned one on independent evaluation data.

A successful call, a synthetic example, a published set of lessons or a growing lesson count is never evidence for the third question.

Status

Last reviewed 2026-10-29.

The hosted gateway (https://api.ultrarouter.io) is invite-only and does not offer learning yet, and the SDKs are not on a package registry. The learning preview is available as source on each public SDK repository's feat/learning-loop branch. Build and install that branch's package using Get started. Source availability does not establish learning effectiveness.

On 2026-10-29, the public-client example passed all 26 operational checks over 20 episodes, using the local subscription gateway. It exercised the packaged TypeScript and Python clients, management MCP, outcome receipts, scheduled reflection, randomized evaluation and lifecycle controls. The automatic evaluation retained the incumbent because it did not establish improvement. During the comparison, the candidate served only its randomly assigned share of eligible episodes. Manual activation was used only to exercise rollback; the run ended with learning disabled and no pending reflection. This verifies that bounded local workflow, not production scale or learning effectiveness.

After tightening episode assignment and the handling of missing outcomes, a fresh live check passed 25 checks over 19 episodes, including both SDKs, randomized evaluation and all lifecycle controls. That invocation failed its MCP check because the separate client had not been built. Nine read-only MCP checks subsequently passed against the updated runtime using a deterministic fixture. These are separate pieces of evidence; the failed invocation remains on record. The later live evaluation also retained the incumbent.

What we can say about improvement

Useful adaptation. In development studies on one public tool-calling benchmark, the same agent did better with learned lessons than with none. Those studies followed repeated experimentation on that benchmark's development scenarios, so they are development findings. They are not a measurement of what learning will do for your agent, and not a guarantee.

Continued improvement. We declared in advance a test of whether lessons learned later beat lessons learned earlier, with a minimum margin. The result fell short of that margin, so the test failed. We keep that failure on record, and we do not claim continued improvement.

What would change this. An evaluation on data the learner has not been tuned on, with the decision rule fixed before any outcome is read, and every comparison also run against a simpler one-shot learner so that iterative learning has to earn its complexity. Negative and inconclusive results are kept.

Research checkpoint tests activate their frozen lessons immediately. They do not establish the business effect of the default customer policy, which evaluates candidates before replacing the active version and can spend time waiting or retain the incumbent. That effect needs its own application comparison, with a trustworthy outcome signal and startup included. A simpler one-shot learner can be a useful product result if it offers comparable quality at materially lower cost; the current comparison does not isolate recursion as the cause of any difference. Early-to-late gains would support a claim over those checkpoints, not indefinite improvement.

Reading the evidence for your own agent

Your own call type's evaluations are the evidence that matters for you. Each one records the rule it was judged by (fixed before it started), the two arms, the interval and the decision. Set UR_CALL_TYPE_ID to the ct_... ID returned when you register or list call types:

curl -sS "$UR_BASE_URL/v1/call-types/${UR_CALL_TYPE_ID:?set the call type ID}/learning/evaluations" \
  -H "Authorization: Bearer $UR_API_KEY"

A candidate that does not win keeps the current version. "Inconclusive" means the evidence did not establish the required advantage; uncertainty or missing outcomes can produce that result. It does not establish zero benefit or equivalence, and it is not a reason to retry until the comparison turns green.

What "recursive" means

Outcome, adaptation, evaluation, deployment, new outcome, and around again. ultrarouter adapts the context it serves your agent from the outcomes you report; it never changes model weights.

Methods

The development studies above are recorded in the ultrarouter repository under docs-internal/results/learn-api2/; the continued-improvement test is the study named B3 there. Those records are internal; ask us if you need them.

The bounded operational run and its failed earlier attempts are recorded under docs-internal/plug-and-play/stage2-evidence/2026-10-29/.