Holdout and lift

The Flywheel only claims an improvement when it can prove one, and proving one needs an untouched comparison. This page explains the standing holdout every workspace carries, the exposure arms a request can land in, and how a candidate becomes a confirmed lift — with the honesty rules that stop a passing test, a deploy or a merged pull request from quietly counting as a result.

The standing holdout is sacred

A fixed fraction of your traffic is permanently assigned to a standing holdout and served the frozen baseline — no learned context, no lever — so there is always a clean baseline to measure against. The default is 10% of traffic; an owner can adjust it within a [5%, 30%] band.

Assignment is deterministic and cluster-stable. It is an FNV-1a bucket of (workspace_id, cluster_key), where the cluster key is the first of these that the request has:

1. the artifact key (ur.artifact / X-UR-Artifact);
2. the trace id (ur.trace_id);
3. for an untagged request, a digest of the conversation's opening: every message up to and including the first user message;
4. the request id, only when the request has no user message at all.

So every request of one artifact or trace shares the same arm, and an artifact never lands in a mixed state merely because two of its requests hashed differently. An untagged chat agent gets the same guarantee per conversation. A chat client resends the whole history on every turn, so every turn of one conversation carries the same opening and lands in the same arm. The system prompt part of the opening is masked, so a system prompt that embeds the current time does not split a conversation. The digest is used only to pick the bucket. It is never stored or logged.

Two limits come with the untagged key. Two different conversations that open identically (same system prompt, same first message) share one arm, which makes the cluster coarser but never splits a conversation. A client that drops or rewrites the first user message mid-conversation starts a new cluster from that turn. When you need exact per-task clusters, send ur.trace_id (or an artifact key). A tagged request is keyed on its tag exactly as before, whatever its messages.

Math.random is forbidden in the flywheel code; the bucket reuses the same seeded primitive the shadow sampler uses.

The one rule that matters: no lever code path ever reads holdout traffic except to skip it. This is not a runtime filter someone can forget — it is a type boundary (see The exemplar path is arm-typed below).

The exposure arms

Every request is stamped, at dispatch and immutably, with an exposure_arm and an exposure_epoch. An artifact derives its arm from its requests. The arms are a closed set:

The exposure_epoch is recorded so that lift is never pooled across epochs: if the holdout fraction changes, membership can move, and comparing observations from two epochs would be comparing two different experiments. The code throws rather than combine them.

The exemplar path is arm-typed

The exemplar index that feeds the learned-context lever is arm-typed at its definition, not by a filter at each read site. Its container (packages/core/src/flywheel/learned-context.ts) keys on the teaching arms only — treated | pre_lever | imported — so a holdout or lever_holdout candidate is a compile error, unrepresentable rather than merely rejected. The defence-in-depth bucketer fails closed: handed a non-teaching arm it throws (HoldoutFactNotRetrievableError), because a silent drop would be indistinguishable from "there were none." A holdout leak is not something you have to test for at every call — it cannot be expressed.

Rollups are computed per arm, never pooled

Attribution and calibration are computed per cell — workspace × claim type × situation class × exposure arm × source — and arms are never merged. This too is enforced by construction: the rollup entry point takes an arm-keyed input and stamps each cell's arm from its bucket key, so two arms cannot collapse into one cell. Declared, inferred and imported sources are rolled separately. Every interval reuses the one statistics implementation (packages/core/src/evals/stats.ts — Wilson, cluster bootstrap); a second stats engine is forbidden. Below a minimum sample (n < 20) a derived statistic reads insufficient_evidence — never a computed-looking number — while raw counts and the censored share are always reported (an unresolved claim is never dropped).

How a lift is proven

A candidate does not become an improvement because a metric moved. It has to clear a sequence of type-distinct evidence, and each kind is kept separate so a weaker one can never be presented as a stronger one:

1. Offline screening — our replay on a disjoint split. A screening pass is not a live result.
2. Application CI — your own test/replay harness, in your own CI, binding real application code. A passed check is not a live confirmation, and a raw model-call comparison can never claim application coverage.
3. Live confirmation — a randomized contrast against the frozen baseline. The candidate is served to the treated arm; the lever_holdout slice is served the frozen baseline. Only this can confirm improvement.

The application-CI guide walks stages 2 and 3 end to end; the code that keeps the three from blending is staged-evidence.ts.

The quality bar: superiority

A quality-lift claim requires the lower bound of the confidence interval to be above zero — a superiority verdict, not merely a point estimate that moved. The lift ledger renders only as an interval with its sample size. Before any lift is read, an A/A false-positive check (K = 20, false-positive rate ≤ 0.10) must pass; a violated A/A parks the instrument rather than reporting a result. The minimum detectable effect (MDE) is a power diagnostic — it tells you whether the experiment could have seen an effect — and is never itself the verdict.

The cost bar: explicit decision intent

Cost is judged under named decision-intent contracts, layered over the same evidence laws that govern Bench and the savings ledger. A cost-optimization verdict keeps its existing lower-task-cost requirement; a required migration or a product-improvement decision may use an explicit total-cost budget — but a budget pass is never relabelled as savings, and the frozen cost bound is never silently replaced by a customer budget. Cost and quality always refer to the same retained model invocation, over the complete execution envelope (tools, response format, generation parameters, streaming, provider policy) — dropping any of those makes the comparison ineligible.

What a lift is not

These are laws, and the product surfaces state them plainly:

• A merged PR is not measured improvement. A code change that passes your CI is evidence your app still works, not evidence the model got better.
• A deploy under a running confirmation never becomes the result. An application deploy or rollback that changes the bound baseline invalidates an open confirmation (a named stale_head / app_config_changed / epoch_changed) rather than crediting an unrelated change as lift.
• A mixed artifact is ineligible. Its requests disagree on their arm, so it can never anchor a contrast.
• The holdout is never the treatment. No adopted lever, no learned module, ever reaches a holdout request.

Honest words follow from all of this: any claim of measured learning or improvement is stated with its scope, its holdout interval and its n nearby. A collecting or insufficient_evidence state does not imply a lift — it says, plainly, that none has been established yet.

Next steps

• Read The Flywheel for how capture, confirmation and adoption fit together.
• Walk Confirm a candidate in your own app for the rollout gate, live confirmation and the adoption lease.
• See The training-ready export for what leaves the holdout-protected system as a dataset you own.