The AI reviewer (automated teacher)

Most of the flywheel learns from a human correction: a reviewer edits a draft, and that edit becomes the lesson. The AI reviewer adds a second, automated source of the same signal. It reads one of your drafts, judges it against its own request, and returns a verdict — approve, edit (with a corrected version) or reject. ultrarouter then records that verdict as a machine-verified correction, exactly where a human correction would land.

It is an independent verifier, not a new teaching path. The reviewer never decides on its own that a lesson is good enough to use. Its corrections are audit-only until it earns a qualification against your own human truth (see below), and even then they stay inferred — a reviewer verdict never becomes a declared (human) one, and the reviewer never reads a holdout draft.

What it judges

The reviewer works from a frozen, versioned, hashed rubric (verifier id ai_review, version v1). The rubric is generic, so it fits any workflow. It asks, of each draft:

1. Does the draft satisfy every explicit instruction in the request?
2. Does it stay faithful to the source material supplied in the request?
3. Does it contain any unsupported or fabricated claim?
4. Does it read as publishable as-is?
5. When your call type declares an objective, does the draft meet it?

The rubric text is pinned by a content hash, so a changed prompt can never pass as v1. The reviewer's answer is parsed strictly: a verdict from the closed set, reason codes from a closed list, and — only for edit — a bounded corrected draft. Anything malformed is dropped by name and teaches nothing.

How a verdict is recorded

Each verdict becomes an inferred correction outcome event (accepted_unchanged for approve, edited for edit, rejected for reject). For an approve or edit, ultrarouter also mints a verification receipt (migration 0126) that binds the verdict to the confirmed objective, the exact target request and version, a decision-prefix hash and the replacement body (the draft itself for approve, the corrected text for edit). A reject records the event only — there is no corrected output to teach from.

The reviewer is driven through the gateway with a dedicated, capped key, so its spend is metered and bounded by that key's limit; the job stops cleanly when the limit is reached.

The qualification gate — why corrections stay audit-only first

A machine-verified correction may only teach once the (ai_review, v1) verifier is qualified in ultrarouter's rule-qualification ledger. Qualification is earned, not assumed: the reviewer's verdicts are compared against your human truth (your review-queue corrections and any imported owner decisions) on the same drafts, and the reviewer qualifies only when, defining a "positive" as a draft that needs a correction:

• it names at least 50 positives,
• its precision is at least 0.85,
• the lower bound of a 95% confidence interval on that precision is at least 0.75, and
• its recall is at least 0.50.

Until every bar is cleared, the reviewer's corrections are audit/rank-only — they are recorded for inspection, but no lever and no training set reads them. If the bars do not pass, the report says not_qualified and nothing changes.

The switches you control

• Corrections consent. The reviewer's replacement bodies are stored and can contribute only under your workspace's corrections_consent. With consent off, a correction contributes its edit-magnitude only, and no replacement body is kept. Revoking consent stops new derivations and supersedes existing ones.
• Running it, or not. The reviewer runs only when an operator runs the job for your workspace and call type with a capped key. Not running it is the simplest "off".
• Qualification. Even with consent on and the job running, nothing teaches until the qualification bars pass. A later dispute of a receipt, or an invalidation of the (ai_review, v1) qualification, removes teaching going forward (already-recorded events are never rewritten; their derived artifacts are re-derived clean).
• Holdouts stay sacred. The reviewer never reads a standing-holdout or lever_holdout draft, so turning the reviewer on never contaminates the control arm the flywheel measures itself against.

Running the job

# Review drafts from a completed import, metered through a capped key:
pnpm flywheel:ai-review run \
  --workspace <id> --call-type <ct> \
  --import <fwimp_…> \
  --key-file <path> \
  --model openai/gpt-5 \
  --max-usd 3

# Add --dry-run to size the prompts and the projected spend without any model call.

# Compare the reviewer's verdicts to your human truth and, if the bars pass, qualify it:
pnpm flywheel:ai-review qualify --workspace <id> --call-type <ct>

# See the verdict counts and the current qualification state:
pnpm flywheel:ai-review status --workspace <id> --call-type <ct>

The default reviewer model is openai/gpt-5 — a priced, keyed model that is not your writer and not either of the screen-kit judges, so the reviewer stays independent of the things it judges. Override it with --model.