6krrt/plans/conceptual-review-premise-and-execution.md

# Conceptual review: is the premise sound, and is execution converging on it?

Status: done -- review of shipped work

**Date:** 2026-09-01
**Scope:** premise and architecture, not bugs. Every number below is measured
from the live `router.db` and the checked-in code on this date; the scripts are
inlined so they can be re-run.
**Verdict:** the premise is sound and the hard part is built. Three structural
problems sit between the current state and the stated goal, and one of them
will actively move routing the wrong way the moment the next obvious step is
taken.

---

## 0. What the evidence base is

| | |
|---|---|
| `route_decisions` | 9,025 rows (9,007 `chat`), 2026-08-24 → 2026-09-01 |
| `energy_observations` | 13,966 rows, **$49.94** billed, **6.51 kWh**, 1,122 gCO₂eq, from 2026-08-17 |
| `verifications` | 13,501 rows; 987 of them client outcomes (816 succeeded / 171 failed) |
| `proficiency` | 141 (model, category) rows |
| tests | **843 pass in 36.8s**, all offline |

This is a real deployment with real traffic, not a demo. The review is worth
doing precisely because there is now enough data to check the design against
itself.

---

## 1. The premise is sound, and the hardest claim in it is now proven

Design doc §1: use a local model to classify the task, then route to the
cheapest/best-fit open-weight model instead of defaulting everything to one
expensive model.

The load-bearing, non-obvious claim in that sentence is that **task category
should change the model**. For a long stretch it demonstrably did not — the
classifier computed a category, the router paid ~10s for it, and
`proficiency` was empty so it changed nothing. That is fixed, and it is
measurable now. Replaying all 9,001 real decisions with per-category
proficiency against a category-agnostic mean:

```
category-aware == category-agnostic: 5,917/9,001 (65.7%)
category changed the winner:         3,084/9,001 (34.3%)
```

**One decision in three turns on the category.** That is the design's central
bet paying off, and it took populating `proficiency`, fixing four harness bugs,
and rebuilding the cost axis to get there. It should be stated as a result.

Two more things are working and deserve to be said plainly before the
criticism:

- **The session cache removed the latency tax.** It landed 2026-08-30 and is
  running at 98%+ hit rate (2026-09-01: 1,402 cached / 28 fresh). The
  "~10s of local overhead on every message" problem in CLAUDE.md is solved for
  the sessions that matter. Fresh classifications average 2,297ms (max 66,319ms
  — one outlier worth a look, but not a design issue).
- **The pure-module discipline is the best thing in the codebase.**
  `routing.py`, `scoring.py`, `tiering.py`, `proficiency.py`, `context_prune.py`
  are I/O-free, take thresholds as arguments, and are exhaustively tested. That
  is why 843 tests run offline in 37 seconds, and it is why every one of the
  measurement-driven reversals in CLAUDE.md was cheap to make. Keep it.

---

## 2. The premise quietly changed, and the docs did not follow

Design §7 names the metric: **"$ saved vs. always-frontier baseline"**, with
the warning that "a router that's cheaper but quietly worse isn't a win."
Nobody has checked the other side of that: a router that's *better but quietly
more expensive* isn't obviously a win either, and that is where this landed.

`baseline_report.py` already tells the story, and it appears not to have been
read recently:

```
$ PYTHONPATH=src python -m baseline_report --since 2026-08-24
  decisions: 9018
  total cost:   actual $113.89   cheapest $51.39   best-prof $166.59
  dominance (selected == cheapest): 4,972/9,018 (55.1%)
  mean proficiency: actual 0.991   cheapest 0.788   best-prof 0.998
```

The router sits at **0.991 of a maximum 0.998** on quality and **2.2x the
cheapest** on cost. It is not a cost/quality tradeoff engine; it is a
quality-maximizer that takes a discount when quality is exactly tied. That may
be what you want — but it is not what §1 says, and it is not what the README
says either ("dispatches to the cheapest/best-fit model").

The comparison that matters more is against the alternative you would actually
have used. Replaying all 9,007 chat decisions against every single-model policy
(same request shapes, same catalog prices, `assumed_cache_rate` 0.917;
`infeasible` = requests exceeding that model's effective window):

| single-model policy | est. cost | vs router | infeasible |
|---|---|---|---|
| `gemma-4-31b` | $18.91 | 0.17x | 1,170 |
| `qwen3.6-35b` | $19.54 | 0.17x | 4,032 |
| **`deepseek-v4-flash`** | **$36.59** | **0.32x** | **0** |
| `kimi-k2.7-code` | $136.41 | 1.20x | 963 |
| `glm-5.2-fast` / `glm-5.3` | $260.20 | 2.28x | 0 |
| `kimi-k3` | $563.97 | 4.95x | 0 |
| *(actual routed)* | *$113.89* | *1.00x* | *0* |

Against always-frontier (`kimi-k3`) the router saves **4.95x** — the design's
own metric, comfortably met. Against "just always use `deepseek-v4-flash`" it
costs **3.1x more**, and deepseek is the only other policy that can serve 100%
of the traffic without a context failure.

Which baseline is honest depends entirely on what you would otherwise have
done. For opencode agent traffic against this catalog, the realistic
counterfactual is not `kimi-k3` — it is deepseek. **The router should report
both baselines, and `baseline_report.py` should grow a single-model column.**

The operational consequence is live right now: **6.51 kWh burned against a
`plan_kwh_per_period` of 6.25.** At roughly $7.67/kWh realized, an
always-deepseek policy would have used about a third of that and stayed inside
the allowance.

---

## 3. Finding #1 — the feedback loop rewards models for not being used

**This is the most important item in the review, and it is a reason not to
take the next obvious step until it is fixed.**

987 client outcomes are sitting in `verifications`; 961 of them unapplied.
`POST /outcome` is correctly identified in CLAUDE.md as "the only ground
truth," so folding them in looks like the highest-value pending action. It is
not, as currently built.

Simulated on a copy of the DB — run `feedback.py` for real, then replay all
9,007 decisions through `select_candidates` + `rank_candidates`:

```
BEFORE   est cost $137.76      AFTER   est cost $179.28   (1.30x)
  kimi-k2.7-code      2,722      glm-5.2-fast        2,243
  qwen3.6-35b         2,646      qwen3.6-35b-fast    1,385
  qwen3.6-35b-fast    1,236      glm-5.3             1,318
  glm-5.3               745      kimi-k2.7-code      1,239
  gemma-4-31b           587      qwen3.6-35b         1,150
  deepseek-v4-flash     500      kimi-k2.7-code-fast   947
  glm-5.2-fast          417      deepseek-v4-flash     596
```

Ingesting ground truth makes routing **30% more expensive** and hands 2,243
decisions to `glm-5.2-fast`, one of the priciest common rows. The mechanism is
visible in the scores. `coding_refactor`, before → after:

| model | before | n | after | n | real outcomes folded in |
|---|---|---|---|---|---|
| `glm-5.2-fast` | 0.994 | 36 | **0.994** | 36 | **zero** |
| `kimi-k2.7-code` | 1.000 | 18 | 0.813 | 80 | 62 |
| `deepseek-v4-flash` | 0.826 | 36 | 0.791 | 135 | 99 |
| `qwen3.6-35b` | 0.900 | 18 | 0.641 | 44 | 26 |

**`glm-5.2-fast` wins the category outright by never having been measured on
real traffic.**

The cause is structural, not a bug: `feedback.py` and `eval_proficiency.py`
both write through `proficiency_store.add_self_eval`, into the *same*
`self_eval_score` running mean, with equal weight per sample. Benchmark tasks
score ~1.00. Real agent turns score ~0.80. So every real sample drags a score
down, and the drag is proportional to how much the model has been used. The
router then routes away from the used model toward the unused one, gathers
outcomes on *that* one, penalizes it, and moves on.

That is a rotation, not a convergence. And because exposure has been
concentrated on the cheap models — they win the cost tiebreak — the rotation is
systematically toward expensive ones.

**Directions (pick one; all are cheap):**

1. **Keep the two sources separate.** Real-traffic outcomes are a different
   measurement from benchmark tasks and should not share a mean. Add
   `outcome_score` / `outcome_samples` columns and blend them explicitly, the
   way `leaderboard` and `self_eval` already are. This also makes the
   ~0.80-vs-1.00 scale difference visible instead of silently mixed.
2. **Shrink toward the prior by sample size.** A score with n=36 and no real
   exposure should not outrank one with n=135 that includes 99 real outcomes.
   Today low sample count is an *advantage*; `self_eval_min_samples` gates the
   leaderboard blend but nothing penalizes thin evidence in the ranking itself.
3. **Compare like with like.** Score a model against the *per-category mean*
   outcome rate rather than absolute, so an 82% pass rate in a category where
   everyone scores 80% is neutral, not a 0.18 penalty.

Until one of these lands, **do not run `feedback.py` on the pending backlog.**
The 961 rows are the most valuable data this project has; spending them through
the current path converts them into a more expensive router.

---

## 4. Finding #2 — exclusions are self-sealing; there is no exploration

`deepseek-v4-flash` scored **0.333 on `tool_use_agentic`** across **3**
benchmark tasks. Blended, it now reads 0.5. With `quality_tolerance: 0.1` that
puts it in band 5 while eleven models sit in band 0, so it loses every
`tool_use_agentic` ranking outright.

Consequence, measured: **deepseek won 0 of 2,512 `tool_use_agentic` decisions,
and has 0 client outcomes in that category.** A 3-sample estimate has
permanently removed the cheapest capable model from the largest category of
traffic, and the design contains no path by which that estimate can ever be
revised.

`config/config.yaml` proposes the experiment that would settle it:

> run with it off, let `POST /outcome` report real pass/fail, and compare
> `tool_use_agentic` proficiency for deepseek before and after

**That experiment cannot run.** `min_tool_proficiency` is already `null` — the
filter is off — and it changes nothing, because the *quality band* excludes
deepseek from the category regardless. The config comment says "with it off,
that advantage applies." It does not.

The general shape: this is a contextual bandit running pure exploitation. Once
a model wins a (category, context-size) cell it wins it forever; alternatives
never accumulate the evidence that would overturn the ranking. The consequence
is worse than a missed saving — it means every proficiency number is
conditioned on the routing that produced it, which is exactly the confound that
makes the outcome data below hard to read.

**Direction:** a small explicit exploration budget. An ε of 2-3% of requests
routed to the highest-cost-advantage *excluded* candidate per category would
have produced ~75 deepseek `tool_use_agentic` samples over this window — enough
to confirm or kill the 0.333 outright — at an estimated cost of under $2.
Alternatively a "probation" rule: any candidate excluded solely by a
proficiency score with `self_eval_samples < 10` gets N requests per week
regardless.

### What the ground truth actually says (with the confound stated)

Client outcomes, Wilson 95% CIs, ≥20 samples:

| model | n | pass |  |
|---|---|---|---|
| `deepseek-v4-flash` | 390 | **82.8%** | [79%, 86%] |
| `kimi-k2.7-code` | 269 | 80.3% | [75%, 85%] |
| `qwen3.6-35b` | 198 | 85.9% | [80%, 90%] |
| `kimi-k2.7-code-fast` | 52 | 80.8% | [68%, 89%] |
| `glm-5.2-fast` | 44 | 88.6% | [76%, 95%] |

Within `coding_refactor`, where three models have real samples:

| model | n | pass | rel. price |
|---|---|---|---|
| `deepseek-v4-flash` | 95 | **81.1%** [72, 88] | 1x |
| `kimi-k2.7-code` | 62 | 75.8% [64, 85] | ~6.8x |
| `qwen3.6-35b` | 25 | 48.0% [30, 67] | ~2.1x |

The cheapest model is at the top, and `qwen3.6-35b`'s CI does not overlap it.

**Read this carefully, not triumphantly.** The comparison is confounded, and
the confound runs *against* the expensive models: context window determines
eligibility, so `qwen3.6-35b` (94k effective) only ever sees short requests
while `kimi-k2.7-code` (192k) takes the 94-192k band and `glm-5.2-fast` (782k)
takes the largest. Bigger context correlates with longer, harder sessions. So
the expensive models are being handed the harder work, and the table cannot
separate "cheaper model is as good" from "cheaper model got easier requests."

That is the point. **The router has ~1,000 samples of its highest-value signal
and cannot draw a conclusion from them, because it never randomizes.** Adding
exploration is what makes this data interpretable, not just what makes deepseek
eligible.

---

## 5. Finding #3 — cost is 93% prompt tokens, and the heuristics gate the other 7%

Decomposing the estimated cost of all 9,007 real decisions at
`assumed_cache_rate: 0.917`:

```
prompt share of est cost:  $105.48  (92.6%)
completion share:            $8.40   (7.4%)

real billed tokens: 1,190,607,380 prompt / 5,880,453 completion  = 202:1
```

Completion price spans 54x across the catalog ($0.28 → $15.00) and decides
**7.4%** of the bill. Cached-prompt price spans 21x ($0.0144 → $0.30) and
decides **92.6%**.

Two consequences:

- **`tiering.cheap_completion_max: 1.00` gates tier 1 on the wrong axis.** Tier
  is the single most consequential filter in the system — it decides the
  eligible set before ranking runs — and it is resolved from completion price
  plus reasoning mode. This is the same substitution the project has now caught
  twice ("list price ranks models backwards"; "cheapness is not a capability
  ceiling"), appearing a third time. For this workload the honest cheapness
  signal is `cost_per_1m_prompt_cached`.

- **Pinch is the highest-leverage lever in the codebase and the least
  instrumented.** It is the only thing that touches the 93%. Post-pinch
  `required_context_tokens` (line 2450 records `measured`, computed on
  `send_messages`) distributes as:

  ```
  p10 49,817   p25 66,097   p50 88,069   p75 127,517   p90 195,683   p99 279,995
  ```

  against `pinch.budget_tokens: 50000`. **The median request ships at 1.8x the
  pinch budget and p90 at 3.9x** — pinch is running and not reaching its own
  target, which makes sense given it may only trim tool results outside the
  last 4 turns while opencode's ~32k system prompt and tool definitions are
  untouchable. Its stats log at `logs.debug` while `logging.level: info`, so
  zero pinch lines exist in seven days of journal. Nothing in `/metrics`, the
  TUI, or the admin dashboard reports pinch effectiveness either.

  A 20% reduction in prompt tokens is worth more than every routing decision in
  this window combined, and right now there is no way to tell whether pinch
  delivers 2% or 40%. **Promote the pinch line to info, and add
  `tokens_saved` / `original_tokens` to `/metrics`.** That is the cheapest
  high-value change on this list.

### Related: the tier ladder does not produce a capability gradient

| tier | decisions | avg est cost | outcome pass rate (approx. join, n) |
|---|---|---|---|
| 1 | 896 | $0.00907 | 76.9% (13) |
| 2 | 6,663 | $0.01337 | 83.6% (1,201) |
| 3 | 1,448 | $0.01155 | **70.4%** (226) |

Tier 3 costs *less* per decision than tier 2 and fails *more*. The failure rate
is the good news — it means the classifier's tier call carries real signal about
task difficulty. The cost figure is the problem: "frontier / high-stakes" is
not buying a more capable model, only a reasoning-enabled subset, because tier 3
is resolved from `reasoning_default_enabled` while deepseek (1M window, 1.00 on
all three coding categories) is pinned at tier 2 and excluded from all 1,448 of
them. That is the tier-1 lesson from CLAUDE.md recurring one rung up the ladder.
*(Outcome join is `session_key` + 5s window, so treat as indicative.)*

---

## 6. Smaller structural notes

**Sample-depth trap, recurring.** CLAUDE.md documents this precisely once
already — `docs_writing` read 0.70-1.00 at n=2 with a model at the ceiling, and
six more passes spread it 0.66-0.97 without touching a task. The same shape is
sitting in four more categories right now:

| category | rows | at exactly 1.00 | mean samples |
|---|---|---|---|
| `reasoning_math` | 16 | 13 | 5.4 |
| `translation` | 16 | 11 | 3.2 |
| `general_chat` | 13 | 11 | 3.5 |
| `summarization` | 16 | 3 | 3.4 |

The lesson was learned and written down; it has not yet been applied to the
categories that still show the symptom. "Try samples before hardening" is
already the project's own guidance.

**"Quality is the objective" is operationally inverted.** 72 of 141 proficiency
rows sit at exactly 1.00, and 98 of 141 (70%) fall inside the top
`quality_tolerance` band. For most requests quality is a tie and **cost is the
sole decider**, with quality acting as a coarse veto on the worst ~30%. That is
a defensible design — arguably the right one — but CLAUDE.md and
`docs/routing.md` read the other way ("there is no weight to tune — quality is
the objective and cost is the tiebreak"), which will mislead anyone tuning it.

**Structural verification is inert for this traffic.** 12,047 of 13,501
verification rows (89.2%) are `unverifiable` — correctly, since agent turns end
in tool calls. It costs nothing (pure Python, post-response), so there is no
harm, but it occupies a verdict-mix panel on the dashboard and a section in the
docs that together imply coverage that does not exist. `feedback.py`'s own
`coverage()` already prints the warning; the dashboard should too.

**`dispatcher.py` is where the discipline stops.** 3,104 lines, with
`chat_completions` at **662 lines** in a single function. Every other module in
this project is small, pure, and injectable; this one holds routing, pinch,
classification, capability checks, local vision, streaming proxy, telemetry
sniffing, and verification scheduling in one call frame. It is the one place
where the next change is expensive, and the one place a reader cannot hold the
whole path in their head. Extracting the pre-dispatch pipeline (prune →
measure → classify → gate → rank) into a pure function taking messages + config
and returning a decision would bring it in line with the rest and would be
directly testable.

**Minor doc drift.** `design/local-llm-model-router.md` §4 still presents the
weighted composite as the scoring model and §9 item 6 still calls proficiency
"the last inert axis" — both superseded. `scoring.composite_score` and
`eco_score` are dead but tested. `README.md:8` and `docs/routing.md`'s
"## Weighted Scoring" heading carry the same stale framing. CLAUDE.md already
declares itself the source of truth, so this is low-stakes, but the design doc
is the file a new reader opens first.

---

## 7. Recommended order

1. **Do not run `feedback.py` on the 961-row backlog** until the exposure bias
   in §3 is fixed. Separate outcome samples from benchmark samples, or weight
   by sample size. This is the one item that will actively degrade routing if
   the obvious next step is taken.
2. **Promote the pinch log to `info` and surface `tokens_saved` in
   `/metrics`.** Cheapest change here, on the axis that carries 93% of the bill.
3. **Add an exploration budget** (ε ≈ 2-3%, or probation for candidates
   excluded on `self_eval_samples < 10`). Without it, no proficiency number in
   the table is interpretable and the experiment `config.yaml` already proposes
   cannot run.
4. **Add a single-model column to `baseline_report.py`.** "vs always-cheapest"
   and "vs always-frontier" bracket the answer; "vs always-deepseek" is the
   decision the operator actually faces.
5. **Re-tier on `cost_per_1m_prompt_cached`**, or drop the price term from
   tiering entirely and keep the context/reasoning signals.
6. Run the eval harness for `reasoning_math`, `translation`, `general_chat`,
   `summarization` — the same six-pass treatment that spread `docs_writing`.
7. Extract the pre-dispatch pipeline out of `chat_completions`.

---

## 8. Bottom line

The premise holds and the machine works. The measurement culture is the real
asset here — this project has overturned its own conclusions on evidence
repeatedly (list price ranks backwards, tier-from-price, benchmark-vs-real cost,
four harness bugs) and each reversal is written down with the reasoning intact.
That is rarer than the router.

The gap is that the loop is not yet closed. Every measurement so far has been
*taken*, then reasoned about, then encoded by hand. The two mechanisms that
were supposed to close the loop automatically — folding client outcomes into
proficiency, and letting the tool-proficiency experiment settle itself — are
both currently pointed slightly the wrong way: one rewards models for being
unmeasured, the other cannot gather the measurement it needs. Neither is a
large fix. Both are the difference between a router that learns from your
traffic and one that is very carefully hand-tuned to it.

---

## Appendix — reproduction scripts

Read-only. Each was run against `router.db` on 2026-09-01 to produce the
numbers above. Save to a scratch dir and run with `PYTHONPATH=src` from the
repo root.

### A. Single-model cost baselines (§2)

```python
"""What would a single-model policy have cost, on the same 9k real requests?"""
import sqlite3, sys
sys.path.insert(0, "src")
from routing import estimated_cost

conn = sqlite3.connect("router.db"); conn.row_factory = sqlite3.Row
models = {r["model_id"]: dict(r) for r in conn.execute("select * from models")}
rows = conn.execute("""
    select required_context_tokens rc, est_cost_usd, selected_model, task_tier
      from route_decisions where kind='chat' and required_context_tokens is not null
""").fetchall()

CACHE, COMP = 0.917, 500
actual = sum(r["est_cost_usd"] or 0 for r in rows)
print(f"decisions: {len(rows)}   actual router est: ${actual:,.2f}\n")
print(f"{'single-model policy':<26} {'cost':>10} {'vs router':>10} {'infeasible':>11}")
print("-"*62)
res=[]
for mid, m in models.items():
    if m["availability"] != "active" or m["access_level"] != "public" or m["latency_class"]=="flex":
        continue
    eff = m["effective_context_window"] or 0
    tot = 0.0; bad = 0
    for r in rows:
        if r["rc"] > eff:
            bad += 1; continue
        tot += estimated_cost(m, r["rc"], COMP, CACHE) or 0
    res.append((tot, mid, bad))
for tot, mid, bad in sorted(res):
    print(f"{mid:<26} ${tot:>9,.2f} {tot/actual:>9.2f}x {bad:>10,}")
```

### B. Prompt-vs-completion cost anatomy (§5)

```python
import sqlite3, sys, statistics
sys.path.insert(0,"src")
conn=sqlite3.connect("router.db"); conn.row_factory=sqlite3.Row
m={r["model_id"]:dict(r) for r in conn.execute("select * from models")}
rows=conn.execute("""select selected_model sm, required_context_tokens rc, est_cost_usd c
   from route_decisions where kind='chat' and selected_model is not null
   and required_context_tokens is not null""").fetchall()
CACHE,COMP=0.917,500
pt=ct=0.0
for r in rows:
    mm=m.get(r["sm"]);
    if not mm: continue
    p=r["rc"]*((1-CACHE)*mm["cost_per_1m_prompt"]+CACHE*(mm["cost_per_1m_prompt_cached"] or mm["cost_per_1m_prompt"]))/1e6
    c=COMP*mm["cost_per_1m_completion"]/1e6
    pt+=p; ct+=c
print(f"prompt share of est cost: ${pt:.2f} ({pt/(pt+ct)*100:.1f}%)")
print(f"completion share:         ${ct:.2f} ({ct/(pt+ct)*100:.1f}%)")
print()
rc=[r["rc"] for r in rows]
rc.sort()
print("required_context_tokens percentiles:")
for q in (0.1,0.25,0.5,0.75,0.9,0.99):
    print(f"  p{int(q*100):>2}: {rc[int(q*len(rc))]:>9,}")
print(f"  max: {rc[-1]:,}   mean: {statistics.mean(rc):,.0f}")
print()
# actual billed vs prompt size
print("actual billed energy_observations, prompt vs completion tokens:")
r=conn.execute("select sum(prompt_tokens) p, sum(completion_tokens) c, count(*) n from energy_observations where task_category!='seed_reference'").fetchone()
print(f"  prompt {r['p']:,}   completion {r['c']:,}   ratio {r['p']/r['c']:.0f}:1   n={r['n']:,}")
```

### C. Does category change the winner? (§1)

```python
"""Does the classifier's CATEGORY output change the routing decision?

Replays every real chat decision twice: once with per-category proficiency
(what ships), once with a category-agnostic mean proficiency per model.
"""
import sqlite3, sys, statistics
sys.path.insert(0,"src")
from config import load_config
from routing import select_candidates, rank_candidates

cfg = load_config("config/config.yaml")
conn = sqlite3.connect("router.db"); conn.row_factory = sqlite3.Row
models = [dict(r) for r in conn.execute("select * from models")]
prof = {(r["model_id"], r["category"]): r["blended_score"]
        for r in conn.execute("select model_id, category, blended_score from proficiency")}
# category-agnostic: mean of a model's blended scores across all categories
avg = {}
for (m,c),v in prof.items():
    if v is not None: avg.setdefault(m,[]).append(v)
avg = {m: statistics.fmean(v) for m,v in avg.items()}

decisions = conn.execute("""select task_category c, task_tier t, required_context_tokens rc,
       latency_tolerance lt, selected_model sm from route_decisions
       where kind='chat' and selected_model is not null and task_category is not null
       and required_context_tokens is not null and task_tier is not null""").fetchall()

def pick(d, use_category):
    rows=[]
    for m in models:
        r=dict(m)
        r["proficiency"] = prof.get((m["model_id"], d["c"])) if use_category else avg.get(m["model_id"])
        rows.append(r)
    cand = select_candidates(rows,
        required_context_tokens=d["rc"], required_tier=d["t"],
        latency_tolerance=d["lt"] or "interactive",
        allowed_access_levels=cfg.routing.allowed_access_levels,
        exclude_stale=cfg.freshness.exclude_stale,
        exclude_deprecated=cfg.freshness.exclude_deprecated)
    rk = rank_candidates(cand, quality_tolerance=cfg.objective.quality_tolerance,
        prompt_tokens=d["rc"], completion_tokens=cfg.objective.assumed_completion_tokens,
        cache_rate=cfg.objective.assumed_cache_rate)
    return (rk[0]["model_id"], rk[0]["cost"]) if rk else (None,0)

same=diff=0; cost_cat=cost_flat=0.0
for d in decisions:
    a,ca = pick(d, True); b,cb = pick(d, False)
    cost_cat+=ca; cost_flat+=cb
    if a==b: same+=1
    else: diff+=1
n=len(decisions)
print(f"replayed {n} real decisions")
print(f"  category-aware == category-agnostic: {same}/{n} ({same/n*100:.1f}%)")
print(f"  category changed the winner:         {diff}/{n} ({diff/n*100:.1f}%)")
print(f"  cost with category:    ${cost_cat:,.2f}")
print(f"  cost without category: ${cost_flat:,.2f}  ({cost_flat/cost_cat:.2f}x)")
```

### D. Simulated feedback fold-in (§3)

Copies the DB first; never mutates `router.db`.

```python
"""Simulate: apply the 1,039 pending client outcomes, then replay routing."""
import sqlite3, sys, os, shutil
sys.path.insert(0,"src")
from config import load_config
import feedback
from routing import select_candidates, rank_candidates

cfg = load_config("config/config.yaml")
db = os.environ["CLAUDE_JOB_DIR"] + "/tmp/sim.db"

def replay(dbpath, label):
    conn = sqlite3.connect(dbpath); conn.row_factory = sqlite3.Row
    models=[dict(r) for r in conn.execute("select * from models")]
    prof={(r["model_id"],r["category"]):r["blended_score"]
          for r in conn.execute("select model_id,category,blended_score from proficiency")}
    ds=conn.execute("""select task_category c,task_tier t,required_context_tokens rc,
        latency_tolerance lt from route_decisions where kind='chat'
        and task_category is not null and required_context_tokens is not null
        and task_tier is not null""").fetchall()
    tot=0.0; mix={}
    for d in ds:
        rows=[{**m,"proficiency":prof.get((m["model_id"],d["c"]))} for m in models]
        cand=select_candidates(rows,required_context_tokens=d["rc"],required_tier=d["t"],
            latency_tolerance=d["lt"] or "interactive",
            allowed_access_levels=cfg.routing.allowed_access_levels,
            exclude_stale=cfg.freshness.exclude_stale,
            exclude_deprecated=cfg.freshness.exclude_deprecated)
        rk=rank_candidates(cand,quality_tolerance=cfg.objective.quality_tolerance,
            prompt_tokens=d["rc"],completion_tokens=cfg.objective.assumed_completion_tokens,
            cache_rate=cfg.objective.assumed_cache_rate)
        if rk:
            tot+=rk[0]["cost"] or 0; mix[rk[0]["model_id"]]=mix.get(rk[0]["model_id"],0)+1
    conn.close()
    print(f"\n=== {label} ===  replayed {len(ds)} decisions   est cost ${tot:,.2f}")
    for m,n in sorted(mix.items(),key=lambda kv:-kv[1]):
        print(f"   {m:<24}{n:>6}")
    return tot

before = replay(db, "BEFORE (current proficiency)")
conn=sqlite3.connect(db)
rows=feedback.unapplied_failures(conn)
feedback.apply_failures(conn, cfg, feedback.summarize(rows), dry_run=False)
conn.close()
after = replay(db, "AFTER folding in 1,039 client outcomes")
print(f"\ncost change: ${before:,.2f} -> ${after:,.2f}  ({after/before:.2f}x)")
```

### E. Client-outcome pass rates with Wilson CIs (§4)

```python
import sqlite3, math
conn=sqlite3.connect("router.db"); conn.row_factory=sqlite3.Row
q="""select model_id m, task_category c,
      sum(verdict='succeeded') s, sum(verdict='failed') f
      from verifications where kind='client_outcome' group by 1,2"""
rows=[dict(r) for r in conn.execute(q)]
def wilson(s,n):
    if n==0: return (0,0)
    z=1.96; p=s/n; d=1+z*z/n
    c=(p+z*z/(2*n))/d; h=z*math.sqrt(p*(1-p)/n+z*z/(4*n*n))/d
    return (c-h,c+h)
print(f"{'model':<22}{'category':<18}{'n':>5}{'pass':>7}   95% CI")
print("-"*66)
for r in sorted(rows,key=lambda r:(-(r['s']+r['f']))):
    n=r['s']+r['f']
    if n<15: continue
    lo,hi=wilson(r['s'],n)
    print(f"{r['m']:<22}{r['c']:<18}{n:>5}{r['s']/n*100:>6.1f}%   [{lo*100:.0f}%, {hi*100:.0f}%]")
print()
print("--- pooled per model (all categories) ---")
agg={}
for r in rows:
    a=agg.setdefault(r['m'],[0,0]); a[0]+=r['s']; a[1]+=r['f']
for m,(s,f) in sorted(agg.items(),key=lambda kv:-(kv[1][0]+kv[1][1])):
    n=s+f
    if n<20: continue
    lo,hi=wilson(s,n)
    print(f"{m:<22}{n:>5}{s/n*100:>6.1f}%   [{lo*100:.0f}%, {hi*100:.0f}%]")
```