6krrt/plans/ten-thousand-foot-review.md

# A 10,000ft review: what this router needs to be a more useful tool

Status: reference -- a diagnosis, not a queue item; the executable work it
feeds is scoped in `plans/token-waste-waves.md`.

**Measured on the LIVE database** at `/home/alee/Sources/6krrt/router.db`
(read-only), 2026-09-13, at repo state `26398d7`. An earlier draft of this file
measured a stale 11:47 copy of `router.db` sitting inside the
`profiles-header-fix` worktree and drew a wrong headline from it; that copy is
~10 hours behind and missing the traffic that matters most. If you are
measuring from a worktree, check the mtime first.

Audience: the operator, and whoever scopes the next round of work. Deliberately
does not repeat the item-level work already scoped in
`code-analysis-refactoring-opportunities.md`.

---

## The headline: the router works, and it just proved it

On 2026-09-06 NeuralWatt billed **$19.85 in one day** across 4,129 decisions.
What followed was the most consequential routing change in the project's
history: OpenRouter came back as an allowlist-gated provider and traffic moved.

| day | decisions | billed |
|---|---|---|
| 2026-09-06 | 4,129 | **$19.85** |
| 2026-09-09 | 594 | $0.32 |
| 2026-09-10 | 722 | $0.25 |
| 2026-09-11 | 755 | $0.25 |
| 2026-09-12 | 116 | $0.40 |
| 2026-09-13 | 154 | $0.21 |

Volume held at 600-750 decisions a day. Billed spend fell roughly **50x**.
That is the system doing precisely what it exists to do, and it is the
strongest evidence in the database that the thing is useful.

Two qualifications, and they are where the rest of this document lives:

- **The human did the adaptation, not the router.** The migration was an
  operator decision — add a provider, write an allowlist, let ranking follow.
  The router has no automatic response to a cost shock; `plan_kwh_per_period`
  gates nothing and `max_energy_per_request` is null, so every cost mechanism
  in the system is advisory. It reported the spike well. It did nothing about
  it.
- **The migration moved 73% of traffic onto a provider the router cannot
  measure.** That is the single most actionable finding here, and it is new
  since the migration.

## The thesis

This project is unusually good at recording *why* a decision was made, and has
no mechanism for noticing when a recorded reason has expired.

| setting | the comment's own premise | what is true now |
|---|---|---|
| `quality_tolerance: 0.1` | "scores currently rest on 2-3 samples per category, so a 0.05 gap is indistinguishable from sampling variation... **Narrow it as samples accumulate**" | 46 rows per category, all outcome-backed; `outcome_blended` rows average 51.8 client samples, max 460 |
| cost-as-tiebreak, not an objective | "60% of every decision adjudicated fractions of a cent (**all real traffic to date totals $0.07**)" | **$118.79** billed. 1,697x the figure the whole argument rests on |
| `assumed_cache_rate: 0.917` | measured once, 2026-08-23, over 40.7M tokens | `cached_prompt_tokens` is NULL in **all 35,086 rows**. The router receives this field and drops it |
| cost feedback generally | catalog estimates "track the real ordering" because billing is capped at 3x list | true for NeuralWatt, untestable for OpenRouter: **95.7% of its rows have NULL `cost_usd`** |

Four settings, four expiry conditions met, nothing re-fired. That pattern is
worth fixing before any individual number. The documentation is a real
strength — the reasoning is preserved, measurements are dated, caveats are
honest. What is missing is anything that *acts* when a premise the code depends
on stops holding.

---

## 1. The provider that now carries the traffic is the one the router cannot see

Recent picks, since 2026-09-09 (2,341 decisions):

| provider | model | n |
|---|---|---|
| openrouter | `xiaomi/mimo-v2.5` | 744 |
| openrouter | `deepseek/deepseek-v4-flash` | 659 |
| neuralwatt | `deepseek-v4-flash` | 266 |
| neuralwatt | `qwen3.6-35b` | 247 |
| openrouter | `nvidia/nemotron-3-nano-omni-...:free` | 178 |
| neuralwatt | `qwen3.6-35b-fast` | 119 |
| openrouter | `z-ai/glm-5.3-flash` | 95 |
| openrouter | `xiaomi/mimo-v2.5-pro` | 26 |

**OpenRouter is 72.8% of current decisions.** Telemetry coverage in
`energy_observations`:

| provider | rows | NULL `cost_usd` | NULL `duration_seconds` |
|---|---|---|---|
| neuralwatt | 31,188 | 66 | 66 |
| **openrouter** | **3,838** | **3,673 (95.7%)** | **3,838 (100%)** |
| ollama-local | 60 | 60 | 60 |

So on the majority of current traffic the router has no billed cost to
calibrate its estimator against and no completion time at all. Every feedback
loop that makes this system smarter than a static config is blind on the
provider it now prefers.

This inverts the priority of everything below: recalibrating the cost model
(item 3) cannot work for 73% of traffic until this is fixed. OpenRouter returns
usage and generation-cost data; it is an ingestion gap, not a provider
limitation.

## 2. `feedback.py` is the only loop without a timer

`proficiency` was written in a single batch at **2026-09-10T00:44:55** — every
row, one instant, three days ago. Since then:

| | |
|---|---|
| client outcomes applied | 2,224 |
| client outcomes **unapplied** | **95** |

And the unapplied 95 are exactly the current traffic:

| model | provider | unapplied | of which failed |
|---|---|---|---|
| `xiaomi/mimo-v2.5` | openrouter | 41 | 9 |
| `deepseek/deepseek-v4-flash` | openrouter | 36 | 15 |
| `deepseek-v4-flash` | neuralwatt | 17 | 3 |
| `xiaomi/mimo-v2.5-pro` | openrouter | 1 | 0 |

Meanwhile **every OpenRouter model's proficiency row is `outcome_prior` with
`outcome_samples = 0`** — an inherited peer-rate prior, not a measurement.
`xiaomi/mimo-v2.5` is the most-routed model in the catalog right now (744
picks) and scores 0.749 / 0.770 / 0.894 on coding_general / coding_refactor /
tool_use_agentic entirely on inheritance. Its 41 real outcomes (78% pass) are
sitting in `verifications` unapplied.

`deploy/` ships timers for the poller, the seed sweep, backup and offsite sync,
and `~/.config/systemd/user` additionally has a baseline-report timer. There is
**no feedback timer**. The loop the docs call "the only ground truth" is the
only one that has to be cranked by hand.

This is the cheapest high-value fix in the document: a oneshot service and a
timer, matching the four that already exist.

## 3. The cost estimator is wrong in a model-dependent way

`routing.estimated_cost` breaks every tie, and the ranking reduces to it
whenever candidates land in the same quality band. Per request, on rows with a
real billed figure:

| model | provider | est µ$ | billed µ$ | est/billed |
|---|---|---|---|---|
| gemma-4-31b | neuralwatt | 3,822 | 6,041 | **0.63x** |
| kimi-k2.7-code-fast | neuralwatt | 28,623 | 21,747 | 1.32x |
| kimi-k3 | neuralwatt | 87,910 | 42,388 | 2.07x |
| deepseek-v4-flash | neuralwatt | 4,154 | 1,796 | 2.31x |
| glm-5.3 | neuralwatt | 48,232 | 20,056 | 2.40x |
| `deepseek/deepseek-v4-flash` | openrouter | 3,830 | 1,383 | 2.77x |
| kimi-k2.7-code | neuralwatt | 22,253 | 3,906 | 5.70x |
| qwen3.6-35b | neuralwatt | 4,091 | 484 | 8.46x |
| `qwen3.6-35b-fast` | neuralwatt | 3,773 | 282 | 13.37x |
| `xiaomi/mimo-v2.5` | openrouter | 9,689 | 715 | **13.55x** |

Total: $401.99 estimated against $118.79 billed. The scale error does not
matter; the **21x spread in the error** does, because it reorders candidates.
`gemma-4-31b` is the estimator's cheapest model in the catalog and billing's
most expensive per request of the ten. The router's current favourite,
`xiaomi/mimo-v2.5`, is the one it overestimates most — it is winning on
quality *despite* the cost model, not because of it.

The cause is already written down in `CLAUDE.md`, two sections apart, never
reconciled: attribution ratio spans **750x between models** and is "most of
the real cost difference in the catalog", while `estimated_cost` prices from
catalog token prices, which contain **no attribution term at all**.

Cost moved off measurement to fix a real problem — a 400-token sweep cannot
price a 150k-token workload — and in the move discarded the term the project
had already proven was dominant. Third recurrence of one pattern: list price
ranks models backwards; the reference sweep ranks them backwards for real
traffic; catalog pricing misorders them because it omits pool concurrency.

**The fix needs no sweep.** 31,119 NeuralWatt rows carry a real billed figure,
joinable to decisions by `request_id`. A per-model correction factor over a
trailing window, refreshed by the poller, recalibrates against the bill and
keeps the per-request shape sensitivity that motivated the move. For OpenRouter
it needs item 1 first.

## 4. Switching models mid-session costs 2.5x, and the estimator prices it as free

No sticky routing, no concept of the previously-used model. On turns with
prompts over 20k tokens and a real billed figure:

| turn | n | avg prompt | µ$ / prompt token | spend |
|---|---|---|---|---|
| same model as previous | 11,246 | 100,994 | 0.037 | $47.91 |
| **switched model** | 858 | 90,309 | **0.091** | **$9.68** |

2.46x, on *smaller* prompts, so the effect is understated. Switch turns are
7.1% of turns and **16.8% of joined spend**.

The mechanism is not mysterious: a switch lands a ~100k-token prompt on a
provider that has never seen it, so the prefix cache is cold and ~92% of the
prompt is suddenly billed fresh. But `rank_candidates` prices every candidate
at `assumed_cache_rate: 0.917`, including the one it is about to switch to,
where the true rate is 0.

Pricing the incumbent at the assumed rate and every challenger at 0 is a small
change that pays for itself, and it makes the router's own decision an input to
its own cost model for the first time.

## 5. Latency is the axis the operator feels and the one nothing optimizes

NeuralWatt, where it is recorded at all:

| model | avg s | max s | tok/s |
|---|---|---|---|
| gemma-4-31b | **22.49** | **1,251.1** | 22.6 |
| glm-5.3 | 7.45 | 163.7 | 143.0 |
| qwen3.6-35b | 4.63 | 172.5 | 99.4 |
| kimi-k2.7-code | 3.65 | 226.9 | 93.5 |
| deepseek-v4-flash-flex | 1.90 | 14.8 | 210.5 |

An 11.8x spread in the mean and a 21-minute worst case. `latency_tolerance`
exists but only gates `-flex` rows; `duration_seconds` is read by nothing in
routing. And per item 1 it is **not collected at all** for OpenRouter or
`ollama-local` — so for 73% of current traffic there is not even a number to
ignore.

`gemma-4-31b` is where every signal disagrees at once: the estimator's
cheapest, billing's most expensive per request, and 6x slower than the field.

## 6. 96.6% of requests are never classified

The classifier is the conceptual center of the system — three backends, a
four-step failure cascade, its own circuit breaker, a degraded-share warning in
`/metrics`, an attribution-exclusion rule in `report_outcome`, and a
fine-tuning roadmap. Last five days:

| `classification_source` | share |
|---|---|
| `cached` | **96.6%** |
| `classifier` | 3.2% (avg 1,129 ms) |
| `override` / null | 0.3% |

The consequence is not only that the machinery is oversized. `task_category` on
a given turn is **whatever the session was doing at the last cache refresh**,
not a classification of that turn — one session ran 3,724 turns and cycled
through all nine categories. So ~96% of turns carry a borrowed label, and
`POST /outcome` attributes results to `(model, task_category)`. The quality
table is largely trained on labels computed for a different turn, and nothing
measures that noise.

## 7. The feedback loop worked, and the tolerance band was never narrowed

`CLAUDE.md` is stale here in the good direction. The categories it records as
"flat at 1.00 awaiting real traffic" are nothing of the kind:

| category | rows | spread | outcome-backed |
|---|---|---|---|
| coding_general | 46 | 0.375 - 0.843 | 46/46 |
| coding_refactor | 46 | 0.423 - 0.846 | 46/46 |
| debugging | 46 | 0.307 - 0.836 | 46/46 |
| tool_use_agentic | 46 | 0.298 - 0.918 | 46/46 |
| reasoning_math | 46 | 0.000 - 0.956 | 46/46 |
| summarization | 46 | 0.250 - 0.556 | 46/46 |
| docs_writing | 46 | 0.607 - 0.949 | 46/46 |

2,319 client outcomes have arrived. The design bet — a 43-task benchmark is the
prior, real traffic is the posterior — paid off comprehensively. Note also that
scores came *down* as evidence thickened (`coding_general` topped out at 0.947
three days ago, 0.843 now) and that `summarization` now caps at 0.556: nothing
in the catalog is good at it, which no benchmark had revealed.

`quality_tolerance` is still 0.1, on a comment that says to narrow it as
samples accumulate. Two things compound: the top band is 0.1 wide, so models
differing by up to 10 points of **outcome-backed pass rate** are treated as
equal; and the tiebreak inside that band is the estimator from item 3, wrong by
up to 13.6x and misordered by 21x. The system trades up to 10 points of
measured quality for a cost saving it cannot compute correctly.

Caveat in the other direction: 379 of 474 proficiency rows are `outcome_prior`
with zero direct samples, including every OpenRouter row. The band is too wide
for the rows that *are* measured and arguably right for the ones that are not,
which is an argument for making the band depend on sample depth rather than
picking a new constant.

## 8. Structural verification is a 99% no-op on this workload

| kind | verdict | n |
|---|---|---|
| structural | **unverifiable** | **28,958** |
| structural | malformed | 218 |
| structural | ok | 64 |
| structural | truncated | 23 |
| local_llm | ok | 480 |
| local_llm | malformed | 47 |
| client_outcome | succeeded | 1,840 |
| client_outcome | failed | 479 |

98.96% of structural verdicts are `unverifiable`, and that is *correct*
behavior: agent turns end in tool calls, and `has_tool_calls` short-circuits
both checkers, which is the documented fix to a real false-failure incident.
The conclusion not drawn is that after that fix the free structural checker
checks nothing on the only workload this router serves. 305 of 29,263 verdicts
were substantive. All of the subsystem's value is in `client_outcome`, the
channel added last — and 527 `local_llm` verdicts have never been applied to
anything.

## 9. The router has no concept of what a model is, only numbers on an id

On 2026-09-06, before the allowlist existed, the ranker selected
`google/lyria-3-clip-preview` — a music generation model — 38 times for coding,
refactoring, debugging and summarization, **17 of them as the ranked winner
rather than an exploration gamble**. No `energy_observations` rows exist for
any: all 38 failed at dispatch.

The allowlist fixed the symptom the same day and is the right pragmatic gate.
But it is manual curation standing in for a structural absence: every filter in
the system is a threshold on a metric, and none asks whether a candidate is a
chat model. A new catalog row arrives with no proficiency — deliberately
treated as unproven rather than bad — and is immediately eligible to win.

---

## Paradigms being shoehorned

### A. "Classify, then dispatch" is a single-request frame on a conversation

The unit of work is a session: hundreds to thousands of turns sharing a
monotonically growing prefix, ~92% of which is cached. The router models each
turn as an independent classify-and-route problem, then bolts on a session
cache to make that affordable — and the cache is now 96.6% of the answer.

Inverting the frame dissolves several items at once. If the session is the unit
and a turn is a delta: classification becomes "has the task changed?", cheap
and usually no, instead of a ~1.1s full-taxonomy inference; the cache stops
being an optimization and becomes the model, making its staleness window a real
parameter rather than a cost hack; switching models becomes a visibly expensive
act (item 4); and `min_tool_proficiency` becomes expressible, because "can this
model be trusted with tools" is a session-level property — which is how the
docs already describe it.

### B. Quality as a per-(model, category) scalar is a leaderboard paradigm

One number per model per category, inherited from benchmarks. The project has
already found twice that this cannot hold what it learns.

`deepseek-v4-flash` scored 1.00 on all three coding categories and 0.33 on
`tool_use_agentic`, and the documented reason is a *failure mode*: given both
times in "it is 1:20pm and my meeting is at 3pm", it calls two tools instead of
subtracting. That is not a lower skill level on a category; it is a disposition
that is hazardous whenever tools are on the table, whatever the task is.

Flattening a mode into a category score forced a second mechanism to express
the real rule — `routing.min_tool_proficiency`, a hard filter — and that filter
is `null` because the flattening made it unusable: opencode sends `tools` on
nearly every request, so switching it on excludes the cheap model from all
traffic. The paradigm mismatch is what left a known hazard ungated.

The inheritance structure shows the same strain from the other side: 379 rows
are peer-rate priors carrying a number with zero evidence behind it, and
nothing downstream distinguishes a 0.749 that was measured from a 0.749 that
was copied — except `source`, which the ranker does not read.

### C. Energy and carbon are structurally central and functionally vestigial

`eco` is "not an objective." `max_energy_per_request` is null.
`plan_kwh_per_period` gates nothing. The project's own conclusion is that
billed energy "ranks how busy the provider was, not how efficient the model
is."

Around that sit: two columns and a comment block in `energy_observations`, a
BTU column kept "purely for comedic dashboard value", `seed_energy.py`, a
6-hourly timer whose accumulation is the stated prerequisite for trusting the
eco ordering, `seed_local_dispatch_energy.py`, grid-intensity and
`carbon_source` logging, open follow-up item 3, and a large share of a 71KB
`CLAUDE.md`.

The measurement work was worth doing — the per-kWh billing discovery and the
attribution decomposition are the most valuable findings in the project, and
item 3 above is an argument for using them *more*. But they belong to the cost
model now, not to an eco objective nothing optimizes. And the provider now
carrying 73% of traffic reports no energy at all, which makes the framing
actively misleading about what the router can see.

---

## Scale check

| | |
|---|---|
| Python | 55,711 lines; `dispatcher.py` 4,974, `admin.py` 2,498, `metrics.py` 2,474 |
| tests | 1,155 |
| config | 129 leaf keys across 22 sections |
| `CLAUDE.md` | 71 KB |
| `plans/` | 62 documents |
| purpose | route one operator's coding agent |

The engineering quality is high in ways that are rare and worth keeping: the
schema-drift and warnings tripwires, the `inherited_from` migration that fails
safe on NULL, the refusal to fabricate leaderboard priors, the four recorded
harness bugs that scored the rig rather than the model. None of that is the
problem. The problem is the absence of any forcing function that retires
machinery, or that re-fires a premise — 62 plan documents and no mechanism that
closes one out.

---

## What it needs, ranked

1. **A `feedback.py` timer.** One oneshot service and one timer, matching the
   four that already exist. The ground-truth loop is the only one cranked by
   hand, `proficiency` is three days stale, and the 95 unapplied outcomes are
   exactly the models carrying today's traffic. Cheapest item here by a wide
   margin.

2. **Capture OpenRouter cost and latency telemetry.** 95.7% of its rows have
   NULL `cost_usd`, 100% have NULL `duration_seconds`, and it is 73% of current
   decisions. This is a prerequisite for items 3 and 5, not a parallel task.

3. **Calibrate `estimated_cost` against the 31,119 billed rows already in the
   DB.** A per-model correction factor over a trailing window. Fixes the 21x
   misordering without a sweep and without abandoning per-request shape
   sensitivity.

4. **Price the switch.** Charge challengers a cold-cache rate in the tiebreak.
   ~17% of spend sits on 7% of turns, and it makes the router's own decision an
   input to its own cost model.

5. **Put latency in the objective.** A p50/p95 per model and a latency term or
   ceiling for `INTERACTIVE` requests. Needs item 2 for most of the catalog.

6. **Make `quality_tolerance` depend on sample depth** rather than replacing one
   constant with another. 95 of 474 rows are measured, 379 are inherited
   priors; a band that is right for one is wrong for the other.

7. **An automatic response to a cost shock.** The 09-06 spike was handled well
   and handled by a human. The minimum honest version: a period spend ceiling
   that narrows the candidate set as it is approached, then refuses with a clear
   reason. Not "the router is unusable without it" — it demonstrably is not —
   but it is the difference between a tool that reports a problem and a tool
   that responds to one.

8. **Decide what the classifier is for, given 96.6% cached.** Either accept it
   as session-level and simplify the machinery to match, or make per-turn
   classification cheap enough to actually run. The current state pays the
   architecture cost of per-turn classification and gets session-level labels.

9. **Retire the eco objective explicitly.** Move the energy and attribution
   findings into the cost model where they are load-bearing; archive the rest —
   the seed timer, the eco scoring path, the BTU column, open item 3.

10. **A premise-expiry mechanism.** The thesis. Every setting whose comment says
    "revisit when X" should have X as a check that fails loudly — a test, a
    `/metrics` warning, a line in the poller. Four have expired silently.

Items 1-5 are independent and small, and 1 and 2 should land first because
everything downstream reads what they produce. Items 6, 8, 9 and 10 are
decisions rather than tasks, and should be made before more machinery is added
to the subsystems they touch.