6krrt/CLAUDE.md

# Local LLM Model Router — project brief

`README.md` documents what is actually built, module by module.
`design/local-llm-model-router.md` holds the architecture and rationale,
including parts still unbuilt. This file is the working state + immediate
next steps, and is the one to trust on what is currently true.

## What this is

A router that uses a local model (served via Ollama) to classify incoming
coding/documentation tasks — category, tier, required context size — and
dispatch each task to the cheapest/best-fit open-weight model on **Neuralwatt
Cloud**, weighted by cost, per-category proficiency, and a per-request energy
ceiling.

Every measurement in this file was taken on one deployment against one
provider account. They are recorded because the reasoning is worth more than
the conclusion, but treat them as observations with a date on them, not as
constants — the catalog, prices, grid intensity and pool load all move. When
a number here decides something, re-run the measurement before trusting it.

Neuralwatt is the only provider. OpenRouter was removed — the `provider`
column and the `(model_id, provider)` primary key stay so a second provider
can be added later without a migration.

## Stack

- Python (chosen over Rust — this is I/O-bound against provider APIs, not
  CPU-bound; iteration speed on the scoring/weighting logic matters more than
  raw execution speed at this scale)
- SQLite for the decision table
- Ollama for local classification, via any OpenAI-compatible endpoint —
  `localhost:11434/v1`, or an Ollama on another machine across a VPN
  (`classifier.base_url`)
- FastAPI for the dispatcher service

## Billing is per-kWh, not per-token — and neither is what scoring uses

**Measured against the live API, 2026-08-11.** Neuralwatt bills a flat
**$8.00 per kWh** and the catalog's `input_per_million` /
`output_per_million` prices are not what this account is charged. Confirmed
across five models; `cost_usd / energy_kwh` came back 8.00 every time:

| model | list $/1M out | completion tokens | billed USD | kWh | $/kWh |
|---|---|---|---|---|---|
| deepseek-v4-flash | 0.28 | 600 | 5.00e-06 | 5.88e-07 | 8.50\* |
| gemma-4-31b | 0.42 | 540 | 3.80e-04 | 4.75e-05 | 8.00 |
| qwen3.6-35b-fast | 1.15 | 600 | 2.53e-04 | 3.16e-05 | 8.01 |
| kimi-k2.7-code-fast | 4.00 | 600 | 2.25e-03 | 2.81e-04 | 8.00 |
| kimi-k3-fast | 15.00 | 539 | 2.17e-04 | 2.71e-05 | 8.00 |

\* rounding — billed cost is quantized to ~1e-06.

The precise rule, validated against all 65 samples of the reference sweep
(61/65 within 2%; the 4 outliers are microdollar rounding, not misses):

```
cost_usd = min( $8.00/kWh x energy_kwh ,  3 x list token price )
```

The ceiling bound in only 2 of 65 samples, both `deepseek-v4-flash` energy
spikes, and matched to the cent: 31 prompt x $0.14/1M + 400 completion x
$0.28/1M = 1.1634e-04, x3 = 3.4902e-04, billed 0.000349.

**List price ranks models backwards.** Not approximately — invertedly:

| | list $/1M | actually billed | gCO2eq |
|---|---|---|---|
| `deepseek-v4-flash` | **0.28** | 9.80e-05 | 8.90e-04 |
| `gemma-4-31b` | 0.42 | **5.02e-05** | **2.32e-04** |

deepseek lists 33% cheaper, costs 95% more, and emits 284% more carbon.

**But cost and eco are NOT the same axis** — the tempting simplification, and
it is wrong. Cost tracks energy, but carbon is energy x the serving region's
grid intensity, and models run in different regions:

| grid | gCO2/kWh | models |
|---|---|---|
| `FI` | ~49-50 | most of the catalog; varies by time |
| `FI` (reported) | 475 | `glm-5.2-fast`, `glm-5.2-flex` |
| `US-MIDA-PJM` | ~442 | the `kimi-k3` family |

A 13.6x spread, so the two axes disagree: `glm-5.2-fast` is the 2nd cheapest
model and only the 6th cleanest; `kimi-k3-flex` draws 3.7x *less* energy than
`kimi-k2.7-code` while emitting 3.6x *more* carbon. Weighting them separately
is load-bearing, and `tests/test_routing.py` pins it.

**Superseded — cost no longer comes from the sweep at all.** `cost` was the
*median* measured USD over the reference sweep. That was measured to be WRONG
for real traffic, because the reference workload is the wrong shape.

The sweep sends a 400-token prompt with a 400-token completion. Real agent
traffic is a 150,000-token prompt with a ~400-token completion and ~92% cache
hits (token-weighted over 50 sessions and 40.7M tokens on 2026-08-23; the
figure was 84% when measured on 2.2M tokens of earlier traffic). The attribution ratio moves with prompt size, so the ranking inverts:

| workload | winner |
|---|---|
| reference sweep (400/400) | `glm-5.2-fast`, 3.2x cheaper |
| realistic (70k prompt, short answer) | **`deepseek-v4-flash`, 5.0x cheaper** |

Same two models, opposite answer. `glm-5.2-fast` sits at attribution 0.006 on
a toy prompt and 0.50 on a 70k one — it batches beautifully on small prompts
and badly on real ones. `deepseek-v4-flash` barely moves (0.21 -> 0.25).

So `routing.estimated_cost` prices each request from **catalog token prices,
scaled to that request's actual shape** (prompt size, assumed completion
length, `assumed_cache_rate`). List price is not what gets billed, but billing
is capped at 3x list, so it tracks the real ordering and bounds it — and on
the one case that was checked live it agrees with the measurement in direction
and magnitude (7.8x predicted vs 5.0x measured). It is also free, needs no
sweep, and refreshes whenever the poller runs.

Three signals said `deepseek-v4-flash` — catalog token price (7.8x cheaper),
NeuralWatt's own published per-request energy (~10x lower), and a live 70k
measurement (5.0x cheaper). Only the 400-token benchmark disagreed. Trust the
workload you actually run.

`eco` still comes from the sweep's median gCO2eq, and is still not an
objective. `flex_cost_multiplier` is gone: a flex row's measured cost already
is its flex cost.

**Open, and worth knowing:** NeuralWatt's model cards publish *gross* energy
(~1.99e-04 kWh for deepseek, ~1.91e-03 for GLM), while the billed figure is
gross x attribution. GLM burns roughly 7x more actual electricity per request
and charges ~5x less, because far more tenants share its GPUs. Anything built
on `eco` inherits that inversion — the attributed carbon figure answers "what
is my share", not "what was burned".

## Energy attribution: signal that looks like noise

Billed energy decomposes exactly:

```
energy_kwh = avg_power_watts x duration_seconds x attribution_ratio
```

`attribution_ratio` is the request's share of a shared multi-tenant GPU pool.
Up close it looks like pure noise — eight rapid identical calls to one model
spanned 20x in billed energy, correlating **+0.997** with the ratio while
power and duration held steady. Two sweeps of the same 13 models with the
same prompt disagreed by up to 36x.

Scoring on the pre-attribution product (`power x duration`) was tried, and it
is **wrong**. Across the sweep:

| | spread |
|---|---|
| median attribution, **between** models | **750x** |
| typical spread **within** one model | **1.8x** |

The ratios are quantized (0.001, 0.25, 0.5, 0.75) — that is serving
concurrency, a stable per-model property, not weather. A model whose GPUs
carry far more concurrent requests genuinely costs less per request, and
that is most of the real cost difference in the catalog: `deepseek-v4-flash`
bills ~1000x under its share of pool gross. Stripping attribution discards a
750x real signal to suppress a 1.8x one.

So scoring reads the attributed figures, and the **median** absorbs what
noise remains. A split-half check on the 7-sample sweep (median of first
three vs last four) shows that working:

- **10 of 13 models agree within 1.4x** — stable enough to route on
- **3 do not**: `kimi-k2.7-code-fast` (29x), `kimi-k3` (14x),
  `glm-5.2-flex` (2.2x). Those need more samples before their position is
  trustworthy.

`dispatcher.gross_energy_kwh` remains as a diagnostic on the identity, not a
scoring input.

### Attribution drifts across hours, so sampling must too

Within about 30 minutes the billed figures reproduce (0.3-1.1x on a
spot-check). Across hours they do not: between two sweeps,
`deepseek-v4-flash` moved roughly 50x and `qwen3.6-35b` about 7x the other
way — enough to **invert their cost ranking**. Attribution tracks pool load,
and pool load tracks time of day.

More samples inside one sweep does not fix this; it measures one moment more
precisely. Coverage across time does. `load_candidates` already takes the
median over ALL `seed_reference` rows, so repeated sweeps accumulate into a
median-across-time for free — hence `llm-router-seed.timer`, which runs a
small sweep every 6 hours.

Until several sweeps have accumulated, treat the eco ordering as provisional.
A single sweep's ranking is one sample of a moving quantity.

**And none have accumulated since 6e729ad.** That commit moved
`log_observation`'s trailing arguments to keyword-only without updating
`seed_energy.py`, so every timer run since spent one billed completion and then
died on `TypeError` — which is not a `RequestException`, so the per-sample
`except` did not catch it. Fixed, and the sweep now has an offline end-to-end
test, but the accumulation this section describes starts from the next run
rather than from months of history.

## What's built and working

- `schema.sql` — `models`, `proficiency`, `energy_observations`. Applies
  cleanly (`sqlite3 router.db < schema.sql`). `models` carries the serving
  class columns (below); `energy_observations` carries real carbon/cost.
- `poller.py` — fetches Neuralwatt's `/models` endpoint (public,
  unauthenticated), normalizes, upserts, marks stale rows. **Verified
  against the live API**: 19 models, and the `metadata.pricing` /
  `metadata.capabilities` / `metadata.limits` field mappings are confirmed
  correct.
- `config.yaml` / `config.py` — weights, thresholds, provider settings,
  Pydantic-validated.
- `scoring.py` — one `normalize_inverted` (cost and eco normalize
  identically; they differ only in what is fed to them) plus the weighted
  composite.
- `seed_energy.py` — runs a fixed reference task N times per routable model
  and writes `energy_observations` rows tagged `seed_reference`. This is what
  makes `cost` and `eco` real numbers instead of the neutral 0.5. Re-run it
  after the catalog gains models: `python seed_energy.py --samples 5`
  (13 models x 5 = 65 calls, and the whole sweep cost **under a cent**).
- `tiering.py` / `tier.py` — pure tier resolver + the DB pass that applies it.
- `routing.py` — pure hard filters and ranking.
- `dispatcher.py` — FastAPI service. `GET /health`, `POST /route` (classify
  and pick, no provider call), `POST /dispatch` (route, call, log), plus an
  OpenAI-compatible `GET /v1/models` and `POST /v1/chat/completions`, and a
  Server-Sent Events `GET /events/decisions` stream that fans every recorded
  routing decision out to the monitoring TUI live.
- `proficiency.py` / `proficiency_store.py` — pure blending arithmetic, and
  the only path that writes the `proficiency` table (so `blended_score` and
  `source` can never drift from the inputs that produced them).
- `events.py` — in-memory decision-event broker (pure stdlib, no Textual): a
  bounded ring buffer of recent route decisions plus thread-safe fan-out to
  SSE subscribers. `persist_route_decision` publishes here after each write,
  so the dashboard sees decisions in near real time without polling. The
  durable source of truth stays the `route_decisions` SQLite table; this is
  just the volatile live fan-out.
- `leaderboards.yaml` / `leaderboard.py` — curated per-family priors and their
  importer. **Ships empty by design**; `python leaderboard.py --check` names
  every active family lacking a prior.
- `evals/tasks.yaml` / `eval_proficiency.py` — the self-eval task set and
  runner. Four scoring kinds, objective wherever the category allows it:
  `code` executes the model's Python against checks, `exact` compares a
  normalized answer, `tool` inspects the tool call structurally, and only the
  four prose categories fall back to a `judge`.
- `capabilities.py` — request-side capability detection. `detect_capabilities`
  reads an OpenAI-format body and returns `RequestCapabilities` (`has_images`,
  `require_json_mode`, `tools_present`, `has_reasoning_request`). It scans all
  messages for `image_url` parts and reads `response_format.type` against
  `{json_object, json_schema}`. Detection is read from the request body, not
  inferred by a classifier.
- `logs.py` — the service's structured logging: a per-request trace id in a
  ContextVar, logfmt rendering, and journald priority prefixes emitted only
  when systemd actually owns stderr. `logs.bind()` exists because a
  StreamingResponse's generator is resumed in a fresh copy of the caller's
  context, so the ContextVar cannot reach it — the path all agent traffic
  takes. Level from `logging.level`, overridden by `LLM_ROUTER_LOG_LEVEL`.
- `metrics.py` / `GET /metrics` — read-only observability aggregation. Keeps
  the dashboard helpers outside `dispatcher.py` to avoid an import cycle:
  `dispatcher` imports `metrics` for `/health` and `/metrics`, so `metrics`
  takes `(conn, cfg)` arguments and never imports `dispatcher`. Returns quota
  burn against `plan_kwh_per_period`, scoring coverage warnings, recent
  `route_decisions`, per-model aggregates over `energy_observations`,
  verification verdict mix, and top proficiency by category.
- `tui.py` — Textual terminal dashboard over `GET /metrics` and
  `GET /events/decisions`. A foreground entrypoint (`python tui.py`), not a
  service. `textual` is imported only in the TUI modules (`tui.py`,
  `tui_screens.py`, `tui_sse.py`), so the router's dispatch path has no UI
  dependency. The dashboard has a live routing-decisions feed (via the SSE
  endpoint, so new decisions appear without waiting for the 5s `/metrics`
  poll), a category → model breakdown panel, and a detail popup (Enter or `e`
  on any decision row) showing the full decision JSON. The pure data layer
  (`build_model`, `build_category_breakdown`, `decision_row`) is split into
  `tui_model.py` so it is testable without a terminal.
- `router_cli.py` — one-shot `/route` probe. Posts a task to the running router
  and prints the decision tree, or emits raw JSON with `--json`. Spends no
  quota because it only routes.
- `tests/` — 562 tests across 27 files, all passing, all offline. Verified on
  Python 3.10 and 3.14; nothing declares `requires-python`, so 3.10 is the
  tested floor rather than a promised one.

### Monitoring: route_decisions persistence

`route_decisions` is the newest observability table (not a scoring input). It
records one row per routing decision — `route` \| `dispatch` \| `chat` |
`passthrough` \| `local_vision` — with category, tier, selected model, runners
up, estimated cost/proficiency, classifier latency, rejection reason, and
request feature flags (`tools`, `images`, `json_mode`, `streamed`). It stores
only a hashed `session_key`; `session_dir`, prompts, and answers are excluded,
and a test enforces that the write path never stores conversation text.

The write is gated by `logging.log_route_decisions` and is **best-effort**: a
failed write is logged and swallowed, because a decision record is worth
having but never worth failing or slowing a request for. The table is created
from code at module load (`_ensure_route_decisions_table`) and again on every
write (`ensure_route_decisions` inside `persist_route_decision`), mirroring
the `proficiency_store.ensure_columns` migration pattern: `schema.sql` is
`CREATE TABLE IF NOT EXISTS`, but a live `router.db` predating this table
needs the code-side migration. Both the module-load hook and the write-path
guarantee are idempotent and leave existing rows intact.

### Request-side capability gates

The router treats some capabilities as hard filters, read directly from the
request body. Two new gates live in `routing.rejection_reason` and
`routing.select_candidates`:

- `require_vision` — active when the request carries `image_url` parts and
  `routing.require_vision` is true. A model passes only if its catalog row says
  `supports_vision = 1`. `supports_vision = NULL` fails closed: an unknown flag
  means the capability cannot be confirmed, and routing an image request to a
  model that might lack vision is a guaranteed provider 400.
- `require_json_mode` — active when `response_format.type` is `json_object` or
  `json_schema` and `routing.require_json_mode` is true. A model passes only if
  `supports_json_mode = 1`; `NULL` also fails closed, for the same reason.

Both gates default to **on** in `config.yaml`, because a wrong guess produces a
400. This is a deliberate asymmetry against the tool-proficiency gate below:
capability **flags** fail closed on unknown, while quality **measurements**
(tool proficiency, energy) admit on absent evidence ("unproven, not bad").

`tools_present` remains a measurement gate, not a flag gate. Every routable
catalog row has `supports_tools = 1`, so that flag would be inert. What matters
is the measured `tool_use_agentic` proficiency. A model with no measured tool
score is admitted; a measured score below `routing.min_tool_proficiency` is
dropped — but only when the request carries a `tools` array. This gated filter
ships disabled (`null`).

`has_reasoning_request` is detected in `capabilities.py` but is informational
only. Reasoning stays on tiering: `reasoning_default_enabled` decides the tier,
not `supports_reasoning` (which only means "the endpoint accepts a reasoning
param" and is true for 17 of 19 rows). There is no reasoning capability gate.

### Pass-through capability check

When a client pins a real model id in `/v1/chat/completions`, the router
dispatches as asked. Before spending a provider call, `_check_pinned_capabilities`
reads the model's `supports_vision` and `supports_json_mode` flags and returns a
clear 422 if the pin cannot satisfy the request. The pin could never have worked,
so failing early is better than an opaque provider 400.

### Local Ollama vision fallback

`local_vision:` in `config.yaml` configures a fallback path for image requests
that find no cloud vision candidate — the cloud catalog excludes the cost
leader (deepseek) on vision, so without this fallback every image request
that would otherwise have routed there 422s instead. It is a **core feature**,
**enabled by default** (`enabled: true`, both in `config.yaml` and in
`LocalVisionConfig`'s own default, so a config that omits the section still
gets it). Disable it explicitly (`enabled: false`) on a host with no local
Ollama, or one that hasn't pulled the vision model.

When enabled and routing returns no selected model, `_run_local_vision` sends
the original messages, with `image_url` parts intact, to a local Ollama model
(`qwen3-vl:4b` by default). The local answer then **replaces** the completion:
`_local_vision_response` returns a normal OpenAI-shaped response, including a
stream-wrapped version for `stream: true`. It does not inject a caption into a
cloud call, because the streaming proxy cannot rewrite bytes mid-stream.

Budget guards refuse images too numerous or too large before the local call is
made. If the local call fails for any reason, it falls through to the ordinary
`422 No model satisfies the hard filters` so the failure is visible rather than a
silent empty response.

### Serving class: one base model, many rows

Neuralwatt ships ~6 base models as 19 catalog rows. The id suffixes are three
**orthogonal** dimensions (hence `glm-5.2-short-fast-flex`), parsed by
`poller.parse_serving_class` into columns:

| suffix | column | meaning |
|---|---|---|
| `-flex` | `latency_class` | discounted async; held server-side during peak until a capacity gap opens |
| `-fast` | `reasoning_mode` | thinking disabled or capped to a short budget — *not* queue priority |
| `-short` | `context_variant` | 200K pool with a bounded reasoning budget |

These rows carry **identical catalog pricing**, so without these columns all
7 GLM rows tie exactly and the router picks arbitrarily — which could send an
interactive request to a preemptible flex row. `latency_tolerance`
(`interactive` | `batch`) is therefore a hard filter in `routing.py`, not a
weight. Suffixes are matched as whole `-`-delimited segments so
`deepseek-v4-flash` is not misread as a `-fast` row.

### Access gating is prose-only

6 of 19 rows are restricted, and the catalog says so only in free text
("Private preview (grant-gated)", "(Canary)") — there is no structured field.
`poller.parse_access_level` parses it into `access_level`, and
`routing.allowed_access_levels` (default `[public]`) excludes them, so the
dispatcher doesn't select a model that 403s. 13 of 19 rows are routable.

### Tiering

Tier on `reasoning_default_enabled` (from `metadata.reasoning.default_enabled`,
falling back to `capabilities.reasoning`), **not** `supports_reasoning`.
`capabilities.reasoning` only means "the endpoint accepts a reasoning
param" — it is true for 17 of 19 rows, and tiering on it put 17 models in
tier 3 and left tier 1 empty. A `-fast` row does not inherit its sibling's
tier 3. Cost is checked before the reasoning rule so $0.28/1M models can
reach tier 1.

**Cheapness is not a capability ceiling** (`tier1_context_max`, default
512000). Tier is a *floor* — `routing.py` drops any row with
`tier < required_tier` — so tier 1 means "simple work only", not "cheap".
Deciding that on price alone put `deepseek-v4-flash` in tier 1 for no reason
but its $0.28/1M completion price, which excluded it **outright** from every
tier-2 request. It has a 1M advertised window and scores 1.00 on all three
coding categories. That was the same substitution the cost axis already had
to unlearn: price is a market signal, not a capability measurement.

Tier 1 now requires the model to be small **and** cheap. The gate reads the
**advertised** `context_window`, whose catalog values are the clean market
classes — 131056 / 199984 / 262128 / 1048560 — rather than
`effective_context_window`, which varies within a class. 512000 sits in the
empty band between the 256K and 1M classes with a 2x margin either side, so it
is not fitted to any one model. The gate only ever demotes; a huge window
never promotes an expensive model into tier 1, and a *missing* window does not
block it, since absent evidence should not decide anything.

Distribution moved **4 / 6 / 9 -> 1 / 9 / 9** — only the three deepseek rows
changed. With cost priced per-request, `deepseek-v4-flash` now wins
`coding_general` at every context size (16.5x cheaper than `kimi-k3` at 200k)
and is still correctly absent from `tool_use_agentic`, where its measured 0.33
drops it out of the quality band. That is the eval data earning it the slot
rather than a thumb on the scale — no `model_tiers` override was needed.

### Why tools and reasoning stay on their existing signals

Tools stay on the measured `tool_use_agentic` proficiency gate, not a
`supports_tools` flag gate. Every routable catalog row already has
`supports_tools = 1`, so a flag gate would be inert. The real signal is the
measured proficiency, because the observed failure is a model over-reaching for
tools on a non-agentic prompt. `routing.min_tool_proficiency` captures that
measurement and only applies when the request carries a `tools` array.

Reasoning stays on tiering (`reasoning_default_enabled`), not on a new flag gate.
`supports_reasoning` only means the endpoint accepts a reasoning parameter, and
that is true for 17 of 19 rows — nearly the whole catalog. `has_reasoning_request`
is detected purely for observation. Making it a gate would add no useful
filtering, because the decision of whether a request needs reasoning is already
encoded in the requested tier.

The fail-closed asymmetry, stated plainly: **capability flags fail closed on
unknown; quality measurements admit on absent evidence.** A missing
`supports_vision` or `supports_json_mode` flag means "cannot confirm", so the
model is dropped. A missing `tool_use_agentic` score or energy measurement means
"unproven, not bad", so the model is admitted. The first wrong guess is a
guaranteed 400; the second is just an empty data point that the neutral default
handles.

## Proficiency: category now changes routing

`proficiency_score` is the ONLY category-dependent term in the ranking, so
until this table had data, `task_category` could not change a decision at
all — the classifier computed it, the router paid ~10s for it, and then it
made no difference. It does now: sweeping 9 categories x 3 tiers currently
returns **5 distinct winners** at both 50k and 120k of context.

| context | winners over 27 decisions |
|---|---|
| 50k | `qwen3.6-35b` (10), `gemma-4-31b` (7), `deepseek-v4-flash` (5), `kimi-k3` (3), `kimi-k3-fast` (2) |
| 120k | `kimi-k2.7-code` (10), `gemma-4-31b` (7), `deepseek-v4-flash` (5), `kimi-k3` (3), `kimi-k3-fast` (2) |

**This spread is recent, and how it got here is the useful part.** For a long
time all 27 decisions returned ONE model, and that was the correct answer at
the time rather than a bug: with cost and eco both populated, `qwen3.6-35b`
was Pareto-dominant — cheapest AND cleanest in the routable set, while
scoring within `quality_tolerance` of the best. No defensible weighting picks
anything else out of that.

Two corrections widened it, and neither was a tuning change:

- **Cost stopped being a benchmark average.** It is now priced per request
  from catalog prices scaled to the request's shape, so the ranking depends
  on the workload instead of on a 400-token reference sweep that no real
  traffic resembles.
- **Tier stopped being inferred from price.** `deepseek-v4-flash` was pinned
  to tier 1 for being cheap, which excluded it from every tier-2 request
  regardless of what any score said.

Note what changes between the two rows above: only the leader, and only
because of the hard context filter. That is the filter working, not the
scoring disagreeing with itself.

**If you see one model win everything again, check for dominance before
reaching for config.** One winner is a legitimate outcome. The lever, if a
genuinely different balance is wanted, is `objective.quality_tolerance` (how
large a quality gap must be before it outranks a cost saving) or
`objective.max_energy_per_request` (a hard ceiling). There is no weight to
tune — quality is the objective and cost is the tiebreak.

### What the task set actually found

**The benchmark could not discriminate these models on coding.** Every row
scored exactly 1.00 on `coding_general`, `coding_refactor` and `debugging` —
and that is after the tasks were deliberately hardened with touching
intervals, full semver, present-but-falsy defaults, late-binding closures and
a binary search that infinite-loops. Every model in this catalog is simply
good at that class of problem, so cost decides coding routes, which is the
right outcome.

**Real traffic broke one of those ties, which the benchmark never could.**
`coding_general` now spans 0.86-1.00: `glm-5.2-fast` fell to 0.862 over 29
samples folded in by `feedback.py` from an actual agent session, and crossed
`self_eval_min_samples` on the way, so it reads `self_eval` rather than
`self_eval_thin`. That is the intended shape of this system — the 23-task
benchmark establishes a floor, and your own traffic is what refines it.
`coding_refactor` and `debugging` are still flat at 1.00, awaiting the same
treatment.

**A 1.00 can also be a sampling artifact, and `docs_writing` was one.** At 2
samples per model the category read 0.70-1.00 with a model at the ceiling, and
the router paid for that ceiling: `kimi-k3-fast` won every docs route. Six more
benchmark passes moved every score and left NOTHING at 1.00:

| model | n=2 | n=11-14 |
|---|---|---|
| `kimi-k3` | 0.85 | **0.973** |
| `kimi-k2.7-code` | 0.85 | 0.886 |
| `deepseek-v4-flash` | 0.80 | 0.864 |
| `kimi-k3-fast` | **1.00** | 0.864 |
| `qwen3.6-35b` | 0.85 | 0.800 |
| `gemma-4-31b` | 0.85 | 0.786 |

The winner moved to `kimi-k2.7-code`, **3.2x cheaper** at 50k of context
($0.0441 -> $0.0136), with no config change — `kimi-k3` scores higher but sits
inside `quality_tolerance`, so cost breaks the tie. `deepseek-v4-flash`
($0.0024) misses the band by 0.009, which is the kind of margin the tolerance
exists to describe rather than a verdict.

**The whole spread rests on one rubric line, though.** `docs_function` is
effectively saturated — 1.00 on nine of every ten samples — and nearly every
`docs_gotcha` deduction is the same omission: the model documents that order is
preserved, that the first occurrence is kept, and what `key` does, then never
says elements must be hashable. That is real discrimination, since it is a real
property of the function, but one sentence is deciding a category. Treat this
ordering as thinner than n=14 makes it look.

**The self-judging guard costs sample density, and it shows up here.** Most
models reached n=14; `kimi-k3` and `kimi-k3-fast` reached only 11, because
those two are the ones diverted to the alternate judge `qwen3.6-35b`, which
returns unparseable JSON more often than `kimi-k3` does. The guard is still
right — a model grading its own family is worse than a thinner sample — but
the alternate judges should be picked for parseability, not just for being
someone else.

The reflex when a category looks flat is to reach for `quality_tolerance`.
Neither tie broken so far was broken that way: `coding_general` opened up when
`feedback.py` folded in real traffic, and `docs_writing` opened up on six more
benchmark passes. Both were samples, not settings. `coding_refactor` and
`debugging` are still flat at 1.00 on 2-3 samples each — which is now a state
this project has mistaken for a measurement once.

Current spread by category, widest first:

| category | spread |
|---|---|
| `tool_use_agentic` | 0.33 - 1.00 |
| `summarization` | 0.60 - 1.00 |
| `reasoning_math` | 0.67 - 1.00 |
| `docs_writing` | 0.66 - 0.97 |
| `general_chat` | 0.80 - 1.00 |
| `translation` | 0.85 - 1.00 |
| `coding_general` | 0.86 - 1.00 |
| `coding_refactor`, `debugging` | flat at 1.00 |

**What does discriminate is tool use, arithmetic traps, and prose.**
`deepseek-v4-flash` scores 1.00 on all three coding categories yet **0.33 on
`tool_use_agentic`** and 0.67 on `reasoning_math`. Verified live, not an
artifact: given "It is 1:20pm and my meeting starts at 3pm, how many minutes
away?" — both times supplied — it calls *two* tools rather than subtracting.
It over-reaches for tools, which is exactly the failure mode that matters in
an agent loop. The router now avoids it for those categories while still
picking it for coding.

Most rows still read `source='self_eval_thin'` (118 of 132): real
measurement, but below `self_eval_min_samples` at 2-3 tasks per category per
run. The 14 that have crossed it are all `docs_writing`, from the six extra
passes above. Two paths thicken it, and they are complementary — re-run
`eval_proficiency.py` to accumulate benchmark samples, or just use the router
and let `feedback.py` fold in real outcomes. Both fold into a running mean
rather than replacing, so samples add up across runs.

### A score is only as fresh as the row it was copied to

Proficiency is a property of the weights, not the queue, so the eval harness
scores one row per family and `propagate_to_variants` copies the result onto
the serving variants — `kimi-k3-flex` gets `kimi-k3`'s number, because no
benchmark rates a `-flex` row separately.

That copy used to happen **exactly once per variant, ever.** The guard skipped
any row with `self_eval_samples > 0`, meaning "measured directly, do not
overwrite" — but inheritance copies the sample count too, so after the first
propagation an inherited row was indistinguishable from a measured one and was
never refreshed again. `kimi-k3-flex` sat at 0.85/n=2 while `kimi-k3` moved to
0.973/n=11.

`proficiency.inherited_from` records the provenance that was missing, and the
**migration** was the delicate half, not the fix: `ADD COLUMN` gives every
existing row NULL, which reads as "measured here", so shipping the guard alone
would have permanently frozen the exact rows it exists to unfreeze. The
backfill infers provenance from the harness's own selection rule rather than
guessing — `eval_identities` only ever evaluates standard rows plus flex rows
with **no** standard equivalent, so a flex row that has one was never a
candidate for direct evaluation, whatever its sample count claims. Everything
else keeps NULL, which fails safe: NULL means "do not overwrite", so no real
measurement can be lost to a wrong guess.

Confirmed on the live database, and on the catalog's one genuine exception —
`glm-5.2` is canary, so `glm-5.2-flex` is the routable row the harness scores
directly, and its NULL is correct.

### Harness bugs this shook out

Three separate defects, each of which scored the rig rather than the model,
and each caught by reading per-task detail rather than the summary:

- **Token budget.** `max_tokens` was shared between a reasoning model's trace
  and its answer. At 1200, qwen3.6-35b spent ~4,200 characters thinking and
  returned an EMPTY content field, scoring 0.00 on tasks it can plainly do.
  Now 24000, clamped per model (gemma-4-31b caps at 16384), and
  `finish_reason: length` skips the sample instead of scoring it.
- **One leading space.** kimi-k2.7-code returns `" def f(...)"`, which becomes
  IndentationError once the harness prepends its imports — 0.00 across all
  nine coding tasks for a model with "code" in its name.
- **Judge failures scored as model failures.** 44% of judge calls returned
  unparseable output (the judge is itself a reasoning model and leaks its
  thinking despite `response_format`). Each was recorded as 0.0. Now the JSON
  is extracted from surrounding prose and an unusable reply yields no sample.

`tests/test_task_set.py` exists so this stops happening: it implements a
reference solution for every `code` task and asserts it passes every check,
recomputes every `exact` answer (one by brute force), and confirms each
refactor target already passes its own checks while each debugging target
fails. It immediately caught a check where the expected value was simply
wrong — which would have docked every model on a task and been
indistinguishable from genuine difficulty.

## Tool competence is read from the request, not guessed at

Neither local classifier can identify agentic work. Asked to label six
unambiguous tool-use prompts ("read the config then update the manifest",
"run the tests and fix what fails"), `qwen3.5` got 2/6 and `mistral-nemo`
1/6 — and `mistral-nemo`'s misses collapse to `general_chat`, which is also
the configured `fallback_category`, so qwen3.5's crashes land in the same
place.

That mattered because `tool_use_agentic` has the widest proficiency spread in
the table (0.33-1.00) and `deepseek-v4-flash` — the current winner on coding —
sits at the bottom of it.

**The fix was not a better classifier.** Whether tools are on the table is
stated in the request: every agent client sends a `tools` array, and
`chat_completions` never looked at it. Reading it is exact and free.

It is applied as a **hard filter**, not a category override, and the
distinction is load-bearing. The question is not "is this task agentic" but
"can this model be trusted with tools that exist". The recorded failure is
precisely the second one: `deepseek-v4-flash` was given a *non*-agentic prompt
("it is 1:20pm and my meeting is at 3pm, how many minutes away?", both times
supplied) and called two tools rather than subtracting. A model that
over-reaches is a hazard on every request where tools are available, whatever
a classifier would have labelled the task.

So `routing.min_tool_proficiency` drops any candidate whose measured
`tool_use_agentic` score is below it, but only when the request carries tools:

| request | winner on `coding_general` @ 50k |
|---|---|
| no tools | `deepseek-v4-flash` ($0.0024) |
| tools present | `qwen3.6-35b` ($0.0041) |

Verified live through `/v1/chat/completions` with identical bodies differing
only by the `tools` array. The cost of safety here is 1.7x on that route,
paid only where tools exist.

**It is currently set to `null`, i.e. OFF**, deliberately and pending
experiment. opencode sends `tools` on essentially every request, so with the
filter on, `deepseek-v4-flash` is excluded from ordinary agent traffic and its
~7x cost advantage goes unused; with it off, that advantage applies and a
model measured at 0.33 on tool use handles requests where tools are on the
table. Which is right is an empirical question and the benchmark cannot
answer it — the 0.33 comes from 3 tasks.

What settles it is `POST /outcome`: run with the filter off, let real pass/fail
reports accumulate, and compare `deepseek-v4-flash`'s `tool_use_agentic`
proficiency before and after. That is the one signal here that knows whether
the work actually worked, and `feedback.py` folds client outcomes in both
directions, so success counts too.

0.5 sits in the empty band between the only two values the catalog holds
(0.33 and 1.00), so it is not fitted to either. A model with **no** measured
tool score is unproven rather than proven bad and is not dropped — the same
rule as the tier-1 context gate. Config load refuses a
`routing.tool_use_category` that is not a real category, because a name
matching nothing yields NULL for every row and NULL means "do not
disqualify": the filter would silently stop filtering.

## Tier is an iteration budget, not just a floor

A tier used to mean only "do not route below this". It now also buys
corrective attempts after a verification failure:

| tier | batch | interactive |
|---|---|---|
| 1 | 0 retries | 0 |
| 2 | 1 | 1 |
| 3 | 2 | 1 |

Interactive is capped below its tier because every retry doubles
time-to-answer, and in interactive use latency **is** a quality loss.

Retries are matched to the failure, since the causes differ:

- **truncated** — raise the token budget on the same model; a different one
  would run out too. If there is no cap to raise, the model's own output
  ceiling is the wall, so escalate to a candidate that can emit more.
- **malformed** — more tokens will not make unparseable output parse, so
  escalate to the next-ranked candidate.
- **ok / unverifiable** — buy nothing. Retrying `unverifiable` would burn
  quota across the majority of prose traffic for no signal.

`escalation.preemptive_on_low_confidence` is now **off by default**. Bumping
the tier because the classifier was unsure pays frontier prices before
anything has gone wrong; spending after a check has actually failed is better
on both mandates — the cheap attempt usually succeeds, and when it fails you
have evidence rather than a hunch.

## The only ground truth: `POST /outcome`

Everything else the router records is a proxy. Structural checks know whether
code *parses*. The local checker guesses whether prose *looks* right. Neither
knows whether the answer did the job — the client does, because it ran the
tests.

```bash
# id comes from the completion body, or any stream chunk
curl -s localhost:8080/outcome -H 'content-type: application/json' \
  -d '{"request_id":"chatcmpl-...","ok":false,"detail":"tests failed"}'
```

Two things make this the highest-value signal available:

- **It is the only quality signal that survives streaming.** A retry cannot
  reach a streamed response because the bytes are already gone; a report
  arrives afterwards and works either way. Every agent client streams.
- **Its successes count.** `feedback.py` folds client outcomes in BOTH
  directions, unlike checks, where only failures count. A parser's 'ok' means
  the code parsed and would inflate scores; a client's 'succeeded' means the
  work worked.

An unknown `request_id` returns 404 rather than being quietly accepted — a
client whose reports go nowhere should find out.

## Verification: what local compute is actually good for

Local inference is a poor substitute for cloud completions here — 7-82x the
energy, ~670x the carbon on Michigan's grid, and slower (6.2s vs 1.4-2.0s).
But it is very good at stopping a cloud completion from being wasted, and
completions are where all the money is: fitted on real traffic, a completion
token costs **201x** a prompt token.

| check | cost | what it catches |
|---|---|---|
| structural (`verification.py`) | **free** | truncation, malformed code/JSON/YAML, empty answers |
| local LLM (Ollama) | ~$6.9e-05, ~6s | refusals, wrong-question answers, incoherence |

Structural checks run on every response and never execute the code — they
parse it. The local LLM check runs only on answers above
`verification.min_completion_tokens`, in the background after the client has
its response, because it costs ~15% of a median 193-token answer and only
pays above ~600 tokens.

### An agent turn is not a prose answer

Both checkers had mirror halves of the same blind spot, and real traffic is
what found it. On the first genuine agent session (63 completions, a shipped
feature, 349 passing tests, clean mypy and ruff):

| checker | called it | actually |
|---|---|---|
| structural | `malformed: empty response` x29 | turns that ended in a tool call |
| local LLM | `cuts off mid-sentence` x8 of 9 | same turns, judged from the other side |

A turn that calls a tool has empty or half-finished text **by design**. Both
paths now take `has_tool_calls` and return `unverifiable`; in `verify_response`
that check outranks even `finish_reason == 'length'`, because stopping
mid-sentence at a call boundary is not a budget overrun. `worth_local_check`
declines outright, which also stops paying ~6s of local inference to
mis-grade a tool call.

Had `feedback.py` run before this, it would have applied ~12 false failures to
the models that had just shipped the feature. That is the **fourth** harness
bug in this project that would have scored the rig rather than the model, and
the first caught by real traffic instead of a synthetic test. The pre-fix rows
are kept but set `model_attributable = 0`, so the record survives without
steering routing.

**`client_capped` was also over-applied.** It marked *every* verdict
non-attributable whenever the client set `max_tokens` — and opencode always
does — so real failures were invisible to feedback. A client's token cap
explains a `truncated` verdict and nothing else; it is now scoped to exactly
that.

`feedback.py` folds observed failures into `proficiency`, so routing learns
from your traffic rather than only the 23-task benchmark. Only **failures**
are folded in: a structural 'ok' means the code parsed, not that it was
correct, and recording those as 1.0 would flatten every score toward the
ceiling. Failures the model did not cause — a client's own tight `max_tokens`
truncating the answer — are recorded but excluded.

## The classifier is the latency floor

Every routed request pays a full local classification round-trip before a
single upstream token is requested, so the classifier model is the single
biggest lever on interactive latency.

**The default is `mistral-nemo:12b`, and it replaced `qwen3.5:latest` on
measurement.** Same prompts, same system prompt, `temperature: 0`, cold load
excluded — 14 unambiguous category cases and 5 tier probes:

| | `qwen3.5` | `mistral-nemo:12b` |
|---|---|---|
| category correct | 10/14 | 9/14 |
| tier correct | 1/5 (2 of them crashed) | **3/5** |
| **hard failures** | **4 of 19 calls (21%)** | **0** |
| latency mean / median / max | 6.6 / 5.5 / **15.6s** | 1.7 / 1.8 / **1.9s** |

Category accuracy is a wash. What decides it is the failure column and the
tail. Every one of those 4 failures is the runaway-thinking-trace mode below:
~15s spent to produce no JSON, which degrades to `source: "fallback"` — tier
2, `general_chat`. A reasoning model is the wrong tool for a job whose entire
output is ~45 tokens of JSON.

`mistral-nemo` does not reason by default, which is why its tail is flat: its
slowest call (1.94s) is faster than qwen3.5's median. End-to-end `/route`
went ~10s -> **~1.7s**. Both scored 5/5 on the local verification task, so
the verifier moved with it and only one model stays resident.

Four settings keep this usable, each fixing a failure seen in practice:

- **`max_retries=0` on the classifier client.** The OpenAI SDK retries twice
  by default, so `timeout_seconds` silently became a 3x wall-clock bound — a
  request hung past 250s on a 120s setting and logged nothing.
- **`max_output_tokens: 1024`.** Bounds a REASONING model's chain of thought.
  Unbounded, it cascades: Ollama keeps generating after the client gives up
  *and* serializes per model, so one runaway request queues every later
  request behind it and the timeouts spread. 256 was too tight — the trace
  consumed the budget and the model was truncated before emitting any JSON,
  which surfaced as an empty response. Does not bind for `mistral-nemo`; kept
  because it costs nothing unused and is the only guard if a reasoning model
  is swapped back in.
- **`max_input_chars: 8000`.** The classifier decides a category and a tier;
  it does not need the document. Feeding it one is harmful, not merely
  wasteful — on a ~20k-token prompt `qwen3.5` spent 28.7s and returned empty
  while `mistral-nemo` spent 41.8s echoing the input back inside its JSON.
  Both land on `source: "fallback"`, i.e. 30-40s of local inference buying the
  answer an instant failure would have given. Clamped to head + tail (the
  instruction sits at one end or the other; never the middle), the same
  prompts classify correctly in ~2.2s. Nothing is lost:
  `chat_completions` measures the real conversation with
  `estimate_prompt_tokens` and takes the larger value.
- **`fallback_tier` / `fallback_category`.** A classifier that times out,
  errors, or returns garbage now degrades to a configured mid tier flagged
  `source: "fallback"` instead of returning 502/503. The caller is a coding
  agent that would rather have a mid-tier answer than an error. Escalation
  deliberately skips fallbacks, so an unavailable local model does not
  silently promote every request to the frontier tier.
- **`temperature: 0`.** At the default, the same prompt classified tier 2
  then tier 1 on consecutive calls and routed to two different models.

But ~10s of local overhead on every message is a real tax for an interactive
agent, where the upstream answer itself may take 2s. Still unaddressed:
classify once per session rather than per message, cache by prompt hash, or
skip classification for short prompts.

### "Local" means your hardware, not this machine

The premise is that a local LLM classifies the task before a cloud model
answers it. That does not require the GPU to be in the machine you are typing
on, and usually it isn't — most developers already have WireGuard or a VPN
back to a home lab. So the normal shape is router and editor on the laptop,
Ollama on the workstation, `classifier.base_url` pointing across the tunnel.

`classifier.base_url` / `api_key_env` / `model` take any OpenAI-compatible
endpoint. Verified end to end against a non-loopback address, classifier and
verifier both.

Ollama binds `127.0.0.1` by default, so this fails with connection-refused
until the serving host applies `deploy/ollama-over-vpn.conf`. Bind it to the
VPN address, not `0.0.0.0`: Ollama has no auth of any kind, so anything that
can reach the port can run inference and enumerate your models, and `0.0.0.0`
publishes it on whatever wifi the laptop is sitting on.

A cloud endpoint also satisfies that interface, and measured better than the
local one — worth knowing before assuming local is the cheap side. Five
prompts, same system prompt, `temperature: 0`:

| | local `qwen3.5` (RTX 6000) | NeuralWatt `deepseek-v4-flash` |
|---|---|---|
| mean latency | **11.58s** (4.95-15.76) | **1.02s** |
| categories agreed with the label | 2 of 4 | **5 of 5** |
| hard failures | 1 of 5 (empty after 15.76s) | 0 |
| energy per call | ~7e-05 kWh, on your meter | 1.17e-05 kWh attributed |
| cost per 1,000 calls | electricity + 6.6GB resident | **$0.093** (0.19% of quota) |

11x faster, more accurate, and less attributed energy, on a machine with a
24GB card sitting idle. The one hard failure is the documented
runaway-thinking-trace mode — `qwen3.5` spent 15.76s and emitted no JSON,
degrading to `source: "fallback"`, i.e. a silent tier-2 guess.
`deepseek-v4-flash` does not reason by default, so that mode does not exist
for it. End-to-end `/route` goes ~10s -> **~1.0s**.

Local classification was assumed to be the cheap option because local compute
felt free. It is not free, it is just unbilled — the same substitution the
cost axis had to unlearn. The shipped default is still local Ollama, because
switching spends quota and that is a deployment choice, not a code one.

### The verifier follows, but only to another Ollama

The local LLM check speaks Ollama's **native** `/api/chat` (the only way to
set `think: False`), so it follows the classifier across a VPN but not to a
cloud provider. It used to derive its URL by stripping `/v1` off
`classifier.base_url`, which meant moving the classifier at all would have
pointed it at `<that host>/api/chat`. It now has its own
`verification.base_url` and `verification.model`.

`verification.model` may be null only while both run on one host. Config load
**refuses** the null once the hostnames differ, because the failure is silent:
observed directly, with the classifier on NeuralWatt the verifier POSTed
`deepseek-v4-flash` to `localhost:11434`, 404'd, caught it, logged "local
verification unavailable" and recorded no sample. Verification would have
looked enabled while producing nothing.

Structural verification needs no model at all — it is pure Python — so
`verification.local_llm_enabled: false` leaves a host with no local inference
fully functional, minus the refusal/incoherence class of failure.

Note also that opencode sends **~32K prompt tokens** of system prompt and
tool definitions on a trivial request, so the measured-size floor in
`estimate_prompt_tokens` does real work — the classifier's own estimate for
that request was two orders of magnitude low.

## What's NOT built yet — pick up here

1. **Leaderboard priors are unfilled.** `leaderboards.yaml` ships empty on
   purpose — inventing plausible-looking benchmark numbers would put
   fabricated data straight into routing, the same failure as the provider's
   `static_fallback` carbon constant this project already excludes. Until real
   sourced figures go in, a newly listed NeuralWatt family has no prior and
   relies entirely on self-eval accumulating. `python leaderboard.py --check`
   lists what is missing.

2. **Sampling depth for three models — now `eco`-only.** 7 samples/model gives
   split-half agreement within 1.4x for 10 of 13, but `kimi-k2.7-code-fast`
   (29x), `kimi-k3` (14x) and `glm-5.2-flex` (2.2x) are still unsettled. This
   no longer touches cost, which is priced per-request from the catalog, so it
   only affects `eco` — which is not an objective. Low priority unless eco
   comes back.

3. **Retry does not reach streaming.** The iteration budget (`iteration.py`)
   retries after a failed check, but only on the non-streaming path — once
   bytes have gone to the client there is nothing to take back. Buffering to
   fix that would cost streaming itself, a worse trade for interactive work.
   `POST /outcome` is the answer for streamed traffic: it arrives afterwards,
   so it works identically either way.

4. **Session-directory attribution picks the wrong directory.** The opencode
   plugin now genuinely reports — 31 accepted, 3 refused as ambiguous, 22
   `succeeded` / 4 `failed` from real test runs, all folded in. But the
   "most frequent path" heuristic resolved one session to a **dependency's
   source directory** inside the project's own virtualenv (24 mentions)
   rather than the project being edited (22), because reading a library's
   source outweighed writing the code under test. It degrades safely — an
   ambiguous session is refused rather than misattributed, which is what the
   3 × 409 were — but the heuristic needs to weight *writes* over reads, or
   anchor on the client's cwd instead of a path histogram.

5. **Local energy is not on the ledger.** The router meters what NeuralWatt
   bills and reports, but the electricity its own classifier and verifier burn
   on local hardware is invisible to it. The whole premise is spending cheap
   local compute to avoid wasting expensive cloud compute, and right now only
   one side of that trade is measured — which is how "local is free" survived
   as an assumption long enough to be wrong (a hosted classifier measured
   faster, more accurate and lower attributed energy than the local one).
   Closing it means metering local draw (nvidia-smi / RAPL / a smart plug) and
   pricing it against a real tariff — utility rate data, ideally the user's own
   plan, including time-of-use bands. That would also make the local-vs-cloud
   comparison an actual number rather than a shrug.

## Known open questions

- Answered: cost and eco stay separate axes — grid intensity spans 13.6x
  across the catalog, so they rank models differently.
- Answered: the GLM rows reporting `grid_id: FI` at 475 gCO2/kWh were
  `carbon_source: static_fallback` — a substituted constant, not a
  measurement. They are now excluded from eco rather than trusted. Still
  worth asking NeuralWatt why the fallback keeps the original `grid_id`,
  since that is what made it look like a real regional difference.
- Three models still fail a split-half stability check at 7 samples. Is the
  instability real (variable serving conditions) or an artifact of when the
  sweep ran? Re-sweeping at a different hour would tell.
- Answered, and the question no longer parses: tier-1 composites used to sit
  within 0.009 of each other because min-max normalization compressed them.
  There is no composite any more — ranking is quality first, cost as the
  tiebreak inside `quality_tolerance` — so nothing normalizes and nothing
  compresses.
- Answered: the eval set exists (`evals/tasks.yaml`, 23 tasks, four scoring
  kinds) and `tests/test_task_set.py` keeps it honest. The open part is
  narrower now — `coding_refactor` and `debugging` are still flat at 1.00
  across every model, so those tasks discriminate nothing and either need
  hardening again or should be conceded as non-discriminating. **Try samples
  before hardening.** `docs_writing` looked flat at the top too, and six more
  passes spread it 0.66-0.97 without touching a task; two samples per model is
  not enough to tell a saturated task from an unsampled one.
- How much context-assembly (RAG-style retrieval) belongs in the classifier
  step vs. a separate pre-step? Leaning decoupled, undecided.
- Should `eco_score` use real-time grid carbon intensity per request or a
  stable per-model average? Currently the latter, from the reference sweep.
  `grid_carbon_intensity` and `grid_id` are logged per observation, so this
  stays answerable from data without a re-run.

## Config is strict: an unknown key is an error

Pydantic ignores extra keys by default, which means a typo or a misplaced
setting loads cleanly, does nothing, and still looks configured. Every config
model now inherits `StrictModel` (`extra="forbid"`), so both of these fail at
load rather than silently:

```
verification.max_input_chars     # right key, wrong section
routing.min_tool_proficency      # sic
```

This is not hypothetical. `max_input_chars` shipped into the `verification:`
block instead of `classifier:` and was accepted and discarded — it happened to
match the code default, so behaviour was correct and the file was a lie.
Editing it would have done nothing.

The corollary worth keeping: **every knob belongs in `config.yaml`, not only
in a Pydantic default.** A default the file never mentions is invisible to
anyone tuning it. `classifier.outcome_attribution_window_seconds` was removed
in the same pass — it was declared, never read, and shadowed the
`verification` one that actually is.

## Setup

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
sqlite3 router.db < schema.sql
cp .env.example .env   # fill in NEURALWATT_API_KEY
python poller.py       # populate the catalog
python tier.py         # resolve tiers
python config.py       # sanity-check config loads
python -m uvicorn dispatcher:app --reload
```

Then set what is deployment-specific in `config.yaml`: `classifier.model` and
`classifier.base_url` for your Ollama, and `objective.plan_kwh_per_period` to
your own plan's quota (it is reported in `/health` as burn against the
allowance; it does not gate anything).

Ollama must be reachable with the classifier model pulled — the name must
match `classifier.model` in `config.yaml`:
```bash
ollama pull mistral-nemo:12b
```

`local_vision` ships **enabled** as a core feature (see below), so also pull
its model unless you're turning it off:
```bash
ollama pull qwen3-vl:4b
```

It does not have to be on this machine. To use one across a VPN, point
`classifier.base_url` and `verification.base_url` at it and apply
`deploy/ollama-over-vpn.conf` on the serving host — Ollama binds loopback-only
by default and will otherwise refuse.

`requirements.txt` is pinned. Recreating the venv with the old `>=` ranges
jumped openai 2.53 -> 3.0 and httpx -> httpx2 without warning; a service that
restarts on boot shouldn't change its dependency tree underneath itself.
Bump deliberately.

`pyproject.toml` puts the repo root on `sys.path` for pytest — the modules
live at the root rather than in a package, so `pytest` (console script) and
`python -m pytest` would otherwise disagree about whether `import config`
resolves.

## Run as a service

`deploy/` holds five systemd **user** units (dispatcher, plus a timer and a
oneshot service each for the poller and the seed sweep) and one drop-in for a
*system* Ollama — see `deploy/README.md` for install and operation. In short:

```bash
echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env
cp deploy/llm-router*.{service,timer} ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now llm-router.service llm-router-poller.timer
```

The dispatcher binds `127.0.0.1:8080`. **The poller timer is load-bearing,
not housekeeping**: `stale_after_days` is 3 with `exclude_stale: true`, so an
unpolled catalog eventually marks every row stale and the router returns zero
candidates for everything.

The service holds a billable API key and has **no auth of its own**. Loopback
bind is the only thing standing between the open internet and your allowance;
add auth before widening `--host`.

The same applies to an Ollama shared over a VPN — it has no auth either, so
`deploy/ollama-over-vpn.conf` binds it to the VPN address rather than
`0.0.0.0`, which would publish it on whatever network the client happens to
be on.

## Pointing a coding agent at it

The `/v1` endpoints are OpenAI-compatible, so any normal client works —
opencode, an SDK, plain curl. Repo-local `opencode.json` is already wired up,
so running `opencode` from a clone of this repo routes by default. For global
use, merge `provider.llm-router` into `~/.config/opencode/opencode.json`.

| model name | behavior |
|---|---|
| `auto` | router picks; flex rows excluded so nothing is held during peak |
| `auto:batch` | router picks; flex rows admitted, for overnight/async work |
| any real model id | dispatched as asked, still logged |

Streaming is proxied chunk by chunk rather than buffered, so tokens still
render as they arrive. NeuralWatt emits its energy and cost blocks as SSE
**comment** lines (`: energy {...}`) before `data: [DONE]` — ordinary clients
ignore comments, so the stream passes through untouched while the router
reads the telemetry on the way past. Without that, streamed calls would log
no energy at all, which is most of the point of this project.

## Try it

```bash
# Pick a model without spending anything
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
  -d '{"task":"Refactor this 800-line Django view into service objects."}'

# Same, but admit flex rows (held during peak, fine for overnight work)
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
  -d '{"task":"nightly code review","latency_tolerance":"batch"}'

# Actually call the winner and log energy/cost
curl -s -X POST localhost:8080/dispatch -H 'content-type: application/json' \
  -d '{"task":"What is a Python context manager?"}'

# The OpenAI-compatible surface any client can use
curl -s localhost:8080/v1/models
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'

# What did routing actually cost and burn?
sqlite3 -header -column router.db \
  "SELECT model_id, prompt_tokens, completion_tokens, energy_kwh, carbon_g_co2eq, cost_usd
     FROM energy_observations ORDER BY id DESC LIMIT 10;"
```

`/route` takes `task_category`, `task_tier`, and `required_context_tokens`
overrides, which skip the classifier — useful for testing routing changes
deterministically.