alee/6krrt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

neuralwatt-router-service #1

Merged
alee merged 20 commits from neuralwatt-router-service into main 2026-08-21 23:10:28 +00:00
Conversation 0 Commits 20 Files Changed 46 +8614 -315
46 changed files with 8614 additions and 315 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.env.example
View File

@@ -1,3 +1,2 @@
# Copy to .env and fill in. Never commit the real .env file.
OPENROUTER_API_KEY=
NEURALWATT_API_KEY=

706
CLAUDE.md
View File

@@ -1,17 +1,23 @@
# Local LLM Model Router — project brief
Read `design/local-llm-model-router.md` first for full architecture/rationale.
This file is the working state + immediate next steps.
`README.md` documents what is actually built, module by module.
`design/local-llm-model-router.md` holds the architecture and rationale,
including parts still unbuilt. This file is the working state + immediate
next steps, and is the one to trust on what is currently true.
## What this is
A router that uses a local model (served via Ollama on an RTX 6000, 24GB) to
classify incoming coding/documentation tasks — category, tier, required
context size — and dispatch each task to the cheapest/best-fit open-weight
model across OpenRouter and Neuralwatt Cloud, weighted by cost, ecological
impact (Neuralwatt exposes real energy-per-request data), and per-category
model on **Neuralwatt Cloud**, weighted by cost, ecological impact
(Neuralwatt exposes real energy-per-request data), and per-category
proficiency.
Neuralwatt is the only provider. OpenRouter was removed — the `provider`
column and the `(model_id, provider)` primary key stay so a second provider
can be added later without a migration.
## Stack
- Python (chosen over Rust — this is I/O-bound against provider APIs, not
@@ -20,90 +26,652 @@ proficiency.
- SQLite for the decision table
- Ollama (OpenAI-compatible endpoint at `localhost:11434/v1`) for local
classification
- FastAPI planned for the dispatcher service (not yet built — see below)
- FastAPI for the dispatcher service
## Billing is per-kWh, not per-token — and neither is what scoring uses
**Measured against the live API, 2026-08-11.** Neuralwatt bills a flat
**$8.00 per kWh** and the catalog's `input_per_million` /
`output_per_million` prices are not what this account is charged. Confirmed
across five models; `cost_usd / energy_kwh` came back 8.00 every time:
| model | list $/1M out | completion tokens | billed USD | kWh | $/kWh |
|---|---|---|---|---|---|
| deepseek-v4-flash | 0.28 | 600 | 5.00e-06 | 5.88e-07 | 8.50\* |
| gemma-4-31b | 0.42 | 540 | 3.80e-04 | 4.75e-05 | 8.00 |
| qwen3.6-35b-fast | 1.15 | 600 | 2.53e-04 | 3.16e-05 | 8.01 |
| kimi-k2.7-code-fast | 4.00 | 600 | 2.25e-03 | 2.81e-04 | 8.00 |
| kimi-k3-fast | 15.00 | 539 | 2.17e-04 | 2.71e-05 | 8.00 |
\* rounding — billed cost is quantized to ~1e-06.
The precise rule, validated against all 65 samples of the reference sweep
(61/65 within 2%; the 4 outliers are microdollar rounding, not misses):
```
cost_usd = min( $8.00/kWh x energy_kwh , 3 x list token price )
```
The ceiling bound in only 2 of 65 samples, both `deepseek-v4-flash` energy
spikes, and matched to the cent: 31 prompt x $0.14/1M + 400 completion x
$0.28/1M = 1.1634e-04, x3 = 3.4902e-04, billed 0.000349.
**List price ranks models backwards.** Not approximately — invertedly:
| | list $/1M | actually billed | gCO2eq |
|---|---|---|---|
| `deepseek-v4-flash` | **0.28** | 9.80e-05 | 8.90e-04 |
| `gemma-4-31b` | 0.42 | **5.02e-05** | **2.32e-04** |
deepseek lists 33% cheaper, costs 95% more, and emits 284% more carbon.
**But cost and eco are NOT the same axis** — the tempting simplification, and
it is wrong. Cost tracks energy, but carbon is energy x the serving region's
grid intensity, and models run in different regions:
| grid | gCO2/kWh | models |
|---|---|---|
| `FI` | ~49-50 | most of the catalog; varies by time |
| `FI` (reported) | 475 | `glm-5.2-fast`, `glm-5.2-flex` |
| `US-MIDA-PJM` | ~442 | the `kimi-k3` family |
A 13.6x spread, so the two axes disagree: `glm-5.2-fast` is the 2nd cheapest
model and only the 6th cleanest; `kimi-k3-flex` draws 3.7x *less* energy than
`kimi-k2.7-code` while emitting 3.6x *more* carbon. Weighting them separately
is load-bearing, and `tests/test_routing.py` pins it.
**Superseded — cost no longer comes from the sweep at all.** `cost` was the
*median* measured USD over the reference sweep. That was measured to be WRONG
for real traffic, because the reference workload is the wrong shape.
The sweep sends a 400-token prompt with a 400-token completion. Real agent
traffic is a 150,000-token prompt with a ~400-token completion and 84% cache
hits. The attribution ratio moves with prompt size, so the ranking inverts:
| workload | winner |
|---|---|
| reference sweep (400/400) | `glm-5.2-fast`, 3.2x cheaper |
| realistic (70k prompt, short answer) | **`deepseek-v4-flash`, 5.0x cheaper** |
Same two models, opposite answer. `glm-5.2-fast` sits at attribution 0.006 on
a toy prompt and 0.50 on a 70k one — it batches beautifully on small prompts
and badly on real ones. `deepseek-v4-flash` barely moves (0.21 -> 0.25).
So `routing.estimated_cost` prices each request from **catalog token prices,
scaled to that request's actual shape** (prompt size, assumed completion
length, `assumed_cache_rate`). List price is not what gets billed, but billing
is capped at 3x list, so it tracks the real ordering and bounds it — and on
the one case that was checked live it agrees with the measurement in direction
and magnitude (7.8x predicted vs 5.0x measured). It is also free, needs no
sweep, and refreshes whenever the poller runs.
Three signals said `deepseek-v4-flash` — catalog token price (7.8x cheaper),
NeuralWatt's own published per-request energy (~10x lower), and a live 70k
measurement (5.0x cheaper). Only the 400-token benchmark disagreed. Trust the
workload you actually run.
`eco` still comes from the sweep's median gCO2eq, and is still not an
objective. `flex_cost_multiplier` is gone: a flex row's measured cost already
is its flex cost.
**Open, and worth knowing:** NeuralWatt's model cards publish *gross* energy
(~1.99e-04 kWh for deepseek, ~1.91e-03 for GLM), while the billed figure is
gross x attribution. GLM burns roughly 7x more actual electricity per request
and charges ~5x less, because far more tenants share its GPUs. Anything built
on `eco` inherits that inversion — the attributed carbon figure answers "what
is my share", not "what was burned".
## Energy attribution: signal that looks like noise
Billed energy decomposes exactly:
```
energy_kwh = avg_power_watts x duration_seconds x attribution_ratio
```
`attribution_ratio` is the request's share of a shared multi-tenant GPU pool.
Up close it looks like pure noise — eight rapid identical calls to one model
spanned 20x in billed energy, correlating **+0.997** with the ratio while
power and duration held steady. Two sweeps of the same 13 models with the
same prompt disagreed by up to 36x.
Scoring on the pre-attribution product (`power x duration`) was tried, and it
is **wrong**. Across the sweep:
| | spread |
|---|---|
| median attribution, **between** models | **750x** |
| typical spread **within** one model | **1.8x** |
The ratios are quantized (0.001, 0.25, 0.5, 0.75) — that is serving
concurrency, a stable per-model property, not weather. A model whose GPUs
carry far more concurrent requests genuinely costs less per request, and
that is most of the real cost difference in the catalog: `deepseek-v4-flash`
bills ~1000x under its share of pool gross. Stripping attribution discards a
750x real signal to suppress a 1.8x one.
So scoring reads the attributed figures, and the **median** absorbs what
noise remains. A split-half check on the 7-sample sweep (median of first
three vs last four) shows that working:
- **10 of 13 models agree within 1.4x** — stable enough to route on
- **3 do not**: `kimi-k2.7-code-fast` (29x), `kimi-k3` (14x),
`glm-5.2-flex` (2.2x). Those need more samples before their position is
trustworthy.
`dispatcher.gross_energy_kwh` remains as a diagnostic on the identity, not a
scoring input.
### Attribution drifts across hours, so sampling must too
Within about 30 minutes the billed figures reproduce (0.3-1.1x on a
spot-check). Across hours they do not: between two sweeps,
`deepseek-v4-flash` moved roughly 50x and `qwen3.6-35b` about 7x the other
way — enough to **invert their cost ranking**. Attribution tracks pool load,
and pool load tracks time of day.
More samples inside one sweep does not fix this; it measures one moment more
precisely. Coverage across time does. `load_candidates` already takes the
median over ALL `seed_reference` rows, so repeated sweeps accumulate into a
median-across-time for free — hence `llm-router-seed.timer`, which runs a
small sweep every 6 hours.
Until several sweeps have accumulated, treat the cost and eco ordering as
provisional. A single sweep's ranking is one sample of a moving quantity.
## What's built and working
- `schema.sql` — `models`, `proficiency`, `energy_observations` tables.
Applies cleanly (`sqlite3 db/router.db < schema.sql`).
- `poller.py` — fetches OpenRouter's and Neuralwatt's `/models` endpoints
(both public/unauthenticated), normalizes pricing/context/capabilities,
upserts into `models`, marks stale rows. Syntax-validated but **not yet
run against live APIs** — this sandbox couldn't reach openrouter.ai or
api.neuralwatt.com, so the field-mapping assumptions (especially
OpenRouter's `supported_parameters` list and Neuralwatt's
`metadata.pricing`/`metadata.capabilities` shape) need to be confirmed
against real payloads on first run. Fix field paths if the actual JSON
differs.
- `config.yaml` / `config.py` — all weights, thresholds, provider settings.
Pydantic-validated (weights must sum to 1.0, etc.), confirmed loading
correctly.
- `schema.sql` — `models`, `proficiency`, `energy_observations`. Applies
cleanly (`sqlite3 router.db < schema.sql`). `models` carries the serving
class columns (below); `energy_observations` carries real carbon/cost.
- `poller.py` — fetches Neuralwatt's `/models` endpoint (public,
unauthenticated), normalizes, upserts, marks stale rows. **Verified
against the live API**: 19 models, and the `metadata.pricing` /
`metadata.capabilities` / `metadata.limits` field mappings are confirmed
correct.
- `config.yaml` / `config.py` — weights, thresholds, provider settings,
Pydantic-validated.
- `scoring.py` — one `normalize_inverted` (cost and eco normalize
identically; they differ only in what is fed to them) plus the weighted
composite.
- `seed_energy.py` — runs a fixed reference task N times per routable model
and writes `energy_observations` rows tagged `seed_reference`. This is what
makes `cost` and `eco` real numbers instead of the neutral 0.5. Re-run it
after the catalog gains models: `python seed_energy.py --samples 5`
(13 models x 5 = 65 calls, and the whole sweep cost **under a cent**).
- `tiering.py` / `tier.py` — pure tier resolver + the DB pass that applies it.
- `routing.py` — pure hard filters and ranking.
- `dispatcher.py` — FastAPI service. `GET /health`, `POST /route` (classify
and pick, no provider call), `POST /dispatch` (route, call, log), plus an
OpenAI-compatible `GET /v1/models` and `POST /v1/chat/completions`.
- `proficiency.py` / `proficiency_store.py` — pure blending arithmetic, and
the only path that writes the `proficiency` table (so `blended_score` and
`source` can never drift from the inputs that produced them).
- `leaderboards.yaml` / `leaderboard.py` — curated per-family priors and their
importer. **Ships empty by design**; `python leaderboard.py --check` names
every active family lacking a prior.
- `evals/tasks.yaml` / `eval_proficiency.py` — the self-eval task set and
runner. Four scoring kinds, objective wherever the category allows it:
`code` executes the model's Python against checks, `exact` compares a
normalized answer, `tool` inspects the tool call structurally, and only the
four prose categories fall back to a `judge`.
- `tests/` — 207 tests across 11 files, all passing.
### Serving class: one base model, many rows
Neuralwatt ships ~6 base models as 19 catalog rows. The id suffixes are three
**orthogonal** dimensions (hence `glm-5.2-short-fast-flex`), parsed by
`poller.parse_serving_class` into columns:
| suffix | column | meaning |
|---|---|---|
| `-flex` | `latency_class` | discounted async; held server-side during peak until a capacity gap opens |
| `-fast` | `reasoning_mode` | thinking disabled or capped to a short budget — *not* queue priority |
| `-short` | `context_variant` | 200K pool with a bounded reasoning budget |
These rows carry **identical catalog pricing**, so without these columns all
7 GLM rows tie exactly and the router picks arbitrarily — which could send an
interactive request to a preemptible flex row. `latency_tolerance`
(`interactive` | `batch`) is therefore a hard filter in `routing.py`, not a
weight. Suffixes are matched as whole `-`-delimited segments so
`deepseek-v4-flash` is not misread as a `-fast` row.
### Access gating is prose-only
6 of 19 rows are restricted, and the catalog says so only in free text
("Private preview (grant-gated)", "(Canary)") — there is no structured field.
`poller.parse_access_level` parses it into `access_level`, and
`routing.allowed_access_levels` (default `[public]`) excludes them, so the
dispatcher doesn't select a model that 403s. 13 of 19 rows are routable.
### Tiering
Tier on `reasoning_default_enabled` (from `metadata.reasoning.default_enabled`,
falling back to `capabilities.reasoning`), **not** `supports_reasoning`.
`capabilities.reasoning` only means "the endpoint accepts a reasoning
param" — it is true for 17 of 19 rows, and tiering on it put 17 models in
tier 3 and left tier 1 empty. A `-fast` row does not inherit its sibling's
tier 3. Cost is checked before the reasoning rule so $0.28/1M models can
reach tier 1.
**Cheapness is not a capability ceiling** (`tier1_context_max`, default
512000). Tier is a *floor* — `routing.py` drops any row with
`tier < required_tier` — so tier 1 means "simple work only", not "cheap".
Deciding that on price alone put `deepseek-v4-flash` in tier 1 for no reason
but its $0.28/1M completion price, which excluded it **outright** from every
tier-2 request. It has a 1M advertised window and scores 1.00 on all three
coding categories. That was the same substitution the cost axis already had
to unlearn: price is a market signal, not a capability measurement.
Tier 1 now requires the model to be small **and** cheap. The gate reads the
**advertised** `context_window`, whose catalog values are the clean market
classes — 131056 / 199984 / 262128 / 1048560 — rather than
`effective_context_window`, which varies within a class. 512000 sits in the
empty band between the 256K and 1M classes with a 2x margin either side, so it
is not fitted to any one model. The gate only ever demotes; a huge window
never promotes an expensive model into tier 1, and a *missing* window does not
block it, since absent evidence should not decide anything.
Distribution moved **4 / 6 / 9 -> 1 / 9 / 9** — only the three deepseek rows
changed. With cost priced per-request, `deepseek-v4-flash` now wins
`coding_general` at every context size (16.5x cheaper than `kimi-k3` at 200k)
and is still correctly absent from `tool_use_agentic`, where its measured 0.33
drops it out of the quality band. That is the eval data earning it the slot
rather than a thumb on the scale — no `model_tiers` override was needed.
## Proficiency: category now changes routing
`proficiency_score` is the ONLY category-dependent term in the composite, so
until this table had data, `task_category` could not change a decision at
all — the classifier computed it, the router paid ~10s for it, and then it
made no difference. It does now:
| scoring | distinct models across 27 decisions (9 categories x 3 tiers) |
|---|---|
| catalog list price | 2 |
| measured cost + eco | 3 |
| proficiency only (cost/eco data missing) | 7 |
| **all three axes live** | **1** |
That last row is not a regression, and it is worth understanding before
tuning anything. With cost, eco and proficiency all populated,
`qwen3.6-35b` is **Pareto-dominant**: cheapest AND cleanest in the routable
set, while scoring within 0.15 of the best model on proficiency. On
`summarization` it costs 7x less and emits 65x less carbon than `kimi-k3`,
which beats it 1.00 to 0.85 on quality — no defensible weighting picks
`kimi-k3` from that. One model winning is the correct answer to the question
the weights ask.
Proficiency is still doing work underneath. The runner-up ordering reorders
by category — `kimi-k3-fast` climbs to 3rd on `docs_writing` and
`summarization` (proficiency 1.00) and drops out of the top four elsewhere.
So the axis is live; it just cannot overturn a leader that wins on two axes
at once.
The 7-model spread appeared only while cost and eco were missing and scoring
neutral 0.5. If a genuinely different balance is wanted, the lever is
`weights` in config.yaml, not more data.
### What the task set actually found
**Coding does not discriminate these models.** All 13 rows score exactly
1.00 on `coding_general`, `coding_refactor` and `debugging` — and that is
after the tasks were deliberately hardened with touching intervals, full
semver, present-but-falsy defaults, late-binding closures and a binary search
that infinite-loops. Every model in this catalog is simply good at that class
of problem, so cost and eco decide coding routes, which is the right outcome.
**What does discriminate is tool use, arithmetic traps, and prose.**
`deepseek-v4-flash` scores 1.00 on all three coding categories yet **0.33 on
`tool_use_agentic`** and 0.67 on `reasoning_math`. Verified live, not an
artifact: given "It is 1:20pm and my meeting starts at 3pm, how many minutes
away?" — both times supplied — it calls *two* tools rather than subtracting.
It over-reaches for tools, which is exactly the failure mode that matters in
an agent loop. The router now avoids it for those categories while still
picking it for coding.
Everything currently reads `source='self_eval_thin'`: real measurement, but
below `self_eval_min_samples` (2-3 tasks per category per run). Re-run
`eval_proficiency.py` to accumulate — scores fold into a running mean rather
than replacing, so samples add up across runs.
### Harness bugs this shook out
Three separate defects, each of which scored the rig rather than the model,
and each caught by reading per-task detail rather than the summary:
- **Token budget.** `max_tokens` was shared between a reasoning model's trace
and its answer. At 1200, qwen3.6-35b spent ~4,200 characters thinking and
returned an EMPTY content field, scoring 0.00 on tasks it can plainly do.
Now 24000, clamped per model (gemma-4-31b caps at 16384), and
`finish_reason: length` skips the sample instead of scoring it.
- **One leading space.** kimi-k2.7-code returns `" def f(...)"`, which becomes
IndentationError once the harness prepends its imports — 0.00 across all
nine coding tasks for a model with "code" in its name.
- **Judge failures scored as model failures.** 44% of judge calls returned
unparseable output (the judge is itself a reasoning model and leaks its
thinking despite `response_format`). Each was recorded as 0.0. Now the JSON
is extracted from surrounding prose and an unusable reply yields no sample.
`tests/test_task_set.py` exists so this stops happening: it implements a
reference solution for every `code` task and asserts it passes every check,
recomputes every `exact` answer (one by brute force), and confirms each
refactor target already passes its own checks while each debugging target
fails. It immediately caught a check where the expected value was simply
wrong — which would have docked every model on a task and been
indistinguishable from genuine difficulty.
## Tier is an iteration budget, not just a floor
A tier used to mean only "do not route below this". It now also buys
corrective attempts after a verification failure:
| tier | batch | interactive |
|---|---|---|
| 1 | 0 retries | 0 |
| 2 | 1 | 1 |
| 3 | 2 | 1 |
Interactive is capped below its tier because every retry doubles
time-to-answer, and in interactive use latency **is** a quality loss.
Retries are matched to the failure, since the causes differ:
- **truncated** — raise the token budget on the same model; a different one
would run out too. If there is no cap to raise, the model's own output
ceiling is the wall, so escalate to a candidate that can emit more.
- **malformed** — more tokens will not make unparseable output parse, so
escalate to the next-ranked candidate.
- **ok / unverifiable** — buy nothing. Retrying `unverifiable` would burn
quota across the majority of prose traffic for no signal.
`escalation.preemptive_on_low_confidence` is now **off by default**. Bumping
the tier because the classifier was unsure pays frontier prices before
anything has gone wrong; spending after a check has actually failed is better
on both mandates — the cheap attempt usually succeeds, and when it fails you
have evidence rather than a hunch.
## The only ground truth: `POST /outcome`
Everything else the router records is a proxy. Structural checks know whether
code *parses*. The local checker guesses whether prose *looks* right. Neither
knows whether the answer did the job — the client does, because it ran the
tests.
```bash
# id comes from the completion body, or any stream chunk
curl -s localhost:8080/outcome -H 'content-type: application/json' \
-d '{"request_id":"chatcmpl-...","ok":false,"detail":"tests failed"}'
```
Two things make this the highest-value signal available:
- **It is the only quality signal that survives streaming.** A retry cannot
reach a streamed response because the bytes are already gone; a report
arrives afterwards and works either way. Every agent client streams.
- **Its successes count.** `feedback.py` folds client outcomes in BOTH
directions, unlike checks, where only failures count. A parser's 'ok' means
the code parsed and would inflate scores; a client's 'succeeded' means the
work worked.
An unknown `request_id` returns 404 rather than being quietly accepted — a
client whose reports go nowhere should find out.
## Verification: what local compute is actually good for
Local inference is a poor substitute for cloud completions here — 7-82x the
energy, ~670x the carbon on Michigan's grid, and slower (6.2s vs 1.4-2.0s).
But it is very good at stopping a cloud completion from being wasted, and
completions are where all the money is: fitted on real traffic, a completion
token costs **201x** a prompt token.
| check | cost | what it catches |
|---|---|---|
| structural (`verification.py`) | **free** | truncation, malformed code/JSON/YAML, empty answers |
| local LLM (Ollama) | ~$6.9e-05, ~6s | refusals, wrong-question answers, incoherence |
Structural checks run on every response and never execute the code — they
parse it. The local LLM check runs only on answers above
`verification.min_completion_tokens`, in the background after the client has
its response, because it costs ~15% of a median 193-token answer and only
pays above ~600 tokens.
### An agent turn is not a prose answer
Both checkers had mirror halves of the same blind spot, and real traffic is
what found it. On the first genuine agent session (63 completions, a shipped
feature, 349 passing tests, clean mypy and ruff):
| checker | called it | actually |
|---|---|---|
| structural | `malformed: empty response` x29 | turns that ended in a tool call |
| local LLM | `cuts off mid-sentence` x8 of 9 | same turns, judged from the other side |
A turn that calls a tool has empty or half-finished text **by design**. Both
paths now take `has_tool_calls` and return `unverifiable`; in `verify_response`
that check outranks even `finish_reason == 'length'`, because stopping
mid-sentence at a call boundary is not a budget overrun. `worth_local_check`
declines outright, which also stops paying ~6s of local inference to
mis-grade a tool call.
Had `feedback.py` run before this, it would have applied ~12 false failures to
the models that had just shipped the feature. That is the **fourth** harness
bug in this project that would have scored the rig rather than the model, and
the first caught by real traffic instead of a synthetic test. The pre-fix rows
are kept but set `model_attributable = 0`, so the record survives without
steering routing.
**`client_capped` was also over-applied.** It marked *every* verdict
non-attributable whenever the client set `max_tokens` — and opencode always
does — so real failures were invisible to feedback. A client's token cap
explains a `truncated` verdict and nothing else; it is now scoped to exactly
that.
`feedback.py` folds observed failures into `proficiency`, so routing learns
from your traffic rather than only the 23-task benchmark. Only **failures**
are folded in: a structural 'ok' means the code parsed, not that it was
correct, and recording those as 1.0 would flatten every score toward the
ceiling. Failures the model did not cause — a client's own tight `max_tokens`
truncating the answer — are recorded but excluded.
## The classifier is the latency floor
Every routed request pays a full local classification round-trip before a
single upstream token is requested. Measured on `qwen3.5:latest`:
| | latency |
|---|---|
| cold (model not resident) | **43s+** — a reload can exceed even the 120s ceiling |
| warm, simple prompt | **~4s** |
| warm, prompt that triggers a long thinking trace | **up to ~25s** |
Four settings keep this usable, each fixing a failure seen in practice:
- **`max_retries=0` on the classifier client.** The OpenAI SDK retries twice
by default, so `timeout_seconds` silently became a 3x wall-clock bound — a
request hung past 250s on a 120s setting and logged nothing.
- **`max_output_tokens: 1024`.** qwen3.5 is a reasoning model and will
otherwise emit an unbounded chain of thought. That cascades: Ollama keeps
generating after the client gives up *and* serializes per model, so one
runaway request queues every later request behind it and the timeouts
spread. 256 was too tight — the trace consumed the budget and the model was
truncated before emitting any JSON, which surfaced as an empty response.
- **`fallback_tier` / `fallback_category`.** A classifier that times out,
errors, or returns garbage now degrades to a configured mid tier flagged
`source: "fallback"` instead of returning 502/503. The caller is a coding
agent that would rather have a mid-tier answer than an error. Escalation
deliberately skips fallbacks, so an unavailable local model does not
silently promote every request to the frontier tier.
- **`temperature: 0`.** At the default, the same prompt classified tier 2
then tier 1 on consecutive calls and routed to two different models.
But ~10s of local overhead on every message is a real tax for an interactive
agent, where the upstream answer itself may take 2s. Unaddressed options:
keep Ollama resident (`OLLAMA_KEEP_ALIVE`), classify once per session rather
than per message, cache by prompt hash, use a smaller classifier, or skip
classification for short prompts. This is a design call, not a tuning one.
Note also that opencode sends **~32K prompt tokens** of system prompt and
tool definitions on a trivial request, so the measured-size floor in
`estimate_prompt_tokens` does real work — the classifier's own estimate for
that request was two orders of magnitude low.
## What's NOT built yet — pick up here
1. **FastAPI dispatcher** (`dispatcher.py`, doesn't exist yet). Should:
- Accept a task (text + optional context/file references)
- Call Ollama with `config.classifier.system_prompt`, parse the JSON
response (`task_category`, `task_tier`, `required_context_tokens`,
`confidence`)
- If `confidence < config.escalation.min_confidence_before_bump`, bump
required tier by one (per config)
- Query `models` + `proficiency` tables: hard-filter on
`effective_context_window >= required_context_tokens`,
`tier >= required_tier`, `availability == 'active'` (respecting
`freshness.exclude_stale`/`exclude_deprecated`)
- Score remaining candidates: `w_cost * cost_score + w_eco * eco_score +
w_prof * proficiency_score[task_category]` — cost_score and eco_score
need a normalization function (e.g. min-max across candidate set,
inverted since lower cost/carbon = better) that doesn't exist yet
- Dispatch to the winning (model, provider) via that provider's
OpenAI-compatible endpoint
- Log the real response's energy/cost into `energy_observations`
(`energy_btu` = `energy_kwh * 3412.14`, yes really, see design doc history)
1. **Leaderboard priors are unfilled.** `leaderboards.yaml` ships empty on
purpose — inventing plausible-looking benchmark numbers would put
fabricated data straight into routing, the same failure as the provider's
`static_fallback` carbon constant this project already excludes. Until real
sourced figures go in, a newly listed NeuralWatt family has no prior and
relies entirely on self-eval accumulating. `python leaderboard.py --check`
lists what is missing.
2. **Energy/cost normalization functions** — cost_score and eco_score are
referenced in the design doc's scoring formula but the actual
min-max-normalize-and-invert logic isn't written anywhere yet.
2. **Sampling depth for three models — now `eco`-only.** 7 samples/model gives
split-half agreement within 1.4x for 10 of 13, but `kimi-k2.7-code-fast`
(29x), `kimi-k3` (14x) and `glm-5.2-flex` (2.2x) are still unsettled. This
no longer touches cost, which is priced per-request from the catalog, so it
only affects `eco` — which is not an objective. Low priority unless eco
comes back.
3. **Benchmark/proficiency poller** — nothing populates the `proficiency`
table yet. Needs both a leaderboard-scraping path (LMSYS Arena, LiveBench,
Aider polyglot — no unified API, likely manual/periodic scripted pulls)
and a self-eval harness (runs a fixed task set against candidate models,
scores results, writes to `proficiency` with `source='self_eval'`).
Blending rule is in `config.yaml` (`proficiency.leaderboard_weight` /
`self_eval_weight`, gated on `self_eval_min_samples`).
3. **Retry does not reach streaming.** The iteration budget (`iteration.py`)
retries after a failed check, but only on the non-streaming path — once
bytes have gone to the client there is nothing to take back. Buffering to
fix that would cost streaming itself, a worse trade for interactive work.
`POST /outcome` is the answer for streamed traffic: it arrives afterwards,
so it works identically either way.
4. **Escalation feedback loop** — a way for a human or downstream agent to
flag "this was routed wrong" and have it retry one tier up. Not designed
in detail yet beyond the config toggle.
4. **Session-directory attribution picks the wrong directory.** The opencode
plugin now genuinely reports — 31 accepted, 3 refused as ambiguous, 22
`succeeded` / 4 `failed` from real test runs, all folded in. But the
"most frequent path" heuristic resolved the session to
`.venv/lib/python3.14/site-packages/c2pa` 24 times versus
`~/Sources/fieldwitness` 22, because reading a dependency's source
outweighed editing the project. It degrades safely — an ambiguous session
is refused rather than misattributed, which is why the 3 × 409 — but the
heuristic needs to weight *writes* over reads, or anchor on the client's
cwd instead of a path histogram.
5. **Scheduling** — poller.py needs a cron/systemd timer, not yet set up.
No script for that exists here.
## Known open questions
## Known open questions (from design doc, still unresolved)
- How much context-assembly (RAG-style doc/code retrieval) should live in
the classifier step vs. a separate pre-step? Leaning toward decoupled but
not decided.
- Self-run eval set: what's the minimum viable task set to start proficiency
scoring without it becoming its own maintenance burden?
- Whether eco_score should use Neuralwatt's real-time grid carbon intensity
per-request or a stable per-model average — real-time is more accurate but
adds volatility to routing decisions.
- Answered: cost and eco stay separate axes — grid intensity spans 13.6x
across the catalog, so they rank models differently.
- Answered: the GLM rows reporting `grid_id: FI` at 475 gCO2/kWh were
`carbon_source: static_fallback` — a substituted constant, not a
measurement. They are now excluded from eco rather than trusted. Still
worth asking NeuralWatt why the fallback keeps the original `grid_id`,
since that is what made it look like a real regional difference.
- Three models still fail a split-half stability check at 7 samples. Is the
instability real (variable serving conditions) or an artifact of when the
sweep ran? Re-sweeping at a different hour would tell.
- The tier-1 composites sit within 0.009 of each other across a 40x cost
range, because min-max compresses once one candidate is far cheaper than
the rest. Ranking is right, but margins are thin — proficiency data will
swing these easily, which is the intent.
- How much context-assembly (RAG-style retrieval) belongs in the classifier
step vs. a separate pre-step? Leaning decoupled, undecided.
- Self-run eval set: minimum viable task set to start proficiency scoring
without it becoming its own maintenance burden?
- Should `eco_score` use real-time grid carbon intensity per request or a
stable per-model average? Currently the latter, from the reference sweep.
`grid_carbon_intensity` and `grid_id` are logged per observation, so this
stays answerable from data without a re-run.
## Setup
```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
sqlite3 db/router.db < schema.sql
cp .env.example .env # fill in OPENROUTER_API_KEY / NEURALWATT_API_KEY
python poller.py # first live run — verify field mappings against real API responses
sqlite3 router.db < schema.sql
cp .env.example .env # fill in NEURALWATT_API_KEY
python poller.py # populate the catalog
python tier.py # resolve tiers
python config.py # sanity-check config loads
python -m uvicorn dispatcher:app --reload
```
Ollama must be running locally with a classifier model pulled, e.g.:
Ollama must be running locally with the classifier model pulled — the name
must match `classifier.model` in `config.yaml`:
```bash
ollama pull qwen2.5:32b
ollama pull qwen3.5:latest
```
(model name must match `classifier.model` in `config.yaml`)
`requirements.txt` is pinned. Recreating the venv with the old `>=` ranges
jumped openai 2.53 -> 3.0 and httpx -> httpx2 without warning; a service that
restarts on boot shouldn't change its dependency tree underneath itself.
Bump deliberately.
`pyproject.toml` puts the repo root on `sys.path` for pytest — the modules
live at the root rather than in a package, so `pytest` (console script) and
`python -m pytest` would otherwise disagree about whether `import config`
resolves.
## Run as a service
`deploy/` holds three systemd **user** units — see `deploy/README.md` for
install and operation. In short:
```bash
echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env
cp deploy/llm-router*.{service,timer} ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now llm-router.service llm-router-poller.timer
```
The dispatcher binds `127.0.0.1:8080`. **The poller timer is load-bearing,
not housekeeping**: `stale_after_days` is 3 with `exclude_stale: true`, so an
unpolled catalog eventually marks every row stale and the router returns zero
candidates for everything.
The service holds a billable API key and has **no auth of its own**. Loopback
bind is the only thing standing between the open internet and your allowance;
add auth before widening `--host`.
## Pointing a coding agent at it
The `/v1` endpoints are OpenAI-compatible, so any normal client works —
opencode, an SDK, plain curl. Repo-local `opencode.json` is already wired up,
so `cd ~/Sources/6krrt && opencode` routes by default. For global use, merge
`provider.llm-router` into `~/.config/opencode/opencode.json`.
| model name | behavior |
|---|---|
| `auto` | router picks; flex rows excluded so nothing is held during peak |
| `auto:batch` | router picks; flex rows admitted, for overnight/async work |
| any real model id | dispatched as asked, still logged |
Streaming is proxied chunk by chunk rather than buffered, so tokens still
render as they arrive. NeuralWatt emits its energy and cost blocks as SSE
**comment** lines (`: energy {...}`) before `data: [DONE]` — ordinary clients
ignore comments, so the stream passes through untouched while the router
reads the telemetry on the way past. Without that, streamed calls would log
no energy at all, which is most of the point of this project.
## Try it
```bash
# Pick a model without spending anything
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
-d '{"task":"Refactor this 800-line Django view into service objects."}'
# Same, but admit flex rows (held during peak, fine for overnight work)
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
-d '{"task":"nightly code review","latency_tolerance":"batch"}'
# Actually call the winner and log energy/cost
curl -s -X POST localhost:8080/dispatch -H 'content-type: application/json' \
-d '{"task":"What is a Python context manager?"}'
# The OpenAI-compatible surface any client can use
curl -s localhost:8080/v1/models
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'
# What did routing actually cost and burn?
sqlite3 -header -column router.db \
"SELECT model_id, prompt_tokens, completion_tokens, energy_kwh, carbon_g_co2eq, cost_usd
FROM energy_observations ORDER BY id DESC LIMIT 10;"
```
`/route` takes `task_category`, `task_tier`, and `required_context_tokens`
overrides, which skip the classifier — useful for testing routing changes
deterministically.

609
README.md Normal file
View File

@@ -0,0 +1,609 @@
# Local LLM Model Router
A router that uses a local model (served via Ollama on an RTX 6000, 24GB) to
classify incoming coding/documentation tasks — category, tier, required
context size — and dispatch each task to the cheapest/best-fit open-weight
model on **Neuralwatt Cloud**, weighted by cost, ecological impact
(Neuralwatt exposes real energy-per-request data), and per-category
proficiency.
Neuralwatt is the only provider. The `provider` column and the
`(model_id, provider)` primary key stay so a second provider can be added
later without a migration.
## At a Glance
| Dimension | Detail |
|---|---|
| **Cost model** | Per-kWh, not per-token. Flat $8.00/kWh measured across the catalog. |
| **Latency** | ~4 s warm classification, up to ~120 s on cold start. Dispatcher response is unaffected — verification runs async. |
| **Quality** | Quality is the objective; cost is a per-request kWh ceiling plus a tiebreak. Eco is logged but no longer optimized. Proficiency blends external benchmarks with a self-run eval harness. |
| **Verification** | Every response structurally checked (free) + local LLM spot-check on long prose answers (~6 s, never blocks). Failures fold back into proficiency via `feedback.py`. |
| **Fault tolerance** | Classifier failure degrades to a mid-tier fallback rather than 502/503. Retry count is zero to prevent silent 3× time-outs. |
| **API surface** | OpenAI-compatible `/v1` endpoints, streaming chunk proxy with SSE telemetry scraping. |
## Verification Pipeline
Every completion passes through a two-layer check **before** routing learns from
it and **without** adding to client-facing latency for the async layer.
### Structural Verification — `verification.py`
Free, exact checks on what a model just returned — no code execution.
Extracts fenced code blocks and validates them by **parsing**, not running:
| Language | Check | Safeguard |
|---|---|---|
| Python (`python`, `py`, `python3`) | `ast.parse` | Syntax tree, no evaluation |
| JSON (`json`, `jsonc`) | `json.loads` | Strict parse |
| YAML (`yaml`, `yml`) | `yaml.safe_load` | No arbitrary objects |
| Shell (`bash`, `sh`, `zsh`) | `bash -n` (caller's job) | Parse-only, never runs |
| Other | `unverifiable` (not a failure) | Prose answers land here |
Plus two universal checks:
- **`finish_reason == "length"`** — decisive truncation, even if parsed content
looks fine (the most dangerous case: a fragment that parses)
- **Unterminated code fences** — the response ran out of tokens mid-block
**Key design decisions:**
- Never executes model output. The eval harness (`eval_proficiency.py`) does
run generated code, but there the prompts are ones this project authored,
so what comes back is bounded. Here the code is whatever the user asked for
and could do anything.
- `unverifiable` is **not** a failure. Most prose answers land there (correctly
— there's nothing structural to check), and counting it as wrong would
penalize models for the checker's limits.
- Measured on real traffic: a wasted cloud completion costs about 52 local
checks at 1,500 tokens and 139 at 4,000. A zero-cost check pays trivially.
### Local LLM Verification — `verification.py` (cont.)
For answers nothing structural can judge (prose, reasoning, refusals), a local
model spot-checks the response **after** it has gone back to the client, so its
~6 s never lands on anyone's latency.
The prompt frames the local model as a judge: "Does this answer actually
address the user's request?" and returns `{"ok": true|false, "reason": "..."}`.
Gating and safeguards:
- **Size threshold** — only responses ≥ 600 completion tokens trigger a local
check (configurable via `verification.min_completion_tokens`). A check on
a 193-token answer costs ~15% of the answer; at 1,500 tokens the break-even
failure rate drops to ~1.9%.
- **Middle elision** — if the answer is longer than the limit (default 4,000),
the **middle** is elided, not the head, so the checker judges the real ending.
An elision marker tells the checker not to flag it as a defect.
- **Thinking disabled** — the local model is a reasoning model and will otherwise
emit unbounded chain-of-thought. Verification uses Ollama's native endpoint
with `think=False`, so it answers in ~25 tokens instead of burning through
the budget on reasoning.
- **Malfunction safety** — an empty or unparseable verdict produces no sample
rather than a false failure. An empty response is settled by an if-statement,
never asked of a model what code can decide.
### Feedback Loop — `feedback.py`
Observation → learning. `feedback.py` folds verification failures into
`proficiency` so routing improves on **your traffic**, not just the fixed
23-task benchmark:
```bash
python feedback.py --dry-run # preview what would change
python feedback.py # apply
```
Key behaviors:
- **Only failures** are folded in. A structural 'ok' means the code parsed,
not that it was correct — scoring a parse-success as a proficiency 1.0 would
flatten every score toward the ceiling. The coding categories already sit
at 1.00 for every model; this would spread that flatness everywhere.
- **One 0.0 sample per failure**, added to the running mean so the penalty
scales with failure rate rather than replacing the benchmark score outright.
- **Idempotent** — `applied_at` marks consumed rows so re-running cannot
penalize a model repeatedly for the same response.
- **Not the model's fault** — a client that sets `max_tokens=40` and gets a
truncated answer caused that itself. Those failures are recorded in the
verifications table but excluded from proficiency feedback.
## Architecture
```
┌─────────────────────┐
incoming task ───▶│ Local Classifier │ Ollama (qwen3.5:latest)
│ - task_category │ ~4s warm, ~120s cold cap
│ - task_tier │ temperature: 0, max 1024 tokens
│ - required_context │ max_retries: 0 (silent 3× cap guard)
│ - confidence │
└──────────┬───────────┘
▼
┌─────────────────────┐
│ Classifier Fallback │ tier 2 / general_chat on failure
│ (graceful degrade) │ — not a 502/503
└──────────┬───────────┘
▼
┌─────────────────┐
│ Escalation │ low-confidence tier bump
│ (optional) │ threshold: 0.6 confidence
└────────┬─────────┘
▼
┌─────────────────────┐
│ Hard Filters │ context window ≥ required
│ (routing.py) │ tier ≥ required
│ │ freshness (active, not stale)
│ │ access_level allowed
│ │ latency_class compatible
└──────────┬───────────┘
▼
┌─────────────────────┐
│ Quality-first select │ max proficiency, cheapest
│ (routing.py) │ among equals, under a
│ │ per-request kWh ceiling
└──────────┬───────────┘
▼
┌──────────────────────┐
│ Dispatcher / Provider │──▶ Neuralwatt Cloud
│ (FastAPI API) │──▶ OpenAI-compatible /v1
│ │ Streaming SSE + energy scrape
└───────┬──────────────┘
▼
┌─────────────────────────┐
│ Verification Pipeline │
│ Layer 1: Structural │ ast.parse, json.load, yaml.safe_load
│ Layer 2: Local LLM │ async, ~6s, >600 token gate
└──────┬──────────────────┘
│
▼
┌─────────────────────┐
│ feedback.py │ failures → proficiency updates
│ (on-demand agent) │ idempotent, attributed-only
└─────────────────────┘
```
## Tech Stack
| Layer | Technology |
|---|---|
| **Language** | Python 3 (IO-bound provider APIs; iteration speed matters more than raw speed) |
| **Framework** | FastAPI + uvicorn |
| **Database** | SQLite (`router.db`) — decision table, energy observations, proficiency, verifications |
| **Local Classification** | Ollama (OpenAI-compatible at `localhost:11434/v1`) |
| **Local Model** | `qwen3.5:latest` |
| **Cloud Provider** | Neuralwatt only |
| **Config** | `config.yaml` loaded & validated by Pydantic (`config.py`) |
| **OpenAI Client** | `openai==3.0.0` (official SDK) |
| **HTTP** | `requests` for poller, `httpx` (via openai/uvicorn) |
| **Testing** | `pytest` — 207 tests across 11 files |
| **Config Files** | `config.yaml`, `leaderboards.yaml`, `evals/tasks.yaml` |
| **Deployment** | systemd user units (`.service` + `.timer` files in `deploy/`) |
| **Integration** | Pre-configured in `opencode.json` — `cd` into repo uses router by default |
## Pinned Dependencies (requirements.txt)
```
pyyaml==6.0.3
pydantic==2.13.4
requests==2.34.2
fastapi==0.141.1
uvicorn[standard]==0.52.1
openai==3.0.0
python-dotenv==1.2.2
pytest==9.1.1
pytest-cov==7.1.0
```
**Pinned intentionally.** Recreating the venv with `>=` constraints silently jumped
`openai 2.53 → 3.0` and `httpx → httpx2` without warning; a service that
restarts on boot shouldn't change its dependency tree underneath itself.
## Modules
| Module | Role | I/O? |
|---|---|---|
| **`dispatcher.py`** | FastAPI service: routes, calls providers, logs, streams | Yes (DB, network) |
| **`poller.py`** | Fetches Neuralwatt catalog, normalizes, upserts `models` table | Yes (network, DB) |
| **`scoring.py`** | `normalize_inverted` + `composite_score` weighted formula | Pure |
| **`routing.py`** | Hard filters (`select_candidates`) + ranking (`rank_candidates`) | Pure |
| **`tiering.py`** | Pure tier resolver: 1=cheap, 2=mid, 3=frontier | Pure |
| **`tier.py`** | DB tiering pass: reads models, resolves, writes `tier` column | Yes (DB) |
| **`proficiency.py`** | Score blending: leaderboard + self-eval → weighted composite | Pure |
| **`proficiency_store.py`** | DB access for `proficiency` table; single write path ensuring `blended_score` never drifts | Yes (DB) |
| **`seed_energy.py`** | Reference workload sweep: fixed prompt × N runs per model | Yes (network, DB) |
| **`eval_proficiency.py`** | Self-eval harness: 4 scoring kinds against N models | Yes (network, DB, subprocess) |
| **`verification.py`** | Two-layer check: structural parse (always) + local LLM spot-check (async, size-gated). Never executes model output | Pure |
| **`feedback.py`** | Folds observed verification failures into `proficiency`; routing learns from real traffic | Yes (DB) |
| **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) |
| **`config.py`** | YAML loader + Pydantic validators (weights sum to 1, valid tiers, etc.) | Yes (file) |
## Decision Table Schema (SQLite)
Three data tables plus one observability table, `PRAGMA foreign_keys = ON`:
### `models` — one row per served model variant
| Column | Type | Notes |
|---|---|---|
| `model_id` | TEXT | Full catalog id (e.g. `glm-5.2-short-fast-flex`) |
| `provider` | TEXT | `neuralwatt` |
| `base_model_id` | TEXT | Model family (e.g. `glm-5.2`). Proficiency/leaderboard keys here. |
| `display_name` | TEXT | Human-readable name |
| `cost_per_1m_prompt` | REAL | Listed USD per 1M input tokens |
| `cost_per_1m_completion` | REAL | Listed USD per 1M output tokens |
| `cost_per_1m_prompt_cached` | REAL | Cached prefix price (null if no cache discount) |
| `context_window` | INTEGER | Advertised max tokens |
| `effective_context_window` | INTEGER | `advertised × safety_factor − reserve` |
| `max_output_tokens` | INTEGER | |
| `tier` | INTEGER | 1–3, set by `tier.py` pass |
| `supports_tools` | INTEGER | Boolean 0/1 |
| `supports_json_mode` | INTEGER | |
| `supports_vision` | INTEGER | |
| `supports_reasoning` | INTEGER | "API accepts reasoning param" — NOT a quality signal |
| `reasoning_default_enabled` | INTEGER | The actual tier-bearing signal |
| `latency_class` | TEXT | `standard` \| `flex` (`-flex`: discounted async, held during peak) |
| `reasoning_mode` | TEXT | `default` \| `reduced` (`-fast`: reasoning capped) |
| `context_variant` | TEXT | `full` \| `short` (`-short`: 200K pool bounded budget) |
| `access_level` | TEXT | `public` \| `preview` \| `canary` |
| `pricing_tbd` | INTEGER | |
| `deprecated` | INTEGER | |
| `availability` | TEXT | `active` \| `deprecated` \| `stale` |
| `last_updated` | TEXT | ISO8601 |
**Serving class:** Neuralwatt ships ~6 base models as 19 catalog rows. The id
suffixes are three **orthogonal** dimensions (`glm-5.2-short-fast-flex`), parsed
by `poller.parse_serving_class` into columns. Rows carry identical catalog
pricing, so routing would pick between them arbitrarily without these — the
`latency_tolerance` hard filter resolves it.
**Access gating:** 6 of 19 rows are prose-gated
("Private preview (grant-gated)", "(Canary)"). `poller.parse_access_level`
parses them into `access_level` and `routing.allowed_access_levels` (default
`[public]`) excludes them, so dispatch won't earn a 403.
### `proficiency` — one row per (model, provider, category)
| Column | Type | Notes |
|---|---|---|
| `model_id` | TEXT | |
| `provider` | TEXT | |
| `category` | TEXT | See category list below |
| `leaderboard_score` | REAL | 0–1, from external benchmarks |
| `self_eval_score` | REAL | 0–1, from self-eval harness |
| `self_eval_samples` | INTEGER | Evidence count for blending threshold |
| `blended_score` | REAL | Computed: `w_lb × lb + w_se × se` |
| `source` | TEXT | `blended` \| `self_eval` \| `self_eval_thin` \| `leaderboard` |
| `last_updated` | TEXT | ISO8601 |
**Category set** (9 categories, defined in `config.yaml`):
| Category | Example | Scoring type |
|---|---|---|
| `coding_general` | Merge intervals, parse semver, word wrap | Code (execution) |
| `coding_refactor` | Remove repetition, refactor dispatch chain | Code (execution) |
| `debugging` | Fix closure leak, fix binary search, fix regex | Code (execution) |
| `reasoning_math` | Percent trap, rate trap, counting | Exact match |
| `tool_use_agentic` | Right tool / right args / no tool when empty | Structural |
| `docs_writing` | Docstring quality, must-mention gotchas | Judge |
| `summarization` | Root-cause isolation, buried-lede identification | Judge |
| `translation` | Technical register, hedging/informal tone | Judge |
| `general_chat` | Simple explanations, measured pushback | Judge |
**Scoring kinds** (4 types, objective wherever the category admits it):
- **`code`** — runs model-generated Python in a subprocess, scores fraction of checks passing
- **`exact`** — normalizes & compares a single answer
- **`tool`** — structural: which tool was called, which args matched
- **`judge`** — a strong model scores against a rubric (prose categories only)
**Blending:** Once `self_eval_samples ≥ self_eval_min_samples` (default 10):
`blended = 0.3 × leaderboard + 0.7 × self_eval`. Before that, falls back
to leaderboard alone. If neither exists, the row scores neutral 0.5.
Source labels distinguish confidence levels: `self_eval_thin` means
real measurement but below the sample threshold — a caller wanting to
exclude it can.
**Proficiency inheritance:** `propagate_to_variants` copies evaluated
`scores` to equivalent serving variants (same weights, same reasoning
setting, same context pool), but only when the variant has no measured
data itself. A `-fast` row is **not** equivalent to its `-standard` sibling.
### `energy_observations` — per-request telemetry
| Column | Type | Notes |
|---|---|---|
| `id` | INTEGER | Autoincrement |
| `model_id` / `provider` | TEXT | |
| `task_category` | TEXT | |
| `prompt_tokens` / `completion_tokens` | INTEGER | |
| `energy_kwh` | REAL | **Attributed** billed figure (noisy, 20× within-model) |
| `energy_btu` | REAL | `kwh × 3412.14`, dashboard value |
| `avg_power_watts` / `duration_seconds` | REAL | Pre-attribution product, ~1.8× within-model |
| `attribution_ratio` | REAL | Request's share of shared GPU pool (stable quantized: 0.001, 0.25, 0.5, 0.75) |
| `carbon_g_co2eq` | REAL | Reported by provider |
| `grid_carbon_intensity` | REAL | gCO2/kWh at call time |
| `grid_id` | TEXT | e.g. `FI` |
| `carbon_source` | TEXT | `static_fallback` (constant, excluded from routing) or live measurement |
| `cost_usd` | REAL | **Billed** figure, not tokens × list price |
| `allowance_remaining_usd` | REAL | |
| `service_tier` | TEXT | As billed |
| `observed_at` | TEXT | ISO8601 |
**Attribution noise:** Billed `energy_kwh = avg_power_watts × duration ×
attribution_ratio`. The attribution term looks like noise up close (8 identical
calls varied 20×), but ranks 750× between models while within-model spread is
1.8× — it's a stable per-model property reflecting serving concurrency. Routing
scores on the attributed figures with a **median** over all `seed_reference`
rows, so repeated sweeps accumulate into a median-across-time.
### `verifications` — response quality observations
| Column | Type | Notes |
|---|---|---|
| `id` | INTEGER | Autoincrement |
| `model_id` / `provider` | TEXT | |
| `task_category` | TEXT | Optional (prose answers may lack a category) |
| `kind` | TEXT | `structural` \| `local_llm` |
| `verdict` | TEXT | `ok` \| `truncated` \| `malformed` \| `unverifiable` |
| `detail` | TEXT | Human-readable reason |
| `completion_tokens` | INTEGER | Wasted answer cost, for payoff sum |
| `observed_at` | TEXT | ISO8601 |
| `applied_at` | TEXT | Set by `feedback.py` when folded into proficiency |
| `model_attributable` | INTEGER | 1 = model's fault; 0 = client caused (e.g. tight cap) |
Verifications drive the feedback loop: `feedback.py` reads unanswered failures,
applies a 0.0 sample per failure to `proficiency`, and marks them
`applied_at` for idempotency. `unverifiable` is recorded but not treated as a
failure — it means the checker had nothing to say, not that the model failed.
## Weighted Scoring
Three hard filters are applied **before** scoring (not weighted — outright disqualification):
1. `effective_context_window ≥ required_context_tokens`
2. `tier ≥ required_tier` (from classifier)
3. Serving class compatible with request's `latency_tolerance`; `access_level` reachable
```
1. drop candidates whose measured energy exceeds objective.max_energy_per_request
2. rank by proficiency for the task's category
3. treat differences smaller than objective.quality_tolerance as equal
4. among equals, pick the cheapest
```
| Setting | Value | Notes |
|---|---|---|
| `quality_tolerance` | 0.10 | Measurement noise, not preference: scores rest on 2-3 samples, so smaller gaps are sampling variation |
| `max_energy_per_request` | null | Per-request kWh ceiling. The plan is a 6.25 kWh quota — a wall, not a bill |
| `plan_kwh_per_period` | 6.25 | Reported in `/health` as burn against the allowance |
This replaced a weighted blend (cost 0.4 / eco 0.2 / proficiency 0.4).
Measurement retired it: turning the cost weight from 0.4 to **zero** changed
the winner in only 2 of 6 categories, so the blend was never steering on
quality — while 60% of every decision adjudicated fractions of a cent.
**Why cost ≠ list price:** Neuralwatt bills flat $8.00/kWh. The catalog's
`input_per_million` / `output_per_million` are **not** what gets charged.
Measured: `cost_usd / energy_kwh = 8.00` across every model. A model listing
at $4/1M can cost 10× more than one at $15/1M on the same prompt.
**Why cost ≠ eco:** Cost tracks energy (kWh), but carbon is energy × grid
intensity. Grid intensity spans ~49 gCO2/kWh (`FI`) to ~442 (`US-MIDA-PJM`) — roughly a 9× spread,
and it moves: the provider's own 24h blended figure was 145.4. The models disagree: `glm-5.2-fast` is
2nd cheapest but 6th cleanest; `kimi-k3-flex` draws 3.7× less energy than
`kimi-k2.7-code` while emitting 3.6× more carbon. Collapsing them picks a
side.
### What routing actually does today — expect one model
Run `/route` across all 9 categories and 3 tiers and every one of those 27
decisions currently returns **`qwen3.6-35b`**. That is not a bug and not a
misconfiguration, so it is worth stating plainly before you go looking for
one.
On this catalog `qwen3.6-35b` is **Pareto-dominant**: cheapest *and* cleanest
in the routable set, while scoring within 0.15 of the best model on
proficiency. On `summarization` it costs 7× less and emits 65× less carbon
than `kimi-k3`, which beats it 1.00 to 0.85 on quality — no defensible
weighting picks `kimi-k3` there. One model winning is the correct answer to
the question the weights ask.
Proficiency still does work underneath: runner-up ordering reorders by
category (`kimi-k3-fast` climbs to 3rd on `docs_writing` and `summarization`
where it scores 1.00, and leaves the top four elsewhere). The axis is live;
it simply cannot overturn a leader that wins on two axes at once.
The spread widens when the leader stops being eligible. Past ~94,196 tokens
of context `qwen3.6-35b` is filtered out and `glm-5.2-fast` takes over; past
~790K nothing qualifies and `/v1/chat/completions` returns 422 naming the
constraint rather than truncating. So on a large codebase you will see the
model change mid-session — that is the context filter working.
If a different balance is wanted, the lever is `weights` in `config.yaml`,
not more data.
## API Endpoints
The dispatcher binds `127.0.0.1:8080`. **No auth of its own** — loopback
is the only thing standing between the open internet and your billing
allowance.
| Method | Path | Description |
|---|---|---|
| `GET` | `/health` | Catalog/reachability status, scoring coverage, warnings |
| `POST` | `/route` | Classify task, rank candidates, return selected model — **no provider call, no cost** |
| `POST` | `/dispatch` | Same as `/route`, plus complete the provider call, stream response, log observation |
| `GET` | `/v1/models` | OpenAI-compatible model list (router virtual models + catalog) |
| `POST` | `/v1/chat/completions` | OpenAI-compatible completions — routes then proxies, **streaming supported** |
Input to `/route` and `/dispatch` can include `task_category`, `task_tier`,
and `required_context_tokens` overrides — these skip the classifier, useful
for testing routing without the classifier in the loop.
Two virtual router models:
- `auto` — normal routing, `-flex` rows excluded (interactive)
- `auto:batch` — admits `-flex` rows (overnight/async work)
Ask for **any real model id** in `/v1/chat/completions` and it dispatches
directly, still logged — routing is transparent, not opaque.
**Streaming** (chunk-by-chunk proxy): Tokens render as they arrive. Neuralwatt
emits energy and cost as SSE **comment** lines (`: energy {...}`) before
`data: [DONE]` — ordinary clients ignore comments, so the stream flows
untouched while the router scrapes telemetry on the way past. Without this,
streamed calls would log no energy at all.
**Verification headers** (non-streaming): When streaming is not used, the
structural verification verdict surfaces in the `X-Router-Verification` header
so a client can inspect it without parsing the response body. Valid values:
`ok`, `truncated`, `malformed`, `unverifiable`, `none`.
## Quick Usage
```bash
# Pick a model without spending anything
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
-d '{"task":"Refactor this 800-line Django view into service objects."}'
# With explicit category/tier overrides (skips classifier)
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
-d '{"task":"Refactor this 800-line Django view","task_category":"coding_refactor","task_tier":2}'
# Actually call the winner and log energy/cost
curl -s -X POST localhost:8080/dispatch -H 'content-type: application/json' \
-d '{"task":"What is a Python context manager?"}'
# OpenAI-compatible — works with any SDK or agent client
curl -s localhost:8080/v1/models
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'
# Admit flex rows for overnight/async work
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
-d '{"task":"nightly code review","latency_tolerance":"batch"}'
```
## Scheduled Jobs (systemd)
Five user units. See `deploy/README.md` for full install/operate instructions.
| Unit | Schedule | What it runs |
|---|---|---|
| `llm-router.service` | Continuous | FastAPI dispatcher |
| `llm-router-poller.timer` | 2 min after boot, then every 2 h | `poller.py` → `tier.py` |
| `llm-router-seed.timer` | Every 6 h | Small `seed_energy.py` sweep |
**The poller timer is load-bearing, not optional.** `freshness.stale_after_days`
is 3 with `exclude_stale: true` — an unpolled catalog marks every row stale
in 3 days and the router returns zero candidates for everything.
**The seed timer spans time.** Energy attribution drifts with pool load
across hours (~50× for `deepseek-v4-flash` between sweeps), so a single
sweep measures one moment. The median has to span time — every 6 h sweep
accumulates into a time-weighted median automatically.
## Self-Eval Harness (`eval_proficiency.py`)
```bash
python eval_proficiency.py # every routable model × every task
python eval_proficiency.py --models kimi-k3 # subset of models
python eval_proficiency.py --categories coding_general
python eval_proficiency.py --dry-run # plan only
```
- Runs every task through the target provider, scores it, writes to `proficiency`.
- Scores **accumulate** (running mean), so repeated runs tighten estimates.
- `propagate_to_variants` copies evaluated scores to equivalent serving variants.
- Safety: code tasks run in a temp directory with a 15s wall-clock timeout —
bounded isolation, not a container.
- Judge tasks skip the model being judged (avoids self-scoring bias).
## Classifier Reliability Notes
Several tuning decisions keep the classifier from cascading failures:
- **`max_retries: 0`** on the classifier client. The OpenAI SDK retries twice
by default, so `timeout_seconds` silently became a 3× wall-clock bound — a
request could hang past 250s on a 120s setting with no log.
- **`max_output_tokens: 1024`** bounds a reasoning model's chain of thought.
Without it, `qwen3.5` emits unbounded traces — Ollama keeps generating after
the client gives up AND serializes per model, so one runaway request queues
every later request behind it. 256 was tried first and was too tight; the
trace would consume the whole budget and the model would be truncated before
emitting JSON.
- **`temperature: 0`** prevents non-reproducible classifications. At the
default, the same prompt classified tier 2 then tier 1 on consecutive calls,
routing to two different models.
- **Graceful fallback** routes to `fallback_tier: 2` / `fallback_category:
general_chat` when the classifier times out, errors, or returns
unparseable output. A coding agent would rather have a mid-tier answer than
an error. Escalation deliberately skips fallbacks so an unavailable local
model doesn't silently promote every request to the frontier tier.
## Pointing a Coding Agent at It
The `/v1` endpoints are OpenAI-compatible. Repo-local `opencode.json` is
already wired up (`cd ~/Sources/6krrt && opencode` routes by the router by
default). For global use, merge `provider.llm-router` into
`~/.config/opencode/opencode.json`.
Virtual model names:
- `auto` → router picks, interactive mode (flex rows excluded)
- `auto:batch` → router picks, admits flex rows for async work
## Testing
```bash
python -m pytest # run all 207 tests
python -m pytest --cov # with coverage
```
| Test file | What it covers |
|---|---|
| `test_scoring.py` | `normalize_inverted`, `composite_score`, None-handling |
| `test_routing.py` | Hard filters, candidate selection & ranking across all 9 categories × 3 tiers |
| `test_tiering.py` | Tier resolver precedence: override → reasoning → cost → mid |
| `test_apply_tiering.py` | DB tiering pass, sanity guards |
| `test_poller_parsing.py` | Serving class, base model, access level parsing |
| `test_proficiency.py` | Blending rule, accumulation |
| `test_load_candidates.py` | `load_candidates`: cost/eco/proficiency join |
| `test_eval_scoring.py` | Code, exact, tool, and judge scoring |
| `test_task_set.py` | Reference solutions validating every `code` task's checks & `exact` answers |
| `test_verification.py` | Structural checks on every language, verifier edge cases, local-LLM gating |
| `test_feedback.py` | Failure identification, idempotency, attribution filtering |
## Setup
```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
sqlite3 router.db < schema.sql
cp .env.example .env # fill in NEURALWATT_API_KEY
python poller.py # populate the catalog
python tier.py # resolve tiers
python seed_energy.py # seed reference energy observations (~5 per model)
python -m uvicorn dispatcher:app --reload
```
Ollama must be running locally:
```bash
ollama pull qwen3.5:latest
```
## Known Limitations & Open Items
- **Leaderboard priors unfilled** — `leaderboards.yaml` ships empty. `python leaderboard.py --check`
names every active family lacking a prior. Must source figures from
published benchmarks; inventing plausible numbers is the same failure as
the provider's `static_fallback` carbon constant.
- **Three models unsettled** — split-half stability at 7 samples:
`kimi-k2.7-code-fast` (29×), `kimi-k3` (14×), `glm-5.2-flex` (2.2×).
More samples needed before their positioning is trustworthy.
- **Verification observes but never intervenes** — failures are detected and
fed back into `proficiency` automatically (`verification.py` →
`feedback.py`), so the router does learn which models fail on real work.
What is missing is action: nothing retries or escalates on a failed check.
That is deliberate — whether auto-escalation pays should be decided from
the observed failure rate, which is now being collected rather than
guessed at.
- **No auth** — the service holds a billable API key with no
authentication of its own. Loopback binds only.
- **Context assembly (RAG)** is out of scope — the classifier sees the full
conversation but does not perform document/code retrieval.

114
config.py
View File

@@ -16,20 +16,35 @@ import yaml
from pydantic import BaseModel, field_validator, model_validator
class Weights(BaseModel):
cost: float
eco: float
proficiency: float
class Objective(BaseModel):
"""What the router optimizes: quality, bounded by cost.
@model_validator(mode="after")
def must_sum_to_one(self) -> "Weights":
total = round(self.cost + self.eco + self.proficiency, 6)
if total != 1.0:
Replaced a three-way weighted blend. See config.yaml for why — briefly,
the cost weight was measured to be nearly inert while consuming 40% of
every decision.
"""
quality_tolerance: float = 0.10
assumed_cache_rate: float = 0.84
assumed_completion_tokens: int = 500
max_energy_per_request: Optional[float] = None
plan_kwh_per_period: Optional[float] = None
@field_validator("quality_tolerance")
@classmethod
def tolerance_in_range(cls, v: float) -> float:
if not (0.0 <= v < 1.0):
raise ValueError("objective.quality_tolerance must be in [0, 1)")
return v
@field_validator("max_energy_per_request")
@classmethod
def ceiling_positive(cls, v: Optional[float]) -> Optional[float]:
if v is not None and v <= 0:
raise ValueError(
f"weights.cost + weights.eco + weights.proficiency must sum to "
f"1.0, got {total}"
"objective.max_energy_per_request must be > 0 kWh, or null to disable"
)
return self
return v
class ContextConfig(BaseModel):
@@ -64,6 +79,7 @@ class ProficiencyConfig(BaseModel):
class TieringConfig(BaseModel):
cheap_completion_max: float
tier1_context_max: float = float("inf")
model_tiers: dict[str, int]
@field_validator("cheap_completion_max")
@@ -73,6 +89,13 @@ class TieringConfig(BaseModel):
raise ValueError("tiering.cheap_completion_max must be > 0")
return v
@field_validator("tier1_context_max")
@classmethod
def context_max_must_be_positive(cls, v: float) -> float:
if v <= 0:
raise ValueError("tiering.tier1_context_max must be > 0")
return v
@field_validator("model_tiers")
@classmethod
def overrides_in_range(cls, v: dict[str, int]) -> dict[str, int]:
@@ -85,10 +108,69 @@ class TieringConfig(BaseModel):
return v
class RoutingConfig(BaseModel):
allowed_access_levels: list[str]
default_latency_tolerance: str
@field_validator("allowed_access_levels")
@classmethod
def levels_known(cls, v: list[str]) -> list[str]:
known = {"public", "preview", "canary"}
unknown = set(v) - known
if unknown:
raise ValueError(
f"routing.allowed_access_levels contains unknown levels {sorted(unknown)}; "
f"must be a subset of {sorted(known)}"
)
if not v:
raise ValueError("routing.allowed_access_levels must not be empty")
return v
@field_validator("default_latency_tolerance")
@classmethod
def tolerance_known(cls, v: str) -> str:
if v not in ("interactive", "batch"):
raise ValueError(
f"routing.default_latency_tolerance must be 'interactive' or 'batch', got {v!r}"
)
return v
class VerificationConfig(BaseModel):
local_llm_enabled: bool = True
min_completion_tokens: int = 600
timeout_seconds: int = 60
max_output_tokens: int = 1024
outcome_attribution_window_seconds: int = 120
class EscalationConfig(BaseModel):
enabled: bool
max_tier: int
min_confidence_before_bump: float
# Off by default: the iteration budget escalates on evidence instead.
preemptive_on_low_confidence: bool = False
class IterationConfig(BaseModel):
"""A tier's budget for corrective attempts after a verification failure."""
enabled: bool = True
attempts_by_tier: dict[int, int] = {1: 0, 2: 1, 3: 2}
max_attempts_interactive: int = 1
@field_validator("attempts_by_tier")
@classmethod
def attempts_sane(cls, v: dict[int, int]) -> dict[int, int]:
for tier, attempts in v.items():
if attempts < 0:
raise ValueError(f"iteration.attempts_by_tier[{tier}] must be >= 0")
if attempts > 5:
raise ValueError(
f"iteration.attempts_by_tier[{tier}]={attempts} is implausibly "
"high; each attempt spends energy against a fixed quota"
)
return v
class FreshnessConfig(BaseModel):
@@ -106,6 +188,11 @@ class ClassifierConfig(BaseModel):
base_url: str
model: str
timeout_seconds: int
temperature: float = 0.0
max_output_tokens: int = 1024
outcome_attribution_window_seconds: int = 120
fallback_tier: int = 2
fallback_category: str = "general_chat"
response_format: str
system_prompt: str
@@ -121,12 +208,15 @@ class LoggingConfig(BaseModel):
class RouterConfig(BaseModel):
weights: Weights
objective: Objective
context: ContextConfig
tiers: dict[int, str]
tiering: TieringConfig
proficiency: ProficiencyConfig
routing: RoutingConfig
verification: VerificationConfig = VerificationConfig()
escalation: EscalationConfig
iteration: IterationConfig = IterationConfig()
freshness: FreshnessConfig
database: DatabaseConfig
classifier: ClassifierConfig

184
config.yaml
View File

@@ -2,11 +2,55 @@
# All weights, thresholds, and provider settings live here so they can be
# tuned without touching code. Loaded/validated by config.py.
weights:
cost: 0.4
eco: 0.2
proficiency: 0.4
# Must sum to 1.0 — config.py will raise on load if they don't.
objective:
# Quality is the objective. Cost is a constraint and a tiebreak. Eco is
# logged per request but is NOT optimized here — that judgement is made
# outside this router.
#
# This replaced a weighted blend (cost 0.4 / eco 0.2 / proficiency 0.4).
# Measurement killed it: turning the cost weight from 0.4 to ZERO changed
# the winner in only 2 of 6 categories, so the blend was never steering on
# quality — while 60% of every decision adjudicated fractions of a cent
# (all real traffic to date totals $0.07).
# Proficiency differences smaller than this are treated as equal and the
# cheaper model wins. This is measurement noise, not preference: scores
# currently rest on 2-3 samples per category, so a 0.05 gap is
# indistinguishable from sampling variation and paying for it buys noise.
# Narrow it as samples accumulate.
quality_tolerance: 0.10
# Cost is priced per-request from catalog prices, NOT from a benchmark
# sweep. A fixed 400-token reference task ranked glm-5.2-fast 3.2x cheaper
# than deepseek-v4-flash; on a realistic 70k-token prompt deepseek is 5.0x
# cheaper. Attribution inverts with prompt size, so a fixed-shape benchmark
# cannot rank models for a workload of another shape. Catalog prices scaled
# to the actual request agree with the live measurement, cost nothing, and
# need no sweep.
# Share of prompt tokens served from the provider's prefix cache. Measured
# from real traffic: 1.9M of 2.2M prompt tokens, 84%. Agent clients resend
# the whole conversation each turn, so most of it is a cache hit.
assumed_cache_rate: 0.84
# Completion length assumed when pricing a request. Real sessions here median
# around 200-400 completion tokens against enormous prompts.
assumed_completion_tokens: 500
# Per-request ceiling on measured ENERGY, in kWh. null disables it.
#
# Denominated in kWh rather than dollars because the plan is a subscription
# with a 6.25 kWh quota, not pay-per-request. Dollars accrue; a quota is a
# wall you hit mid-task. So this is the cost mandate stated as a guarantee.
# For scale: the reference task runs ~5e-06 kWh on the cheapest model and
# ~2.2e-04 on the most expensive.
max_energy_per_request: null
# The subscription's kWh allowance per billing period, for reporting burn in
# /health. Set to match your plan; null disables the report. NeuralWatt also
# returns allowance_remaining_usd per request, which is logged, but that is a
# dollar figure while the plan is denominated in energy.
plan_kwh_per_period: 6.25
context:
safety_factor: 0.75 # fraction of advertised context treated as usable
@@ -28,6 +72,19 @@ tiering:
# model_tiers overrides the heuristic per model_id and applies to ALL
# providers (limitation vs a (model_id, provider) key).
cheap_completion_max: 1.00
# Advertised context_window at or above which a model is NOT eligible for
# tier 1, whatever it costs. Tier is a capability FLOOR (routing drops any
# row with tier < required_tier), so tier 1 means "simple work only" — and
# deciding that on price alone excluded deepseek-v4-flash from every tier-2
# request purely for being $0.28/1M, despite a 1M window and 1.00 on all
# three coding categories.
#
# 512000 sits in the empty band between the catalog's 256K class (262128)
# and its 1M class (1048560) — a 2x margin either side, so it is not fitted
# to any one model. gemma-4-31b (256K) stays tier 1; the 1M rows do not.
tier1_context_max: 512000
model_tiers: {}
proficiency:
@@ -51,10 +108,78 @@ proficiency:
escalation:
enabled: true
max_tier: 3
# If the local model's classification confidence is below this, bump
# required_tier by one as a precaution rather than trusting a shaky call.
# Bump the tier when the classifier is unsure of its own call. DEFAULT OFF:
# this pays frontier prices on a hunch, before anything has gone wrong. The
# iteration budget below spends after a check has actually failed, which is
# strictly better on both mandates — the cheap attempt usually succeeds and
# costs nothing extra, and when it fails you have evidence.
preemptive_on_low_confidence: false
min_confidence_before_bump: 0.6
iteration:
# A tier is not only a capability floor, it is a budget for getting the
# answer right. These are corrective attempts AFTER a verification failure,
# not speculative retries.
#
# Retries are matched to the failure: a truncated answer gets a bigger token
# budget on the SAME model (a different one would also run out), while a
# malformed answer escalates to the next-best candidate (more tokens will
# not make unparseable output parse).
enabled: true
attempts_by_tier:
1: 0 # cheap/simple — one shot; iterating costs more than it is worth
2: 1
3: 2
# Interactive requests are capped below their tier's budget regardless of
# tier: every retry doubles time-to-answer, and in interactive use latency
# IS a quality loss. Batch work does not care.
max_attempts_interactive: 1
routing:
# Access gating is prose-only in the NeuralWatt catalog ("Private preview
# (grant-gated)", "(Canary)"), so the poller parses it into access_level and
# routing excludes anything not listed here. Add 'preview'/'canary' only if
# the account actually holds the grant — otherwise dispatch earns a 403.
allowed_access_levels:
- public
# '-flex' rows are held server-side during peak until a capacity gap opens.
# That's correct for overnight/batch agent work and wrong for anything
# interactive, so a request has to opt in via latency_tolerance.
default_latency_tolerance: interactive # 'interactive' | 'batch'
# There is deliberately no flex discount knob. Cost scoring reads the mean
# cost actually billed for the reference workload (see seed_energy.py), and
# a flex row's measured cost already is its flex cost.
verification:
# Structural checks (parse the code, never run it) are free and always on.
# This section governs the LOCAL LLM check, which is not free.
local_llm_enabled: true
# Only check answers this large. Measured on real traffic: a local check
# costs ~15% of a median 193-token answer, so it would only pay if such
# answers failed more than ~15% of the time. At 1,500 completion tokens the
# break-even failure rate drops to ~1.9%, which is plausible. Below the
# threshold a check costs more than the risk it removes.
min_completion_tokens: 600
# The check runs AFTER the response has gone back to the client, so it never
# adds its ~6s to anyone's latency. It exists to learn which models fail on
# real work, not to gate answers.
timeout_seconds: 60
max_output_tokens: 1024
# How far back /outcome looks when a report carries no request_id and no
# matching directory. A test run follows the completion that caused it within
# seconds, so this is deliberately short: a wide window sweeps in sessions
# that finished long ago and makes every report look ambiguous. If more than
# one conversation was active inside it, the report is refused rather than
# guessed at.
outcome_attribution_window_seconds: 120
freshness:
stale_after_days: 3
# Router refuses to route to a model whose row is stale/deprecated,
@@ -69,23 +194,54 @@ classifier:
# Local Ollama instance doing task classification / context sizing.
provider: "ollama"
base_url: "http://localhost:11434/v1"
model: "qwen2.5:32b"
timeout_seconds: 30
model: "qwen3.5:latest" # must match a model `ollama list` reports
# A cold Ollama took 43s to answer the first classification, which blew the
# old 30s ceiling and returned 503 to the client. Warm it is ~5s. The
# ceiling is for a cold model load, not the steady state.
timeout_seconds: 120
# Classification must be reproducible: at the default temperature the same
# task was classified tier 2 then tier 1 on consecutive calls, which routed
# it to two different models. Routing that changes under an identical
# prompt is untraceable.
temperature: 0
# Hard cap on the classifier's generation, to bound a failure mode that
# cascades: qwen3.5 is a reasoning model and will sometimes emit a long
# chain of thought, blowing past timeout_seconds — and Ollama keeps
# generating after the client gives up AND serializes per model, so one
# runaway request queues every later request behind it.
#
# Must leave room for the thinking trace, not just the ~45-token answer. At
# 256 the trace consumed the whole budget on some prompts and the model was
# truncated before emitting any JSON at all, which read as an empty
# response. 1024 bounds the runaway while leaving the answer reachable.
max_output_tokens: 1024
# Where routing lands when the classifier times out, errors, or returns
# something unparseable. A local model being slow should degrade routing,
# not refuse the request — the caller is a coding agent that would rather
# have a mid-tier answer than a 502.
fallback_tier: 2
fallback_category: general_chat
response_format: "json" # ask Ollama to constrain output to valid JSON
# The dispatcher appends the authoritative category list from
# proficiency.categories to this prompt at call time. Do not enumerate the
# categories here as well — a hand-copied list drifts, and a category the
# model invents joins against nothing in the proficiency table.
system_prompt: |
You are a task router. Given a task description and any attached context,
respond with ONLY a JSON object with these fields:
{
"task_category": one of the configured categories,
"task_tier": integer 1-3,
"task_category": one of the allowed categories listed below,
"task_tier": integer 1-3, where 1 is cheap/simple, 2 is mid/general,
and 3 is frontier/high-stakes,
"required_context_tokens": integer estimate of prompt+context token count,
"confidence": float 0-1
}
dispatch_providers:
openrouter:
base_url: "https://openrouter.ai/api/v1"
api_key_env: "OPENROUTER_API_KEY"
# NeuralWatt is the only provider. The dict shape and the models table's
# (model_id, provider) key are kept so a second one can be added without a
# migration.
neuralwatt:
base_url: "https://api.neuralwatt.com/v1"
api_key_env: "NEURALWATT_API_KEY"

81
deploy/README.md Normal file
View File

@@ -0,0 +1,81 @@
# Deploying the router
Three units. The dispatcher runs continuously; the poller runs on a timer and
is **not** optional — `freshness.stale_after_days` is 3 and
`freshness.exclude_stale` is true, so a catalog that goes unpolled for three
days marks every row stale and the router stops returning any candidate at
all.
| file | what it does |
|---|---|
| `llm-router.service` | the FastAPI dispatcher, on `127.0.0.1:8080` |
| `llm-router-poller.service` | one-shot: `poller.py` then `tier.py` |
| `llm-router-poller.timer` | fires the poller 2 min after boot, then every 2 h |
| `llm-router-seed.service` | one-shot: a small `seed_energy.py` reference sweep |
| `llm-router-seed.timer` | every 6 h — energy attribution drifts with pool load across hours, so the median has to span time rather than one sweep |
These are **user** units — no root, and they run as you with your own
`$HOME`. The tradeoff is that a user service does not inherit your shell
environment, so the API key has to come from a file.
## Install
```bash
# 1. The key. User units don't see your shell env, so .env is required.
cd ~/Sources/6krrt
echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env
# 2. Install and start
cp deploy/llm-router*.{service,timer} ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now llm-router.service llm-router-poller.timer llm-router-seed.timer
# 3. Survive logout/reboot (user units stop with your session otherwise)
loginctl enable-linger "$USER"
# 4. Check. The health endpoint reports whether cost/eco/proficiency
# actually have data behind them, which is otherwise silent.
curl -s localhost:8080/health | python -m json.tool
systemctl --user list-timers 'llm-router*'
```
## Operating it
```bash
systemctl --user status llm-router.service
journalctl --user -u llm-router.service -f # request log
journalctl --user -u llm-router-poller.service # catalog refreshes
systemctl --user restart llm-router.service # after editing config.yaml
systemctl --user start llm-router-poller.service # force a refresh now
```
`config.yaml` is read once at startup, so weight and threshold changes need a
restart. The catalog is read per-request, so a poller run takes effect
immediately.
## A note on the bind address
`--host 127.0.0.1` is deliberate. The service holds a billable API key and
has **no authentication of its own** — anything that reaches it can spend
your allowance. `ProtectHome=read-only` plus a `ReadWritePaths` exception for
the repo limits the blast radius on the filesystem, but nothing limits spend.
Putting this on a LAN address needs an auth layer first.
## Pointing opencode at it
The repo-local `opencode.json` sets this up already, so running `opencode`
from inside `~/Sources/6krrt` uses the router by default. To use it from
anywhere, merge the `provider.llm-router` block into
`~/.config/opencode/opencode.json` and set `"model": "llm-router/auto"`.
Two model names:
- `llm-router/auto` — normal routing; flex rows excluded, so nothing gets
held server-side during peak
- `llm-router/auto:batch` — admits flex rows, for overnight/async work
`limit.context` is declared as 782324, the largest effective window in the
routable catalog. The router hard-filters on the measured conversation size,
so a prompt too big for the smaller models simply won't be routed to them;
if it fits nothing, `/v1/chat/completions` returns a 422 naming the
constraint rather than truncating.

21
deploy/llm-router-poller.service Normal file
View File

@@ -0,0 +1,21 @@
[Unit]
Description=Refresh the NeuralWatt model catalog and re-resolve tiers
Documentation=file:%h/Sources/6krrt/CLAUDE.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
WorkingDirectory=%h/Sources/6krrt
EnvironmentFile=%h/Sources/6krrt/.env
# poller.py refreshes the catalog; tier.py re-resolves tiers from it. Tiers
# are derived from cost and reasoning fields the poll may have changed, so
# they always run as a pair.
ExecStart=%h/Sources/6krrt/.venv/bin/python poller.py
ExecStart=%h/Sources/6krrt/.venv/bin/python tier.py
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=%h/Sources/6krrt

15
deploy/llm-router-poller.timer Normal file
View File

@@ -0,0 +1,15 @@
[Unit]
Description=Refresh the NeuralWatt model catalog periodically
[Timer]
OnBootSec=2min
OnUnitActiveSec=2h
# Catch up after the machine has been asleep rather than waiting a full
# interval — this is not optional maintenance. freshness.stale_after_days is
# 3 and freshness.exclude_stale is true, so a catalog that goes unpolled for
# three days marks every row stale and the router stops returning ANY
# candidate. The timer is what keeps the service able to route at all.
Persistent=true
[Install]
WantedBy=timers.target

19
deploy/llm-router-seed.service Normal file
View File

@@ -0,0 +1,19 @@
[Unit]
Description=Sample the reference workload to keep cost/eco scoring current
Documentation=file:%h/Sources/6krrt/CLAUDE.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
WorkingDirectory=%h/Sources/6krrt
EnvironmentFile=%h/Sources/6krrt/.env
# Fewer samples per run than a manual sweep, because the point is coverage
# across TIME rather than depth at one moment — see the timer.
ExecStart=%h/Sources/6krrt/.venv/bin/python seed_energy.py --samples 3
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=%h/Sources/6krrt

10
deploy/llm-router-seed.timer Normal file
View File

@@ -0,0 +1,10 @@
[Unit]
Description=Sample the reference workload every few hours
[Timer]
OnBootSec=15min
OnUnitActiveSec=6h
Persistent=true
[Install]
WantedBy=timers.target

34
deploy/llm-router.service Normal file
View File

@@ -0,0 +1,34 @@
[Unit]
Description=Local LLM model router (FastAPI dispatcher)
Documentation=file:%h/Sources/6krrt/CLAUDE.md
# The classifier talks to Ollama on localhost and the dispatcher talks to
# NeuralWatt, so both loopback and real network need to be up. Ollama is a
# system service and cannot be ordered against from a user unit, so a failed
# classifier call is handled by Restart= below rather than by ordering.
After=network-online.target
Wants=network-online.target
[Service]
Type=exec
# config.yaml, router.db and router.log are all referenced as relative paths,
# so this has to be the repo root.
WorkingDirectory=%h/Sources/6krrt
# Holds NEURALWATT_API_KEY. Create it with:
# echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env
EnvironmentFile=%h/Sources/6krrt/.env
ExecStart=%h/Sources/6krrt/.venv/bin/uvicorn dispatcher:app --host 127.0.0.1 --port 8080
Restart=on-failure
RestartSec=5s
# Loopback only by default — this service holds a billable API key and has no
# auth of its own. Anything that widens --host should add auth first.
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
# ...except the repo, which needs to be writable for router.db and router.log.
ReadWritePaths=%h/Sources/6krrt
[Install]
WantedBy=default.target

108
deploy/opencode-plugin/router-outcome.js Normal file
View File

@@ -0,0 +1,108 @@
/**
* Report test outcomes back to the local LLM router.
*
* This closes the only loop the router cannot close by itself. Everything it
* records on its own is a proxy: structural checks know whether code *parses*,
* the local checker guesses whether prose *looks* right. Neither knows whether
* the answer did the job. opencode does — it runs your tests.
*
* Hooks `tool.execute.after`, watches for test/build commands, and POSTs the
* pass/fail to the router's /outcome endpoint. The router folds client
* outcomes into proficiency in BOTH directions (a client's "succeeded" means
* the work worked, unlike a parser's "ok" which only means it parsed), so this
* is what eventually makes routing discriminate on quality.
*
* Install:
* mkdir -p ~/.config/opencode/plugins
* cp router-outcome.js ~/.config/opencode/plugins/
*
* Or per-project, in .opencode/plugins/.
*
* PARALLEL SESSIONS: the report carries this session's directory, which the
* router matches against the working directory it sees in the conversation.
* That is exact even with several sessions running. If it cannot match, and
* more than one conversation has been routed recently, the router answers 409
* and records nothing rather than guessing — a misattributed failure would
* penalize a model for work it never did.
*/
const ROUTER = process.env.LLM_ROUTER_URL || "http://127.0.0.1:8080";
// Commands whose exit status is a real verdict on the work. Deliberately
// narrow: a failing `ls` says nothing about model quality, and a false signal
// is worse than no signal — it trains the router on noise.
const TEST_COMMAND = new RegExp(
[
"\\bpytest\\b",
"\\bunittest\\b",
"\\bnpm\\s+(run\\s+)?test\\b",
"\\bpnpm\\s+(run\\s+)?test\\b",
"\\byarn\\s+test\\b",
"\\bvitest\\b",
"\\bjest\\b",
"\\bcargo\\s+(test|check|build)\\b",
"\\bgo\\s+(test|build|vet)\\b",
"\\bmake\\s+(test|check)\\b",
"\\bmvn\\s+test\\b",
"\\bgradle\\s+test\\b",
"\\btsc\\b",
"\\bruff\\b",
"\\bmypy\\b",
"\\beslint\\b",
].join("|"),
);
// Failure signatures, for tools that exit 0 while reporting failures.
const FAILURE_TEXT =
/\b(\d+\s+failed|FAILED|FAIL\b|Traceback \(most recent call last\)|error(s)?:|panic:|AssertionError|✗|✖)/;
function looksFailed(output) {
const exit = output?.exitCode ?? output?.exit_code;
if (typeof exit === "number" && exit !== 0) return true;
const text = `${output?.stdout ?? ""}\n${output?.stderr ?? ""}\n${
typeof output?.output === "string" ? output.output : ""
}`;
// "0 failed" and "no errors" must not trip the failure regex.
if (/\b0 failed\b|\bno errors?\b/i.test(text)) return false;
return FAILURE_TEXT.test(text);
}
function commandOf(input) {
const args = input?.args ?? input?.arguments ?? {};
return args.command ?? args.cmd ?? args.script ?? "";
}
export const RouterOutcome = async ({ directory }) => {
return {
"tool.execute.after": async (input, output) => {
// Only shell-ish tools carry a command whose exit status is a verdict.
const command = commandOf(input);
if (!command || !TEST_COMMAND.test(command)) return;
const ok = !looksFailed(output);
const detail = `${command.slice(0, 120)}${ok ? " — passed" : " — failed"}`;
try {
const res = await fetch(`${ROUTER}/outcome`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ ok, detail, source: directory }),
// The router being down must never break the user's session.
signal: AbortSignal.timeout(3000),
});
if (res.status === 409) {
// Several sessions active and the directory did not match one the
// router had seen. Dropping the sample is the correct outcome.
console.error(
"[router-outcome] ambiguous session; outcome not recorded",
);
} else if (!res.ok && res.status !== 404) {
console.error(`[router-outcome] ${res.status} reporting outcome`);
}
} catch (err) {
// Swallowed on purpose: a reporting failure is not the user's problem.
console.error(`[router-outcome] could not reach ${ROUTER}: ${err.message}`);
}
},
};
};

129
design/local-llm-model-router.md
View File

@@ -2,7 +2,13 @@
**Status:** design draft
**Owner:** Aaron Lee
**Hardware:** RTX 6000 24GB (local classifier/context-assembly), routes to OpenRouter + Neuralwatt Cloud
**Hardware:** RTX 6000 24GB (local classifier/context-assembly), routes to Neuralwatt Cloud
> **Revised 2026-08-11 after first live run.** Neuralwatt is now the only provider (OpenRouter
> dropped). Two findings from real API traffic change this document materially: billing is
> **per-kWh, not per-token** (§4.1), and one base model appears as several catalog rows that
> differ only by serving class (§3.4). Sections below are annotated where the original design
> assumed otherwise.
## 1. Goal
@@ -32,9 +38,9 @@ the marginal cost of a routing decision is near zero.
└──────────┬───────────┘
▼
┌─────────────────────┐
│ Dispatcher │──▶ OpenRouter
│ (OpenAI-compatible) │──▶ Neuralwatt Cloud
└──────────┬───────────┘──▶ (future providers)
│ Dispatcher │──▶ Neuralwatt Cloud
│ (OpenAI-compatible) │──▶ (future providers)
└──────────┬───────────┘
▼
┌─────────────────────┐
│ Verify / Escalate │ (local model spot-checks output)
@@ -54,7 +60,7 @@ per model+category), joined at query time.
| column | type | notes |
|---|---|---|
| `model_id` | text | canonical model name |
| `provider` | text | `openrouter`, `neuralwatt`, future providers |
| `provider` | text | `neuralwatt` today; column kept for future providers |
| `cost_per_1m_prompt` | real | listed token price |
| `cost_per_1m_completion` | real | listed token price |
| `cost_per_1m_prompt_cached` | real | e.g. Neuralwatt bills cached prefix at 25% |
@@ -130,13 +136,36 @@ primary category (and optionally a secondary one for hybrid tasks, e.g. "refacto
category — not an average across categories — so a model that's great at `docs_writing` but
mediocre at `coding_refactor` doesn't get miscredited on refactor tasks.
### 3.4 Serving class (added after first live run)
Neuralwatt ships ~6 base models as 19 catalog rows. The id suffixes are three **orthogonal**
dimensions — hence ids like `glm-5.2-short-fast-flex` — and they are *not* visible anywhere
except the id string and the prose description:
| suffix | column | meaning |
|---|---|---|
| `-flex` | `latency_class` | discounted async tier; requests held server-side during peak until a capacity gap opens |
| `-fast` | `reasoning_mode` | thinking disabled or capped to a short budget. **Not** queue priority |
| `-short` | `context_variant` | 200K pool with a bounded reasoning budget |
These rows carry **identical catalog pricing**, so they tie exactly on every scored dimension.
Left to the weighted score, a coin flip decides whether an interactive request lands on a
preemptible flex row. So latency tolerance is a **third hard filter**, not a weight:
`interactive` (default) excludes `-flex`; `batch` admits it.
Access gating is likewise prose-only — 6 of 19 rows say "Private preview (grant-gated)" or
"(Canary)" in free text with no structured field. Those are parsed into `access_level` and
excluded by default, since routing to one earns a 403 at dispatch rather than a bad answer.
## 4. Weighted scoring
Router computes a composite score per candidate model and picks the max, after two hard
Router computes a composite score per candidate model and picks the max, after three hard
filters are applied (not weighted — these disqualify a candidate outright):
1. `effective_context_window >= required_context` (§3.2)
2. `tier >= required_tier` (from classifier)
3. serving class compatible with the request's latency tolerance, and `access_level`
actually reachable by this account (§3.4)
```
score = (w_cost × cost_score)
@@ -145,12 +174,10 @@ score = (w_cost × cost_score)
```
- **cost_score**: normalized inverse cost (cheapest candidate = 1.0, scaled down from there).
For Neuralwatt, compute both token-price and energy-price cost and take whichever the account
is actually billed under.
- **eco_score**: normalized inverse `carbon_g_co2eq_per_req` where available (Neuralwatt only
today); models without a carbon figure get a neutral/default score rather than being
penalized, so the OpenRouter side of the table isn't unfairly downweighted until more
providers expose this.
⚠️ **See §4.1 — the input this currently reads is not the price being billed.**
- **eco_score**: normalized inverse `carbon_g_co2eq_per_req`, which Neuralwatt reports
per-completion; models without a carbon figure get a neutral/default score rather than being
penalized, so a model new to the table isn't downweighted just for lacking history.
- **proficiency_score**: from the benchmark refresh — general leaderboard score blended with
your own eval set (see §5), weighted toward the latter over time as it accumulates.
@@ -162,13 +189,70 @@ cost don't matter much, crank `w_prof` up for anything touching production code.
let a great cost/eco score smuggle a task into a model that's actually too weak for it) — the
weights decide *within* the eligible set, not whether a bad-fit model can win purely on price.
### 4.1 Billing is per-kWh — the cost axis needs rethinking
The line above ("take whichever the account is actually billed under") turned out to be the
whole ballgame. Measured across five models on 2026-08-11, `cost_usd / energy_kwh` came back
**8.00 every time**: Neuralwatt bills a flat **$8.00/kWh** and the catalog's
`input_per_million` / `output_per_million` figures are not what this account is charged.
The ranking consequence is not subtle. On the same prompt:
| model | list $/1M out | billed USD | kWh |
|---|---|---|---|
| `kimi-k2.7-code-fast` | 4.00 | 2.25e-03 | 2.81e-04 |
| `kimi-k3-fast` | 15.00 | 2.17e-04 | 2.71e-05 |
The model listing at **a quarter the price cost 10x more to run.** Any `cost_score` computed
from list price ranks these backwards.
This collapses two of the three scoring axes into one. `cost_usd = 8.00 x energy_kwh` exactly,
so `w_cost` and `w_eco` are weighting the same underlying quantity — 0.6 of the composite on
one signal, with proficiency left holding 0.4. Cost and eco separate *only* through grid carbon
intensity (`grid_carbon_intensity_gco2perkwhr`, 37.0 on grid `FI` at time of writing), which is
a real distinction only when routing across regions.
Two further wrinkles before this can be fixed:
- **Energy is noisy.** Identical token counts (18 -> 900) billed anywhere from 1.8e-05 to
8.8e-05 — a 4x spread run to run. A single observation is not a per-model estimate.
- **Cold start.** `energy_observations` is empty until real traffic accrues, so energy-based
scoring needs a seeding sweep (fixed prompt, N runs per routable model) before it can drive
anything.
### 4.2 Resolved: measured cost, and the two axes stay separate
Both sub-scores now read **measured** values from a fixed reference workload
(`seed_energy.py`, 13 models x 5 samples, tagged `seed_reference`): mean USD actually billed
for cost, mean gCO2eq for eco. No billing formula lives in the code — the rule in §4.1 is
documentation for *why* list price is not used, not logic.
The tempting simplification — cost = 8 x energy, therefore cost and eco are one axis — is
**wrong**, and the sweep is what caught it. Cost tracks energy, but carbon is energy x the
serving region's grid intensity, and the catalog spans 37 gCO2/kWh (`FI`) to 505
(`US-MIDA-PJM`). So the axes disagree: `glm-5.2-fast` is the 2nd cheapest model and the 6th
cleanest; `kimi-k3-flex` draws 3.7x less energy than `kimi-k2.7-code` while emitting 3.6x more
carbon. Collapsing them would have silently picked a side.
Only reference-workload observations feed scoring. Organic traffic stays in the table for cost
accounting but is excluded from routing, because per-request energy varies far more with the
*shape* of a request than with the model — across organic traffic on a single model it spanned
19x purely from differing prompt and completion sizes. Averaging that in would rank models by
what they happened to be asked.
Effect on the hard-filter-plus-score pipeline: with list-price scoring, 27 routing decisions
(9 categories x 3 tiers) produced 2 distinct models. With measured scoring, 3 — and tier 1
flipped from `deepseek-v4-flash` to `gemma-4-31b`, correctly. Category still has zero influence,
because `proficiency` remains empty and is the only category-dependent term (§3.3).
## 5. Freshness / scheduled jobs
Two independent pollers, different cadences:
- **Pricing poller** (hourly–daily): hits `OpenRouter /models` and `Neuralwatt /v1/models`
(both unauthenticated for public listing), writes cost/context/energy columns, flags rows
unchanged > N days as `stale`.
- **Pricing poller** (hourly–daily): hits `Neuralwatt /v1/models` (unauthenticated for public
listing), writes cost/context/serving-class/access columns, flags rows unchanged > N days as
`stale`. Note the catalog carries **no energy fields at all** — energy arrives only
per-completion, so the dispatcher populates `energy_observations`, not this poller.
- **Benchmark poller** (weekly, or on-demand): blends general leaderboard data (LMSYS Arena,
LiveBench, Aider polyglot) with a small self-run eval set built from your actual task
patterns (docstring quality, refactor correctness, doc-formatting) — the self-run set is more
@@ -211,7 +295,14 @@ reported together.
## 9. Next build steps
1. Poller: OpenRouter + Neuralwatt → SQLite decision table (start here, it's the load-bearing piece)
2. Minimal FastAPI dispatcher: reads table, applies weighted score, calls provider, returns
3. Local classifier prompt/schema: `{task_tier, required_context, confidence, escalation_flag}`
4. Wire into one real workflow (coding agent subagent spin-off) end to end before generalizing
1. ~~Poller: Neuralwatt → SQLite decision table~~ **done**, verified against the live API
2. ~~Minimal FastAPI dispatcher: reads table, applies weighted score, calls provider, returns~~
**done** — `/health`, `/route` (dry run), `/dispatch`
3. ~~Local classifier prompt/schema~~ **done** — the allowed category list is injected from
config at call time, since the model otherwise invents labels that join against nothing
4. ~~Re-base the cost axis~~ **done** (§4.2) — cost and eco now read measured billed USD and
gCO2eq rather than list price
5. ~~Seed `energy_observations` with a fixed-prompt sweep~~ **done** — `seed_energy.py`
6. **Populate `proficiency`** — the last inert axis. 0.4 of the composite weight does nothing,
and `task_category` cannot change a routing decision until it does
7. Wire into one real workflow (coding agent subagent spin-off) end to end before generalizing

1522
dispatcher.py Normal file
View File

File diff suppressed because it is too large Load Diff

527
eval_proficiency.py Normal file
View File

@@ -0,0 +1,527 @@
#!/usr/bin/env python3
"""Run the self-eval task set against models and write ``proficiency``.
Why this exists: ``proficiency_score`` is the ONLY category-dependent term in
the composite, so until this table has data, ``task_category`` cannot change a
routing decision at all — the classifier's category output is computed, paid
for, and then discarded. This is what makes it matter.
Scoring is objective wherever the category admits it (see evals/tasks.yaml):
code is executed against checks, exact answers are compared, tool calls are
inspected structurally. Only the four prose categories fall back to a judge.
Scores accumulate rather than replace (``proficiency_store.add_self_eval``),
so running this repeatedly tightens estimates and pushes categories past
``self_eval_min_samples`` into a proper blend with the leaderboard prior.
python eval_proficiency.py --dry-run # plan only, no calls
python eval_proficiency.py # every identity, every task
python eval_proficiency.py --models kimi-k3 --categories coding_general
SAFETY: `code` tasks execute model-generated Python. Isolation is a
subprocess with a wall-clock timeout, running in a temp directory — not a
container and not a real sandbox. The task prompts ask for small pure
functions, so nothing invites filesystem or network use, but treat this as
"bounded", not "safe against a hostile model".
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sqlite3
import subprocess
import sys
import tempfile
from collections import defaultdict
from pathlib import Path
from typing import Optional
import requests
import yaml
from config import RouterConfig, load_config
from proficiency_store import add_self_eval, propagate_to_variants
TASKS_PATH = "evals/tasks.yaml"
CODE_TIMEOUT_SECONDS = 15
CALL_TIMEOUT_SECONDS = 300
# Generous on purpose. These are reasoning models and the trace is billed
# against the same budget as the answer: at 1200, qwen3.6-35b spent ~4,200
# characters thinking and returned an EMPTY content field, scoring 0.00 on
# tasks it can plainly do. The cap must clear the trace, not just the answer,
# and the harder task set makes traces longer still.
#
# Clamped per model at request time: this exceeds gemma-4-31b's advertised
# 16384 output limit, and asking for more than a model allows is an error
# rather than a silent truncation.
EVAL_MAX_TOKENS = 24000
# Preamble for the sandbox: common imports so ordinary solutions run, plus the
# `raises` helper the checks use. Defined AFTER the model's code so a model
# that shadows one of these does not break the harness.
HARNESS_PREAMBLE = "import collections, itertools, json, math, re, string, time\n"
HARNESS_HELPERS = '''
def raises(exc, fn, *a, **kw):
try:
fn(*a, **kw)
except exc:
return True
except Exception:
return False
return False
'''
FENCE_RE = re.compile(r"^\s*```[a-zA-Z]*\n(.*?)```", re.DOTALL | re.MULTILINE)
# --- extraction and scoring ----------------------------------------------
def strip_fences(text: str) -> str:
"""Pull code out of a markdown block if the model added one anyway.
Every code prompt says "no markdown fences" and models add them regularly,
so this is normalization rather than leniency — a model that solved the
task shouldn't score zero for formatting.
"""
match = FENCE_RE.search(text or "")
code = match.group(1) if match else (text or "")
# Strip the WHOLE reply's leading/trailing whitespace: a single leading
# space turns an otherwise perfect function into IndentationError, which
# is how kimi-k2.7-code scored 0.00 on every coding task in an early run.
# Internal indentation is untouched.
return code.strip()
def score_code(model_output: str, checks: list[str]) -> tuple[float, str]:
"""Execute the model's code and eval each check against it.
Returns (fraction of checks passing, detail). Partial credit is
deliberate: a function correct on three of four cases is genuinely better
than one that fails everything, and a binary score throws that away.
"""
code = strip_fences(model_output)
harness = (
HARNESS_PREAMBLE
+ code
+ "\n"
+ HARNESS_HELPERS
+ f"\nCHECKS = {checks!r}\n"
+ "for _i, _c in enumerate(CHECKS):\n"
+ " try:\n"
+ " _ok = bool(eval(_c))\n"
+ " except Exception:\n"
+ " _ok = False\n"
+ " print('CHECK', _i, 'PASS' if _ok else 'FAIL')\n"
)
with tempfile.TemporaryDirectory() as tmp:
script = Path(tmp) / "harness.py"
script.write_text(harness)
try:
proc = subprocess.run(
[sys.executable, str(script)],
capture_output=True,
text=True,
timeout=CODE_TIMEOUT_SECONDS,
cwd=tmp,
)
except subprocess.TimeoutExpired:
return 0.0, "timeout"
passed = proc.stdout.count("PASS")
if not checks:
return 0.0, "no checks"
if passed == 0 and proc.returncode != 0:
# Distinguish "wrote broken code" from "wrote code that fails cases"
first_line = (proc.stderr or "").strip().splitlines()[-1:] or [""]
return 0.0, f"did not run: {first_line[0][:80]}"
return passed / len(checks), f"{passed}/{len(checks)} checks"
def normalize_answer(text: str) -> str:
"""Reduce a free-text reply to a comparable token.
Models wrap a number in prose, punctuation, or thousands separators even
when told not to; none of that is the thing being measured.
"""
cleaned = (text or "").strip().replace(",", "")
numbers = re.findall(r"-?\d+(?:\.\d+)?", cleaned)
if numbers:
value = numbers[-1] # the conclusion, if it reasoned out loud
return value.rstrip("0").rstrip(".") if "." in value else value
return cleaned.lower().strip(" .!\"'")
def score_exact(model_output: str, expected: str) -> tuple[float, str]:
got = normalize_answer(model_output)
want = normalize_answer(expected)
return (1.0, f"{got!r}") if got == want else (0.0, f"got {got!r} want {want!r}")
def score_tool(tool_calls: list, task: dict) -> tuple[float, str]:
"""Score a tool-use task structurally — no judge required.
Three things are worth distinguishing and each is a task in the set:
calling the right tool, choosing correctly between several, and NOT
calling a tool when none applies. The last is scored because a model that
reaches for a tool on every prompt is a real failure mode in an agent
loop.
"""
expected_tool = task.get("expect_tool")
if expected_tool is None:
return (1.0, "correctly abstained") if not tool_calls else (
0.0,
f"called {tool_calls[0]['function']['name']} when none applied",
)
if not tool_calls:
return 0.0, "no tool call"
called = tool_calls[0]["function"]["name"]
if called != expected_tool:
return 0.0, f"called {called}, wanted {expected_tool}"
# Right tool. Give partial credit and check the arguments that matter.
expected_args = task.get("expect_args") or {}
if not expected_args:
return 1.0, "correct tool"
try:
got_args = json.loads(tool_calls[0]["function"].get("arguments") or "{}")
except json.JSONDecodeError:
return 0.5, "correct tool, unparseable arguments"
matched = 0
for key, want in expected_args.items():
got = got_args.get(key)
if isinstance(want, str) and isinstance(got, str):
ok = want.lower() in got.lower()
else:
ok = str(got) == str(want)
matched += bool(ok)
# Floor at 0.5 for picking the right tool; arguments carry the rest.
return 0.5 + 0.5 * (matched / len(expected_args)), f"correct tool, {matched}/{len(expected_args)} args"
# --- provider calls -------------------------------------------------------
def call_model(
base_url: str,
api_key: str,
model_id: str,
task: dict,
max_output_tokens: Optional[int] = None,
) -> tuple[str, list, bool]:
"""One completion. Returns (text, tool_calls, truncated)."""
budget = EVAL_MAX_TOKENS
if max_output_tokens:
budget = min(budget, max_output_tokens)
body = {
"model": model_id,
"messages": [{"role": "user", "content": task["prompt"]}],
"temperature": 0,
"max_tokens": budget,
}
if task.get("tools"):
body["tools"] = task["tools"]
resp = requests.post(
f"{base_url}/chat/completions",
headers={"authorization": f"Bearer {api_key}"},
json=body,
timeout=CALL_TIMEOUT_SECONDS,
)
resp.raise_for_status()
choice = (resp.json().get("choices") or [{}])[0]
message = choice.get("message") or {}
# finish_reason 'length' means the budget ran out mid-answer. Whatever
# came back is an artifact of the cap, so the caller skips rather than
# scoring it — the same rule as an unusable judge reply.
truncated = choice.get("finish_reason") == "length"
return message.get("content") or "", message.get("tool_calls") or [], truncated
JUDGE_SYSTEM = (
"You are scoring one model's answer against a rubric. Reply with ONLY a "
'JSON object: {"score": <number 0 to 1>, "reason": "<at most 12 words>"}. '
"Do not explain your reasoning outside the JSON. "
"Be strict: award a high score only if every rubric requirement is met."
)
JSON_OBJECT_RE = re.compile(r"\{.*\}", re.DOTALL)
def extract_judge_json(raw: str) -> Optional[dict]:
"""Pull the score object out of a judge reply.
Judges are themselves reasoning models and leak their thinking into the
content despite response_format, so the object usually arrives wrapped in
prose ("Let me evaluate the answer against the rubric: ... {...}"). Before
this existed, 44% of judge calls were unparseable.
"""
if not raw:
return None
candidates = [raw]
match = JSON_OBJECT_RE.search(raw)
if match:
candidates.append(match.group(0))
for candidate in candidates:
try:
parsed = json.loads(candidate)
except json.JSONDecodeError:
continue
if isinstance(parsed, dict) and "score" in parsed:
return parsed
return None
def score_judge(
base_url: str, api_key: str, judge_model: str, task: dict, answer: str
) -> Optional[tuple[float, str]]:
"""Score a prose task against its rubric.
Returns None when the judge itself failed, so the caller can SKIP the
task rather than record it. Scoring a judge malfunction as 0.0 would
charge the model for the judge's formatting — it put genuine 0.0 entries
in translation and general_chat on the first run, for answers that were
in fact fine.
"""
body = {
"model": judge_model,
"messages": [
{"role": "system", "content": JUDGE_SYSTEM},
{
"role": "user",
"content": (
f"TASK GIVEN TO THE MODEL:\n{task['prompt']}\n\n"
f"RUBRIC:\n{task['rubric']}\n\n"
f"THE MODEL'S ANSWER:\n{answer}"
),
},
],
"temperature": 0,
# Generous, because a reasoning judge spends tokens thinking before
# the JSON and a truncated object is unparseable.
"max_tokens": 1500,
"response_format": {"type": "json_object"},
}
resp = requests.post(
f"{base_url}/chat/completions",
headers={"authorization": f"Bearer {api_key}"},
json=body,
timeout=CALL_TIMEOUT_SECONDS,
)
resp.raise_for_status()
raw = ((resp.json().get("choices") or [{}])[0].get("message") or {}).get("content") or ""
parsed = extract_judge_json(raw)
if parsed is None:
return None
try:
score = float(parsed["score"])
except (KeyError, TypeError, ValueError):
return None
return max(0.0, min(1.0, score)), str(parsed.get("reason", ""))[:70]
# --- selection ------------------------------------------------------------
def eval_identities(conn: sqlite3.Connection, cfg: RouterConfig) -> list[dict]:
"""Model rows worth measuring directly.
Flex rows are normally excluded: same weights, same reasoning setting,
different queue — their quality is the standard row's, inherited via
``propagate_to_variants``. ``-fast`` rows ARE measured, because reasoning
being off genuinely changes answers.
But that exclusion assumes an evaluated standard equivalent exists, and
it does not always: glm-5.2-flex reasons by default, while the only
routable standard glm row is glm-5.2-fast (reasoning reduced) — the
matching glm-5.2 is canary and therefore never evaluated. Such orphans
are measured directly rather than left with no proficiency at all.
"""
placeholders = ",".join("?" * len(cfg.routing.allowed_access_levels))
rows = conn.execute(
f"""
SELECT model_id, base_model_id, reasoning_mode, max_output_tokens FROM models m
WHERE availability = 'active'
AND access_level IN ({placeholders})
AND (
latency_class = 'standard'
-- ...or a flex row with no routable standard equivalent to
-- inherit from, which would otherwise score nothing at all
OR NOT EXISTS (
SELECT 1 FROM models s
WHERE s.base_model_id = m.base_model_id
AND s.provider = m.provider
AND s.reasoning_mode = m.reasoning_mode
AND s.context_variant = m.context_variant
AND s.latency_class = 'standard'
AND s.availability = 'active'
AND s.access_level IN ({placeholders})
)
)
ORDER BY model_id
""",
tuple(cfg.routing.allowed_access_levels) * 2,
).fetchall()
return [
{
"model_id": r[0],
"base_model_id": r[1],
"reasoning_mode": r[2],
"max_output_tokens": r[3],
}
for r in rows
]
ALTERNATE_JUDGE = "qwen3.6-35b"
def judge_for(model_id: str, default_judge: str) -> str:
"""Pick a judge that is not the model being judged.
A model scoring its own prose is a known bias, and the default judge is
itself in the evaluated set. Swapping in an alternate for its own family
costs nothing and removes the obvious conflict.
"""
from poller import parse_base_model_id
if parse_base_model_id(model_id) == parse_base_model_id(default_judge):
return ALTERNATE_JUDGE
return default_judge
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--tasks", default=TASKS_PATH)
ap.add_argument("--models", help="comma-separated model_ids")
ap.add_argument("--categories", help="comma-separated categories")
ap.add_argument("--judge-model", default="kimi-k3")
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args()
cfg = load_config("config.yaml")
tasks = (yaml.safe_load(Path(args.tasks).read_text()) or {}).get("tasks") or []
if args.categories:
wanted = {c.strip() for c in args.categories.split(",")}
tasks = [t for t in tasks if t["category"] in wanted]
unknown = {t["category"] for t in tasks} - set(cfg.proficiency.categories)
if unknown:
print(f"tasks reference unknown categories: {sorted(unknown)}", file=sys.stderr)
return 1
conn = sqlite3.connect(cfg.database.path)
identities = eval_identities(conn, cfg)
if args.models:
wanted = {m.strip() for m in args.models.split(",")}
identities = [i for i in identities if i["model_id"] in wanted]
if not identities or not tasks:
print("nothing to run", file=sys.stderr)
return 1
judged = sum(1 for t in tasks if t["kind"] == "judge")
print(
f"{len(identities)} models x {len(tasks)} tasks = "
f"{len(identities) * len(tasks)} calls"
+ (f" (+{len(identities) * judged} judge calls)" if judged else "")
)
if args.dry_run:
for i in identities:
budget = min(EVAL_MAX_TOKENS, i["max_output_tokens"] or EVAL_MAX_TOKENS)
print(
f" {i['model_id']:24s} reasoning={i['reasoning_mode']:8s} "
f"budget={budget}"
)
print(f" judge: {args.judge_model}")
return 0
settings = cfg.dispatch_providers["neuralwatt"]
api_key = os.environ.get(settings.api_key_env)
if not api_key:
print(f"{settings.api_key_env} is not set", file=sys.stderr)
return 1
for identity in identities:
model_id = identity["model_id"]
by_category: dict[str, list[float]] = defaultdict(list)
print(f"\n{model_id}")
for task in tasks:
try:
text, tool_calls, truncated = call_model(
settings.base_url,
api_key,
model_id,
task,
identity["max_output_tokens"],
)
except requests.RequestException as e:
print(f" {task['id']:30s} CALL FAILED {type(e).__name__}")
continue
if truncated and not tool_calls:
print(
f" {task['id']:30s} --- truncated at "
f"{EVAL_MAX_TOKENS} tokens, skipped"
)
continue
kind = task["kind"]
try:
if kind == "code":
score, detail = score_code(text, task["checks"])
elif kind == "exact":
score, detail = score_exact(text, str(task["answer"]))
elif kind == "tool":
score, detail = score_tool(tool_calls, task)
elif kind == "judge":
judged = score_judge(
settings.base_url,
api_key,
judge_for(model_id, args.judge_model),
task,
text,
)
if judged is None:
# No sample rather than a zero: a judge that failed to
# emit parseable JSON says nothing about the model.
print(f" {task['id']:30s} --- judge unusable, skipped")
continue
score, detail = judged
else:
print(f" {task['id']:30s} unknown kind {kind!r}")
continue
except requests.RequestException as e:
print(f" {task['id']:30s} SCORING FAILED {type(e).__name__}")
continue
by_category[task["category"]].append(score)
print(f" {task['id']:30s} {score:4.2f} {detail}")
for category, scores in by_category.items():
add_self_eval(conn, cfg, model_id, "neuralwatt", category, scores)
conn.commit()
# Equivalent serving variants inherit from the row actually measured —
# not from the family id, which may name a row that was never evaluated
# (glm-5.2 is canary, so glm-5.2-flex inherited nothing and scored blank).
propagated = 0
for identity in identities:
propagated += propagate_to_variants(
conn, cfg, identity["model_id"], "neuralwatt"
)
conn.commit()
print(f"\npropagated {propagated} inherited rows to serving variants")
conn.close()
return 0
if __name__ == "__main__":
raise SystemExit(main())

490
evals/tasks.yaml Normal file
View File

@@ -0,0 +1,490 @@
# Self-eval task set. One score per task per model; scores accumulate into
# proficiency.self_eval_score and self_eval_samples.
#
# Four kinds, chosen so each category is scored the most objective way it
# admits:
#
# code model writes Python; each check is eval'd against it in a
# subprocess. Score = fraction of checks passing. Fully objective.
# exact model replies with one value; compared after normalization.
# tool model is given a tool schema; scored on whether it calls the right
# tool with the right arguments. Structural, no judge needed.
# judge a strong model scores the output against a rubric. Only for the
# prose categories, where nothing checkable exists.
#
# DIFFICULTY: the happy path is not worth testing. Every current model passes
# "reverse a list", and a category where everyone scores 1.00 discriminates no
# better than the constant 0.5 it replaced. Each task here carries at least one
# edge case a plausible-looking solution gets wrong: touching vs overlapping
# intervals, present-but-falsy values, late-binding closures, greedy regexes,
# empty input, or a trap in the arithmetic.
#
# Every `code` task is validated against a reference solution by
# tests/test_task_set.py. A check my own reference cannot pass is a broken
# check, and would score the task set rather than the model — which has
# already happened here once.
#
# Keep prompts tight and self-contained. An ambiguous task scores the prompt.
tasks:
# --- coding_general -----------------------------------------------------
- id: merge_intervals
category: coding_general
kind: code
entrypoint: merge_intervals
prompt: |
Write a Python function `merge_intervals(intervals)` where intervals is
a list of [start, end] lists. Merge all overlapping intervals and return
a new list of [start, end] lists sorted by start. Intervals that merely
touch (one ends exactly where the next begins) must be merged. Input may
be unsorted and may contain intervals fully nested inside others.
Reply with ONLY the function definition — no explanation, no fences.
checks:
- 'merge_intervals([[1,3],[2,6],[8,10]]) == [[1,6],[8,10]]'
- 'merge_intervals([]) == []'
- 'merge_intervals([[1,4],[4,5]]) == [[1,5]]'
- 'merge_intervals([[1,10],[2,3]]) == [[1,10]]'
- 'merge_intervals([[5,6],[1,2]]) == [[1,2],[5,6]]'
- 'merge_intervals([[1,2]]) == [[1,2]]'
- id: parse_semver
category: coding_general
kind: code
entrypoint: parse_semver
prompt: |
Write a Python function `parse_semver(version)` that parses a semantic
version string into a dict with keys: major, minor, patch (ints), and
prerelease, build (strings, or None when absent). Valid examples:
"1.2.3", "1.2.3-alpha.1", "1.2.3+build.5", "1.2.3-rc.1+exp.sha.5114f85".
Raise ValueError if the string is not a valid semantic version, for
example "1.2" or "1.2.x".
Reply with ONLY the function definition — no explanation, no fences.
checks:
- 'parse_semver("1.2.3") == {"major":1,"minor":2,"patch":3,"prerelease":None,"build":None}'
- 'parse_semver("1.2.3-alpha.1")["prerelease"] == "alpha.1"'
- 'parse_semver("1.2.3+build.5")["build"] == "build.5"'
- 'parse_semver("1.2.3-rc.1+exp.sha.5114f85")["prerelease"] == "rc.1"'
- 'parse_semver("1.2.3-rc.1+exp.sha.5114f85")["build"] == "exp.sha.5114f85"'
- 'parse_semver("0.0.0")["major"] == 0'
- 'raises(ValueError, parse_semver, "1.2")'
- 'raises(ValueError, parse_semver, "1.2.x")'
- id: word_wrap
category: coding_general
kind: code
entrypoint: word_wrap
prompt: |
Write a Python function `word_wrap(text, width)` returning a list of
lines. Split on whitespace and pack as many words per line as fit within
`width` characters, joining words with a single space. Never split a
word: a word longer than `width` gets its own line. Runs of whitespace
collapse. Empty or whitespace-only text returns an empty list.
Reply with ONLY the function definition — no explanation, no fences.
checks:
- 'word_wrap("the quick brown fox", 10) == ["the quick", "brown fox"]'
- 'word_wrap("", 5) == []'
- 'word_wrap(" ", 5) == []'
- 'word_wrap("supercalifragilistic", 5) == ["supercalifragilistic"]'
- 'word_wrap("a b c", 3) == ["a b", "c"]'
- 'word_wrap("aa bb cc", 5) == ["aa bb", "cc"]'
# --- coding_refactor ----------------------------------------------------
- id: refactor_falsy_defaults
category: coding_refactor
kind: code
entrypoint: apply_settings
prompt: |
Refactor this function to remove the repetition. Behaviour must be
preserved EXACTLY, including for values that are present but falsy.
Reply with ONLY the rewritten function — no explanation, no fences.
def apply_settings(overrides):
result = {}
if "retries" in overrides:
result["retries"] = overrides["retries"]
else:
result["retries"] = 3
if "timeout" in overrides:
result["timeout"] = overrides["timeout"]
else:
result["timeout"] = 30
if "verbose" in overrides:
result["verbose"] = overrides["verbose"]
else:
result["verbose"] = False
return result
checks:
- 'apply_settings({}) == {"retries":3,"timeout":30,"verbose":False}'
- 'apply_settings({"retries":0})["retries"] == 0'
- 'apply_settings({"timeout":0})["timeout"] == 0'
- 'apply_settings({"verbose":True})["verbose"] is True'
- 'apply_settings({"retries":5}) == {"retries":5,"timeout":30,"verbose":False}'
- id: refactor_first_match
category: coding_refactor
kind: code
entrypoint: first_match
prompt: |
Refactor this to remove the nested loops and the flag variable.
Behaviour must be preserved exactly, including which item wins when
several match. Reply with ONLY the rewritten function — no explanation,
no fences.
def first_match(items, predicates):
found = None
done = False
for item in items:
if done:
break
for p in predicates:
if p(item):
found = item
done = True
break
return found
checks:
- 'first_match([1,2,3,4], [lambda x: x > 2]) == 3'
- 'first_match([1,2,3], [lambda x: x > 10]) is None'
- 'first_match([], [lambda x: True]) is None'
- 'first_match([1,2,3], []) is None'
- 'first_match([5,2,9], [lambda x: x > 8, lambda x: x < 3]) == 2'
- 'first_match([0,1], [lambda x: x == 0]) == 0'
- id: refactor_dispatch
category: coding_refactor
kind: code
entrypoint: describe
prompt: |
Refactor this if/elif chain into a table-driven lookup. Behaviour must be
preserved exactly for every input, including inputs that match no case.
Reply with ONLY the rewritten code — no explanation, no fences.
def describe(code):
if code == 200:
return "ok"
elif code == 201:
return "created"
elif code == 404:
return "not found"
elif code == 500:
return "server error"
else:
return "unknown"
checks:
- 'describe(200) == "ok"'
- 'describe(404) == "not found"'
- 'describe(500) == "server error"'
- 'describe(418) == "unknown"'
- 'describe(0) == "unknown"'
- 'describe(None) == "unknown"'
# --- debugging ----------------------------------------------------------
- id: debug_late_binding
category: debugging
kind: code
entrypoint: make_multipliers
prompt: |
This should return one multiplier function per factor, but every
returned function behaves the same. Fix it. Reply with ONLY the
corrected function — no explanation, no fences.
def make_multipliers(factors):
out = []
for f in factors:
out.append(lambda x: x * f)
return out
checks:
- '[m(2) for m in make_multipliers([1,2,3])] == [2,4,6]'
- '[m(10) for m in make_multipliers([0,1])] == [0,10]'
- 'make_multipliers([]) == []'
- 'make_multipliers([7])[0](3) == 21'
- id: debug_binary_search
category: debugging
kind: code
entrypoint: bsearch
prompt: |
This binary search should return the index of target in a sorted list,
or -1 if absent. It is wrong for some inputs — one case loops forever.
Fix it. Reply with ONLY the corrected function — no explanation, no
fences.
def bsearch(items, target):
lo, hi = 0, len(items)
while lo < hi:
mid = (lo + hi) // 2
if items[mid] == target:
return mid
elif items[mid] < target:
lo = mid
else:
hi = mid
return -1
checks:
- 'bsearch([1,3,5,7], 7) == 3'
- 'bsearch([1,3,5,7], 1) == 0'
- 'bsearch([], 1) == -1'
- 'bsearch([1], 1) == 0'
- 'bsearch([1,3], 2) == -1'
- 'bsearch([1,2,3,4,5,6], 6) == 5'
- id: debug_greedy_regex
category: debugging
kind: code
entrypoint: extract_tags
prompt: |
This should return the name inside each angle-bracket tag, in order, but
it returns the wrong thing when there is more than one tag. Fix it.
Reply with ONLY the corrected function — no explanation, no fences.
import re
def extract_tags(text):
return re.findall(r"<(.+)>", text)
checks:
- 'extract_tags("<a><b>") == ["a","b"]'
- 'extract_tags("<one>") == ["one"]'
- 'extract_tags("") == []'
- 'extract_tags("no tags here") == []'
- 'extract_tags("x <a> y <bc> z") == ["a","bc"]'
# --- reasoning_math -----------------------------------------------------
- id: math_percent_trap
category: reasoning_math
kind: exact
answer: "100"
prompt: |
A price rises by 20%, then falls by 20% of its new value. The final
price is 96. What was the original price? Reply with ONLY the number.
- id: math_rate_trap
category: reasoning_math
kind: exact
answer: "3"
prompt: |
Three machines take 3 minutes to make 3 widgets, each machine working
independently at the same constant rate. How many minutes do 100
machines take to make 100 widgets? Reply with ONLY the number.
- id: math_counting
category: reasoning_math
kind: exact
answer: "4536"
prompt: |
How many 4-digit whole numbers have four distinct digits and do not
begin with 0? Reply with ONLY the number.
# --- tool_use_agentic ---------------------------------------------------
- id: tool_multi_arg
category: tool_use_agentic
kind: tool
prompt: Convert 250 US dollars into Japanese yen.
expect_tool: convert_currency
expect_args:
amount: 250
from_currency: USD
to_currency: JPY
tools:
- type: function
function:
name: get_weather
description: Get the current weather for a location.
parameters:
type: object
properties:
location: {type: string}
required: [location]
- type: function
function:
name: convert_currency
description: Convert an amount between two currencies.
parameters:
type: object
properties:
amount: {type: number}
from_currency: {type: string, description: ISO 4217 code}
to_currency: {type: string, description: ISO 4217 code}
required: [amount, from_currency, to_currency]
- id: tool_no_tool_needed
category: tool_use_agentic
kind: tool
prompt: |
It is 1:20pm and my meeting starts at 3pm. How many minutes away is it?
expect_tool: null # plain arithmetic; both times are already given
tools:
- type: function
function:
name: get_calendar_event
description: Look up a calendar event by title.
parameters:
type: object
properties:
title: {type: string}
required: [title]
- type: function
function:
name: get_current_time
description: Get the current wall-clock time.
parameters:
type: object
properties: {}
- id: tool_abstain_creative
category: tool_use_agentic
kind: tool
prompt: Write me a haiku about winter.
expect_tool: null
tools:
- type: function
function:
name: get_weather
description: Get the current weather for a location.
parameters:
type: object
properties:
location: {type: string}
required: [location]
# --- docs_writing -------------------------------------------------------
- id: docs_function
category: docs_writing
kind: judge
prompt: |
Write a docstring for this function. Reply with ONLY the docstring text.
def retry(fn, attempts=3, backoff=2.0):
delay = 1.0
for i in range(attempts):
try:
return fn()
except Exception:
if i == attempts - 1:
raise
time.sleep(delay)
delay *= backoff
rubric: |
Score 0-1. Award 1.0 ONLY if it states what the function does, documents
every parameter including defaults, states the return value, AND states
that the last exception is re-raised when all attempts fail. Deduct 0.3
for any invented behaviour the code does not have. Deduct 0.2 if it omits
that the delay grows by `backoff` between attempts.
- id: docs_gotcha
category: docs_writing
kind: judge
prompt: |
Write a docstring for this function. Reply with ONLY the docstring text.
def dedupe(items, key=None):
seen = set()
out = []
for item in items:
k = key(item) if key else item
if k in seen:
continue
seen.add(k)
out.append(item)
return out
rubric: |
Score 0-1. Award 1.0 ONLY if it states that ORDER IS PRESERVED and that
the FIRST occurrence is kept, documents `key`, and notes that elements
(or their keys) must be hashable. Deduct 0.4 if it omits the
order-preservation guarantee — that is the whole reason to use this over
set(). Deduct 0.3 if it omits the hashability requirement.
# --- summarization ------------------------------------------------------
- id: summarize_incident
category: summarization
kind: judge
prompt: |
Summarize in at most two sentences:
At 02:14 UTC the checkout service began returning 502s. The on-call
engineer found the connection pool exhausted. A deploy at 01:58 had
lowered the pool size from 50 to 5 through a bad template variable. The
deploy was rolled back at 02:31 and errors stopped by 02:34. Roughly
12,000 requests failed. No data was lost.
rubric: |
Score 0-1. Award 1.0 ONLY if it names the ROOT CAUSE specifically (a bad
template variable in the 01:58 deploy cut the pool from 50 to 5), the
resolution (rollback), and the impact (~12k failed requests, no data
lost), in two sentences or fewer. Deduct 0.4 for saying only "the
connection pool was exhausted" — that is the symptom, not the cause.
Deduct 0.5 for any invented detail.
- id: summarize_buried_lede
category: summarization
kind: judge
prompt: |
Summarize the single most important point in one sentence:
The migration ran for six hours. Throughput averaged 4,200 rows per
second, peaking at 6,100. The team used a rolling window of 5,000 rows
per batch. Disk usage on the replica grew steadily. Partway through, a
unique constraint on the accounts table silently rejected 812 rows,
which were logged to a dead-letter file that nobody has yet processed.
CPU stayed under 40% throughout.
rubric: |
Score 0-1. The important point is that 812 rows were silently dropped and
remain unprocessed — everything else is routine telemetry. Award 1.0 only
if the summary leads with that. Score 0.3 or below if it summarizes the
throughput statistics instead. Must be one sentence.
# --- translation --------------------------------------------------------
- id: translate_technical
category: translation
kind: judge
prompt: |
Translate into French. Reply with ONLY the translation.
"The connection pool was exhausted because a recent deploy reduced its
size. Roll back the deploy and the errors should stop within a few
minutes."
rubric: |
Score 0-1 on accuracy and fluency. Award 1.0 only for correct technical
register on "connection pool", "deploy" and "roll back", AND natural
French rather than a word-for-word calque. Deduct 0.3 per omission or
untranslated fragment.
- id: translate_register
category: translation
kind: judge
prompt: |
Translate into Spanish, preserving the hedging and the informal tone.
Reply with ONLY the translation.
"I'm not totally sure this is the right call, but I'd lean towards
shipping it and seeing what breaks — we can always roll it back."
rubric: |
Score 0-1. Award 1.0 only if the HEDGING is preserved ("not totally
sure", "I'd lean towards") rather than flattened into a confident
statement, the register stays informal, and "roll it back" is rendered
idiomatically. Deduct 0.4 if the hedging is lost.
# --- general_chat -------------------------------------------------------
- id: chat_explain
category: general_chat
kind: judge
prompt: |
Explain to a non-programmer, in under 100 words, why a program can be
correct and still be too slow to use.
rubric: |
Score 0-1. Award 1.0 only if it distinguishes correctness from
performance, gives at least one concrete relatable example, stays under
100 words, and leaves no jargon unexplained. Deduct 0.3 if over 100
words.
- id: chat_pushback
category: general_chat
kind: judge
prompt: |
A colleague says "we should rewrite the whole service in Rust, it'll be
faster." Reply in under 80 words, taking the suggestion seriously but
identifying what you would want to know first.
rubric: |
Score 0-1. Award 1.0 only if it avoids both pure agreement and pure
dismissal, names at least two specific things worth establishing first
(for example where time is actually spent, migration cost, team
familiarity), and stays under 80 words. Score 0.3 or below for a reply
that simply agrees or simply refuses.

172
feedback.py Normal file
View File

@@ -0,0 +1,172 @@
#!/usr/bin/env python3
"""Fold observed verification failures back into proficiency.
The eval harness measures models on a fixed 23-task benchmark. This measures
them on YOUR traffic, which is more predictive of routing quality and
accumulates for free as you work.
python feedback.py --dry-run # show what would change
python feedback.py # apply
Two sources, treated differently, because they are not the same kind of
evidence.
**Checks (`structural`, `local_llm`) contribute failures only.** A pass there
is weak: structural 'ok' means the code parsed, not that it was correct, and a
model emitting syntactically valid nonsense would score 1.0. Recording those
passes would flood `self_eval_score` with 1.0 samples and wash out the
benchmark's discrimination — the coding categories already sit at 1.00 for
every model, and this would spread that flatness everywhere. A failure is the
opposite: truncated or malformed output is definitive.
**Client outcomes contribute BOTH ways.** A client reporting success is not
the same claim as a parser reporting success — it ran the tests, or used the
answer, and knows the work worked. That is the only ground truth available
here, so it counts in both directions. It is also the only signal that
survives streaming, where a retry cannot reach.
Each failure is applied once. `applied_at` marks consumed rows so re-running
cannot penalize a model repeatedly for the same bad response.
"""
from __future__ import annotations
import argparse
import sqlite3
import sys
from collections import defaultdict
from config import load_config
from proficiency_store import add_self_eval
# Verdicts that count as observed failures. 'unverifiable' is excluded: it
# means the checker had nothing to say, which is not evidence about the model.
FAILURE_VERDICTS = ("truncated", "malformed", "failed")
# The one verdict that counts as a positive sample. A parser's 'ok' does not
# qualify — it means the code parsed. This means the client ran it.
SUCCESS_VERDICTS = ("succeeded",)
# Failures the model did not cause are excluded. The obvious case: a client
# that sets max_tokens=40 and gets a truncated answer caused that itself, and
# counting it would let any agent with a tight cap drag down whatever model it
# happened to route to. Found by forcing exactly that during testing.
def unapplied_failures(conn: sqlite3.Connection) -> list[sqlite3.Row]:
"""Rows that should move a model's score, with the sample each contributes.
Named for what it mostly is. Successes only enter via client outcomes; a
check passing is not evidence the answer was right.
"""
conn.row_factory = sqlite3.Row
scored = FAILURE_VERDICTS + SUCCESS_VERDICTS
placeholders = ",".join("?" * len(scored))
return conn.execute(
f"""
SELECT id, model_id, provider, task_category, kind, verdict, detail
FROM verifications
WHERE verdict IN ({placeholders})
AND applied_at IS NULL
AND task_category IS NOT NULL
AND model_attributable = 1
ORDER BY id
""",
scored,
).fetchall()
def summarize(
rows: list[sqlite3.Row],
) -> dict[tuple[str, str, str], list[tuple[int, float]]]:
"""Group by (model, provider, category) as (row id, sample score) pairs."""
grouped: dict[tuple[str, str, str], list[tuple[int, float]]] = defaultdict(list)
for r in rows:
score = 1.0 if r["verdict"] in SUCCESS_VERDICTS else 0.0
grouped[(r["model_id"], r["provider"], r["task_category"])].append(
(r["id"], score)
)
return grouped
def apply_failures(conn: sqlite3.Connection, cfg, grouped, dry_run: bool) -> int:
applied = 0
for (model_id, provider, category), pairs in sorted(grouped.items()):
ids = [i for i, _ in pairs]
scores = [sc for _, sc in pairs]
wins = sum(1 for sc in scores if sc == 1.0)
print(
f" {model_id:24s} {category:18s} "
f"{len(scores) - wins} failure(s), {wins} success(es)"
)
if dry_run:
continue
# Folded into the running mean by proficiency.accumulate, so the effect
# scales with the observed rate rather than replacing the benchmark.
add_self_eval(conn, cfg, model_id, provider, category, scores)
conn.executemany(
"UPDATE verifications SET applied_at = datetime('now') WHERE id = ?",
[(i,) for i in ids],
)
applied += len(ids)
if not dry_run:
conn.commit()
return applied
def coverage(conn: sqlite3.Connection) -> None:
"""Report what verification has seen, so its usefulness stays visible."""
conn.row_factory = sqlite3.Row
rows = conn.execute(
"""
SELECT kind, verdict, COUNT(*) n FROM verifications
GROUP BY kind, verdict ORDER BY kind, verdict
"""
).fetchall()
if not rows:
print(" no verifications recorded yet")
return
total = sum(r["n"] for r in rows)
print(f" {'kind':16s}{'verdict':16s}{'n':>6}{'share':>9}")
for r in rows:
print(f" {r['kind']:16s}{r['verdict']:16s}{r['n']:>6}{r['n']/total*100:>8.1f}%")
unver = sum(r["n"] for r in rows if r["verdict"] == "unverifiable")
if total and unver / total > 0.8:
print(
f"\n NOTE: {unver/total*100:.0f}% of responses were unverifiable. "
"Structural checking is not earning much here —\n"
" most traffic is prose. The local LLM check covers that, but only "
"above the size threshold."
)
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--dry-run", action="store_true")
args = ap.parse_args()
cfg = load_config("config.yaml")
conn = sqlite3.connect(cfg.database.path)
print("verification coverage so far:")
coverage(conn)
print()
rows = unapplied_failures(conn)
if not rows:
print("no unapplied signals — nothing to fold in")
conn.close()
return 0
grouped = summarize(rows)
print(f"{'would apply' if args.dry_run else 'applying'} "
f"{len(rows)} sample(s) across {len(grouped)} (model, category) pair(s):")
applied = apply_failures(conn, cfg, grouped, args.dry_run)
conn.close()
if not args.dry_run:
print(f"\napplied {applied} sample(s)")
return 0
if __name__ == "__main__":
raise SystemExit(main())

142
iteration.py Normal file
View File

@@ -0,0 +1,142 @@
"""Pure logic for spending a tier's iteration budget.
Reframes what a tier means. It used to be only a capability floor —
"don't route below this". It is now also a budget for getting the answer
right: tier 1 buys one attempt, higher tiers buy corrective attempts after a
verification failure.
Why this is better than what it replaces. `apply_escalation` bumps the tier
*preemptively* when the classifier is unsure of its own call, so an uncertain
guess costs frontier prices before anything has gone wrong. Spending after a
check has actually failed is strictly better on both mandates: the cheap
attempt usually succeeds and costs nothing extra, and when it fails you have
evidence rather than a hunch.
The retry strategy depends on the failure, because the failures have different
causes:
- **truncated** — the answer ran out of budget. If there is a cap to raise,
raise it on the same model. If there is not, the model's own output ceiling
is the wall, so escalate to a candidate that can emit more. (Without that
second case the branch was unreachable in practice: through /v1 the cap is
either the client's, which is not ours to override, or absent.)
- **malformed** — the model produced something that would not parse. More
tokens will not help, so this escalates to the next-best candidate.
Latency is treated as a cost. Every retry doubles time-to-answer, which in
interactive use *is* a quality loss, so interactive work gets a lower cap than
batch work regardless of tier.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal, Optional
INTERACTIVE = "interactive"
BATCH = "batch"
# Multiplier applied to the token budget when retrying a truncated answer.
TRUNCATION_BUDGET_MULTIPLIER = 2
@dataclass
class RetryPlan:
"""How to spend one attempt from the budget."""
reason: Literal["truncated", "malformed"]
model_id: str
max_tokens: Optional[int]
detail: str
def attempts_allowed(
tier: int,
latency_tolerance: str,
attempts_by_tier: dict[int, int],
max_attempts_interactive: int,
) -> int:
"""How many corrective attempts this request has earned.
Interactive requests are capped below their tier's budget: a user waiting
on an answer pays for every retry in latency, and a slow correct answer can
be worth less than a fast one they can judge themselves.
"""
budget = attempts_by_tier.get(tier, 0)
if latency_tolerance == INTERACTIVE:
return min(budget, max_attempts_interactive)
return budget
def plan_retry(
verdict: str,
current_model: str,
current_max_tokens: Optional[int],
model_ceiling: Optional[int],
runners_up: list[tuple[str, Optional[int]]],
client_capped: bool,
) -> Optional[RetryPlan]:
"""Decide how to retry a failed response, or decline to.
``runners_up`` is (model_id, output_ceiling) in ranked order, because a
truncation retry needs to know which alternatives can actually hold a
longer answer.
Returns None when retrying cannot help, which is as important as returning
a plan — a wasted attempt costs energy against a fixed quota.
"""
if verdict == "truncated":
# A client that set its own max_tokens chose this outcome. Overriding
# it would ignore an explicit instruction, and the caller may well want
# a short answer.
if client_capped:
return None
if current_max_tokens is not None:
bigger = current_max_tokens * TRUNCATION_BUDGET_MULTIPLIER
if model_ceiling:
bigger = min(bigger, model_ceiling)
if bigger > current_max_tokens:
return RetryPlan(
"truncated",
current_model,
bigger,
f"retrying {current_model} with {bigger} tokens",
)
# No cap to raise, or already at this model's limit: the model's own
# output ceiling is the wall. Retrying it changes nothing, but a
# candidate that can emit MORE might finish the answer.
roomier = next(
(
(m, ceiling)
for m, ceiling in runners_up
if ceiling and (not model_ceiling or ceiling > model_ceiling)
),
None,
)
if roomier is None:
return None
return RetryPlan(
"truncated",
roomier[0],
current_max_tokens,
f"escalating to {roomier[0]} ({roomier[1]} output tokens)",
)
if verdict == "malformed":
# More tokens will not make unparseable output parse. Try the next
# candidate the ranking preferred.
if not runners_up:
return None
return RetryPlan(
"malformed",
runners_up[0][0],
current_max_tokens,
f"escalating to {runners_up[0][0]}",
)
# 'ok' and 'unverifiable' are not failures and buy no retry. Spending an
# attempt on 'unverifiable' would burn quota on the majority of prose
# traffic for no signal at all.
return None

169
leaderboard.py Normal file
View File

@@ -0,0 +1,169 @@
#!/usr/bin/env python3
"""Import curated leaderboard priors into the ``proficiency`` table.
Reads ``leaderboards.yaml`` (family -> category -> score), writes
``leaderboard_score`` for every catalog row in each family, and re-blends.
Run manually, or after editing the YAML:
python leaderboard.py
python leaderboard.py --check # report coverage, write nothing
The coverage report is the point of running this on a schedule. NeuralWatt
adds models; a newly listed family has no prior and no self-eval history, so
it sits at the neutral 0.5 and is indistinguishable from a model that was
measured and found average. Naming those families is what turns "quietly
unmeasured" into "needs a prior".
"""
from __future__ import annotations
import argparse
import sqlite3
import sys
from pathlib import Path
import yaml
from config import RouterConfig, load_config
from proficiency_store import set_leaderboard
LEADERBOARDS_PATH = "leaderboards.yaml"
def load_priors(path: str | Path, cfg: RouterConfig) -> dict[str, dict[str, float]]:
"""Parse and validate the curated file into {family: {category: score}}.
Validation is strict rather than forgiving: a typo'd category silently
dropped would look exactly like a benchmark that does not cover it, and
an out-of-range score would skew the min-max normalization downstream.
"""
raw = yaml.safe_load(Path(path).read_text()) or {}
families = raw.get("families") or {}
if not isinstance(families, dict):
raise ValueError("leaderboards.yaml: 'families' must be a mapping")
allowed = set(cfg.proficiency.categories)
priors: dict[str, dict[str, float]] = {}
for family, entry in families.items():
if not isinstance(entry, dict):
raise ValueError(f"leaderboards.yaml: {family!r} must be a mapping")
scores = entry.get("scores") or {}
if not isinstance(scores, dict):
raise ValueError(f"leaderboards.yaml: {family!r}.scores must be a mapping")
unknown = set(scores) - allowed
if unknown:
raise ValueError(
f"leaderboards.yaml: {family!r} has categories not in "
f"proficiency.categories: {sorted(unknown)}"
)
for category, score in scores.items():
if not isinstance(score, (int, float)) or not (0.0 <= score <= 1.0):
raise ValueError(
f"leaderboards.yaml: {family!r}.{category} must be a number "
f"in 0..1, got {score!r}"
)
if scores:
priors[family] = {c: float(s) for c, s in scores.items()}
return priors
def active_families(conn: sqlite3.Connection, cfg: RouterConfig) -> dict[str, list[str]]:
"""Routable families mapped to their catalog rows."""
placeholders = ",".join("?" * len(cfg.routing.allowed_access_levels))
rows = conn.execute(
f"""
SELECT base_model_id, model_id FROM models
WHERE availability = 'active'
AND access_level IN ({placeholders})
ORDER BY base_model_id, model_id
""",
tuple(cfg.routing.allowed_access_levels),
).fetchall()
out: dict[str, list[str]] = {}
for base, model_id in rows:
out.setdefault(base, []).append(model_id)
return out
def report_coverage(
families: dict[str, list[str]], priors: dict[str, dict[str, float]]
) -> list[str]:
"""Print per-family coverage; return the families with no prior at all."""
missing = []
print(f"{'family':22s}{'rows':>6}{'categories with a prior':>26}")
for family, model_ids in sorted(families.items()):
scores = priors.get(family, {})
if not scores:
missing.append(family)
print(f"{family:22s}{len(model_ids):>6}{'— none —':>26}")
else:
print(f"{family:22s}{len(model_ids):>6}{len(scores):>26}")
return missing
def apply_priors(
conn: sqlite3.Connection,
cfg: RouterConfig,
families: dict[str, list[str]],
priors: dict[str, dict[str, float]],
) -> int:
"""Write each family's prior onto every catalog row in that family."""
written = 0
for family, model_ids in families.items():
scores = priors.get(family)
if not scores:
continue
for model_id in model_ids:
for category, score in scores.items():
set_leaderboard(conn, cfg, model_id, "neuralwatt", category, score)
written += 1
conn.commit()
return written
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--file", default=LEADERBOARDS_PATH)
ap.add_argument(
"--check", action="store_true", help="report coverage without writing"
)
args = ap.parse_args()
cfg = load_config("config.yaml")
try:
priors = load_priors(args.file, cfg)
except (ValueError, FileNotFoundError) as e:
print(f"{e}", file=sys.stderr)
return 1
conn = sqlite3.connect(cfg.database.path)
families = active_families(conn, cfg)
missing = report_coverage(families, priors)
if not args.check:
written = apply_priors(conn, cfg, families, priors)
print(f"\nwrote {written} leaderboard scores")
conn.close()
if missing:
print(
f"\nWARNING: {len(missing)} active famil"
f"{'y has' if len(missing) == 1 else 'ies have'} no leaderboard prior: "
f"{', '.join(missing)}",
file=sys.stderr,
)
print(
" Until self-eval accumulates, these score the neutral 0.5 — "
"indistinguishable from a model measured and found average.\n"
f" Add real, sourced figures to {args.file}.",
file=sys.stderr,
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

41
leaderboards.yaml Normal file
View File

@@ -0,0 +1,41 @@
# Curated leaderboard priors, keyed by model FAMILY (models.base_model_id).
#
# Why this file is hand-maintained rather than scraped: there is no unified
# leaderboard API, model naming differs everywhere (this catalog's `kimi-k3`
# against a leaderboard's "Kimi K3 (0731)"), and a scraper that silently
# starts returning nothing looks identical to a model that scores nothing.
#
# What it is FOR: cold start. NeuralWatt adds models, and a newly listed one
# has zero self-eval history — without a prior it sits at the neutral 0.5 and
# is indistinguishable from a model that was measured and found average. A
# prior gives it a defensible starting position until the eval harness has
# run enough samples to take over (proficiency.self_eval_min_samples).
#
# SHIPPED EMPTY ON PURPOSE. Inventing plausible-looking numbers here would put
# fabricated data directly into routing decisions — the same failure as the
# provider's static_fallback carbon constant, which this project already
# excludes for exactly that reason. Fill in only figures you actually looked
# up, and say where they came from in `source`.
#
# Serving variants inherit from their family automatically: an entry for
# `kimi-k3` covers kimi-k3-fast and kimi-k3-flex. Do not list variants.
#
# `python leaderboard.py` imports this and names any active family missing an
# entry. Categories must come from proficiency.categories in config.yaml, and
# scores are 0-1. Partial entries are fine — list only the categories a
# benchmark actually measures, and leave the rest to self-eval.
families: {}
# Shape, for when you have real numbers to add:
#
# kimi-k3:
# source: "aider polyglot 2026-08-14; livebench 2026-08-01"
# scores:
# coding_general: 0.82
# coding_refactor: 0.79
# reasoning_math: 0.88
#
# gemma-4-31b:
# source: "lmarena 2026-08-10"
# scores:
# general_chat: 0.61

30
opencode.json Normal file
View File

@@ -0,0 +1,30 @@
{
"$schema": "https://opencode.ai/config.json",
"model": "llm-router/auto",
"provider": {
"llm-router": {
"name": "Local LLM Router",
"npm": "@ai-sdk/openai-compatible",
"options": {
"baseURL": "http://127.0.0.1:8080/v1",
"apiKey": "unused"
},
"models": {
"auto": {
"name": "auto (router picks, interactive)",
"limit": {
"context": 782324,
"output": 16384
}
},
"auto:batch": {
"name": "auto (router picks, admits flex/async)",
"limit": {
"context": 782324,
"output": 16384
}
}
}
}
}
}

227
poller.py
View File

@@ -2,9 +2,9 @@
"""
Pricing/catalog poller for the local LLM router.
Fetches model catalogs from OpenRouter and Neuralwatt (both unauthenticated
public endpoints), normalizes into a common shape, and upserts into the
`models` table in router.db.
Fetches the NeuralWatt model catalog (unauthenticated public endpoint),
normalizes it into the `models` table in router.db, and flags rows that have
gone stale.
Run manually:
python poller.py
@@ -13,7 +13,11 @@ Run on a schedule (cron example, every 2 hours):
0 */2 * * * /usr/bin/python3 /path/to/poller.py >> /var/log/router-poller.log 2>&1
Does NOT touch energy data — energy is only available per-completion, not from
a models list. See energy_logger.py (dispatcher-side) for that.
a models list, so the dispatcher writes `energy_observations` instead.
NeuralWatt is the only provider. The `provider` column and the (model_id,
provider) primary key are kept so a second provider can be added without a
migration.
"""
from __future__ import annotations
@@ -26,26 +30,84 @@ from typing import Optional
import requests
DB_PATH = "router.db"
from config import RouterConfig, load_config
OPENROUTER_MODELS_URL = "https://openrouter.ai/api/v1/models"
NEURALWATT_MODELS_URL = "https://api.neuralwatt.com/v1/models"
REQUEST_TIMEOUT = 20 # seconds
# Fraction of advertised context to treat as usable, before subtracting the
# output reserve. Tune per-provider/model once you have real data; this is a
# deliberately conservative global default to start.
CONTEXT_SAFETY_FACTOR = 0.75
DEFAULT_OUTPUT_RESERVE_TOKENS = 4096
# Serving-class suffixes. NeuralWatt ships one base model as several catalog
# rows that differ only by these tokens, and they combine freely — hence ids
# like 'glm-5.2-short-fast-flex'. They are stripped from the end of the id one
# segment at a time so a base name that merely *contains* a lookalike token is
# never misread (e.g. 'deepseek-v4-flash' is not a '-fast' row).
SUFFIX_FLEX = "flex"
SUFFIX_FAST = "fast"
SUFFIX_SHORT = "short"
SERVING_SUFFIXES = frozenset({SUFFIX_FLEX, SUFFIX_FAST, SUFFIX_SHORT})
STALE_AFTER_DAYS = 3
def parse_serving_class(model_id: str) -> tuple[str, str, str]:
"""Derive (latency_class, reasoning_mode, context_variant) from a model id.
Returns the schema defaults ('standard', 'default', 'full') for a base
model. Suffixes are matched as whole '-'-delimited segments only.
"""
segments = model_id.lower().split("-")
found = set()
while len(segments) > 1 and segments[-1] in SERVING_SUFFIXES:
found.add(segments.pop())
return (
"flex" if SUFFIX_FLEX in found else "standard",
"reduced" if SUFFIX_FAST in found else "default",
"short" if SUFFIX_SHORT in found else "full",
)
def parse_base_model_id(model_id: str) -> str:
"""Reduce a catalog id to the model family underneath it.
``glm-5.2-short-fast-flex`` and ``glm-5.2`` are the same weights served
differently, and ``deepseek-ai/DeepSeek-V4-Flash`` is the HF-style
duplicate of ``deepseek-v4-flash``. Proficiency is a property of the
weights, not of the queue they sit in, so scores are keyed on this and
every serving variant inherits from its family. Leaderboard priors work
the same way — no benchmark rates a ``-flex`` row separately.
Note this deliberately collapses ``-fast`` too, even though reasoning
being off does change answer quality. The eval runner scores ``-fast``
variants separately and overrides the inherited value; the family is the
fallback, not the final word.
"""
namespace_stripped = model_id.rsplit("/", 1)[-1]
segments = namespace_stripped.lower().split("-")
while len(segments) > 1 and segments[-1] in SERVING_SUFFIXES:
segments.pop()
return "-".join(segments)
def parse_access_level(display_name: Optional[str], description: Optional[str]) -> str:
"""Derive an access level from the catalog's prose.
NeuralWatt exposes no structured gating field — restricted models are only
marked in free text ("Private preview (grant-gated)", "(Canary)"). Routing
to one earns a 403 at dispatch, so this is parsed defensively: anything
that looks gated is treated as gated.
"""
blob = f"{display_name or ''} {description or ''}".lower()
if "grant-gated" in blob or "private preview" in blob:
return "preview"
if "canary" in blob:
return "canary"
return "public"
@dataclass
class ModelRow:
model_id: str
provider: str
base_model_id: str
display_name: Optional[str]
cost_per_1m_prompt: Optional[float]
cost_per_1m_completion: Optional[float]
@@ -56,62 +118,24 @@ class ModelRow:
supports_json_mode: bool
supports_vision: bool
supports_reasoning: bool
reasoning_default_enabled: bool
latency_class: str
reasoning_mode: str
context_variant: str
access_level: str
pricing_tbd: bool
deprecated: bool
def effective_context_window(self) -> Optional[int]:
def effective_context_window(self, cfg: RouterConfig) -> Optional[int]:
if not self.context_window:
return None
reserve = self.max_output_tokens or DEFAULT_OUTPUT_RESERVE_TOKENS
usable = int(self.context_window * CONTEXT_SAFETY_FACTOR) - reserve
# 11 of 19 catalog rows report no max_output_tokens, so the configured
# reserve carries most of the catalog.
reserve = self.max_output_tokens or cfg.context.default_output_reserve_tokens
usable = int(self.context_window * cfg.context.safety_factor) - reserve
return max(usable, 0)
def fetch_openrouter() -> list[ModelRow]:
resp = requests.get(OPENROUTER_MODELS_URL, timeout=REQUEST_TIMEOUT)
resp.raise_for_status()
payload = resp.json()
rows = []
for m in payload.get("data", []):
pricing = m.get("pricing", {}) or {}
def _price(key: str) -> Optional[float]:
# OpenRouter returns per-token price as a string, e.g. "0.0000007"
val = pricing.get(key)
if val is None:
return None
try:
return float(val) * 1_000_000 # normalize to per-1M
except (TypeError, ValueError):
return None
top_provider = m.get("top_provider", {}) or {}
rows.append(
ModelRow(
model_id=m.get("id"),
provider="openrouter",
display_name=m.get("name"),
cost_per_1m_prompt=_price("prompt"),
cost_per_1m_completion=_price("completion"),
cost_per_1m_prompt_cached=_price("input_cache_read"),
context_window=m.get("context_length")
or top_provider.get("context_length"),
max_output_tokens=top_provider.get("max_completion_tokens"),
supports_tools="tools" in (m.get("supported_parameters") or []),
supports_json_mode="response_format"
in (m.get("supported_parameters") or []),
supports_vision="image" in (m.get("architecture", {}) or {}).get(
"input_modalities", []
),
supports_reasoning="reasoning" in (m.get("supported_parameters") or []),
pricing_tbd=False,
deprecated=False,
)
)
return rows
def fetch_neuralwatt() -> list[ModelRow]:
resp = requests.get(NEURALWATT_MODELS_URL, timeout=REQUEST_TIMEOUT)
resp.raise_for_status()
@@ -119,15 +143,29 @@ def fetch_neuralwatt() -> list[ModelRow]:
rows = []
for m in payload.get("data", []):
model_id = m.get("id")
meta = m.get("metadata", {}) or {}
pricing = meta.get("pricing", {}) or {}
caps = meta.get("capabilities", {}) or {}
limits = meta.get("limits", {}) or {}
reasoning = meta.get("reasoning") or {}
supports_reasoning = bool(caps.get("reasoning"))
# capabilities.reasoning only means "the API accepts a reasoning
# param" and is true for nearly the whole catalog. default_enabled is
# the discriminating signal; a few models (the kimi-k2.7-code family)
# expose no reasoning block at all, so fall back to the capability.
default_enabled = reasoning.get("default_enabled")
if default_enabled is None:
default_enabled = supports_reasoning
latency_class, reasoning_mode, context_variant = parse_serving_class(model_id)
rows.append(
ModelRow(
model_id=m.get("id"),
model_id=model_id,
provider="neuralwatt",
base_model_id=parse_base_model_id(model_id),
display_name=meta.get("display_name"),
cost_per_1m_prompt=pricing.get("input_per_million"),
cost_per_1m_completion=pricing.get("output_per_million"),
@@ -138,7 +176,14 @@ def fetch_neuralwatt() -> list[ModelRow]:
supports_tools=bool(caps.get("tools")),
supports_json_mode=bool(caps.get("json_mode")),
supports_vision=bool(caps.get("vision")),
supports_reasoning=bool(caps.get("reasoning")),
supports_reasoning=supports_reasoning,
reasoning_default_enabled=bool(default_enabled),
latency_class=latency_class,
reasoning_mode=reasoning_mode,
context_variant=context_variant,
access_level=parse_access_level(
meta.get("display_name"), meta.get("description")
),
pricing_tbd=bool(pricing.get("pricing_tbd")),
deprecated=bool(meta.get("deprecated")),
)
@@ -146,20 +191,23 @@ def fetch_neuralwatt() -> list[ModelRow]:
return rows
def upsert(conn: sqlite3.Connection, rows: list[ModelRow]) -> None:
def upsert(conn: sqlite3.Connection, rows: list[ModelRow], cfg: RouterConfig) -> None:
now = datetime.now(timezone.utc).isoformat()
for r in rows:
availability = "deprecated" if r.deprecated else "active"
conn.execute(
"""
INSERT INTO models (
model_id, provider, display_name,
model_id, provider, base_model_id, display_name,
cost_per_1m_prompt, cost_per_1m_completion, cost_per_1m_prompt_cached,
context_window, effective_context_window, max_output_tokens,
supports_tools, supports_json_mode, supports_vision, supports_reasoning,
reasoning_default_enabled, latency_class, reasoning_mode,
context_variant, access_level,
pricing_tbd, deprecated, availability, last_updated
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(model_id, provider) DO UPDATE SET
base_model_id = excluded.base_model_id,
display_name = excluded.display_name,
cost_per_1m_prompt = excluded.cost_per_1m_prompt,
cost_per_1m_completion = excluded.cost_per_1m_completion,
@@ -171,6 +219,11 @@ def upsert(conn: sqlite3.Connection, rows: list[ModelRow]) -> None:
supports_json_mode = excluded.supports_json_mode,
supports_vision = excluded.supports_vision,
supports_reasoning = excluded.supports_reasoning,
reasoning_default_enabled = excluded.reasoning_default_enabled,
latency_class = excluded.latency_class,
reasoning_mode = excluded.reasoning_mode,
context_variant = excluded.context_variant,
access_level = excluded.access_level,
pricing_tbd = excluded.pricing_tbd,
deprecated = excluded.deprecated,
availability = excluded.availability,
@@ -179,17 +232,23 @@ def upsert(conn: sqlite3.Connection, rows: list[ModelRow]) -> None:
(
r.model_id,
r.provider,
r.base_model_id,
r.display_name,
r.cost_per_1m_prompt,
r.cost_per_1m_completion,
r.cost_per_1m_prompt_cached,
r.context_window,
r.effective_context_window(),
r.effective_context_window(cfg),
r.max_output_tokens,
int(r.supports_tools),
int(r.supports_json_mode),
int(r.supports_vision),
int(r.supports_reasoning),
int(r.reasoning_default_enabled),
r.latency_class,
r.reasoning_mode,
r.context_variant,
r.access_level,
int(r.pricing_tbd),
int(r.deprecated),
availability,
@@ -199,39 +258,39 @@ def upsert(conn: sqlite3.Connection, rows: list[ModelRow]) -> None:
conn.commit()
def mark_stale(conn: sqlite3.Connection) -> None:
def mark_stale(conn: sqlite3.Connection, cfg: RouterConfig) -> None:
"""Flag rows that weren't touched by this poll run as stale, rather than
silently leaving old data looking current."""
conn.execute(
f"""
"""
UPDATE models
SET availability = 'stale'
WHERE availability = 'active'
AND julianday('now') - julianday(last_updated) > {STALE_AFTER_DAYS}
"""
AND julianday('now') - julianday(last_updated) > ?
""",
(cfg.freshness.stale_after_days,),
)
conn.commit()
def main() -> int:
conn = sqlite3.connect(DB_PATH)
cfg = load_config("config.yaml")
conn = sqlite3.connect(cfg.database.path)
conn.execute("PRAGMA foreign_keys = ON")
total = 0
for name, fetch_fn in (("openrouter", fetch_openrouter), ("neuralwatt", fetch_neuralwatt)):
try:
rows = fetch_fn()
upsert(conn, rows)
print(f"[{name}] upserted {len(rows)} models")
total += len(rows)
except requests.RequestException as e:
print(f"[{name}] FAILED: {e}", file=sys.stderr)
# Don't let one provider's outage nuke the whole poll run.
continue
try:
rows = fetch_neuralwatt()
except requests.RequestException as e:
print(f"[neuralwatt] FAILED: {e}", file=sys.stderr)
conn.close()
return 1
mark_stale(conn)
upsert(conn, rows, cfg)
print(f"[neuralwatt] upserted {len(rows)} models")
mark_stale(conn, cfg)
conn.close()
print(f"done, {total} rows upserted total")
print(f"done, {len(rows)} rows upserted total")
return 0

103
proficiency.py Normal file
View File

@@ -0,0 +1,103 @@
"""Pure proficiency blending for the local LLM model router.
Like ``scoring.py``, ``tiering.py`` and ``routing.py``, this module is free of
I/O: scores and thresholds come in as arguments. ``eval_proficiency.py`` owns
the DB writes and the model calls.
Two independent sources feed one number per (model, category):
- **leaderboard** — a curated prior from published benchmarks. Its job is
cold start: NeuralWatt adds models, and a newly listed one has no self-eval
history at all. Without a prior it scores the neutral 0.5 and is
indistinguishable from a model that was measured and found average.
- **self-eval** — this router's own task set, run against the real endpoint.
More predictive of actual routing quality, but it accumulates slowly.
Design doc §3.3 gives the blend as ``0.3 x leaderboard + 0.7 x self_eval``
once self-eval crosses ``self_eval_min_samples``, falling back to the
leaderboard alone before that so thin, noisy self-eval data cannot dominate
early.
That rule assumes a leaderboard entry exists. It often will not — the curated
file is hand-maintained and NeuralWatt ships models faster than public
benchmarks cover them. Taken literally, a model with no prior and 9 samples
would score nothing at all, which is strictly worse than the 9 samples it
actually has. So the fallback ladder is:
leaderboard + enough self-eval -> weighted blend 'blended'
enough self-eval, no prior -> self-eval alone 'self_eval'
prior, not enough self-eval -> leaderboard alone 'leaderboard'
thin self-eval, no prior -> self-eval alone 'self_eval_thin'
neither -> None (neutral 0.5 downstream)
``self_eval_thin`` is deliberately distinguishable: it is real measurement,
but from too few samples to trust as much as the label 'self_eval' implies,
and a caller wanting to exclude it can.
"""
from __future__ import annotations
from typing import Literal, Optional
Source = Literal["blended", "self_eval", "leaderboard", "self_eval_thin"]
def blend(
leaderboard_score: Optional[float],
self_eval_score: Optional[float],
self_eval_samples: int,
*,
leaderboard_weight: float,
self_eval_weight: float,
min_samples: int,
) -> tuple[Optional[float], Optional[Source]]:
"""Combine the two sources into one score, with the source that produced it.
Returns ``(None, None)`` when neither source has anything, which leaves
the candidate on the neutral 0.5 in ``scoring.proficiency_score`` rather
than penalizing it for being unmeasured.
"""
has_leaderboard = leaderboard_score is not None
has_self_eval = self_eval_score is not None and self_eval_samples > 0
enough_samples = has_self_eval and self_eval_samples >= min_samples
if has_leaderboard and enough_samples:
return (
leaderboard_weight * leaderboard_score
+ self_eval_weight * self_eval_score,
"blended",
)
if enough_samples:
return self_eval_score, "self_eval"
if has_leaderboard:
return leaderboard_score, "leaderboard"
if has_self_eval:
return self_eval_score, "self_eval_thin"
return None, None
def accumulate(
previous_score: Optional[float],
previous_samples: int,
new_scores: list[float],
) -> tuple[Optional[float], int]:
"""Fold a fresh eval run into a running mean.
Keeps a running average rather than replacing, so ``self_eval_samples``
means what the blending rule assumes it means: how much evidence stands
behind the score. Re-running the harness therefore tightens an estimate
instead of discarding everything learned before it.
Returns ``(previous_score, previous_samples)`` unchanged when handed no
new scores, so a run where every task errored cannot quietly reset a
model's history to zero.
"""
if not new_scores:
return previous_score, previous_samples
total_samples = previous_samples + len(new_scores)
if previous_score is None or previous_samples <= 0:
return sum(new_scores) / len(new_scores), len(new_scores)
weighted = previous_score * previous_samples + sum(new_scores)
return weighted / total_samples, total_samples

200
proficiency_store.py Normal file
View File

@@ -0,0 +1,200 @@
"""DB access for the ``proficiency`` table.
Pairs with the pure ``proficiency.py`` the way ``tier.py`` pairs with
``tiering.py``: the blending arithmetic stays testable without a database,
and every write goes through here so ``blended_score`` and ``source`` can
never drift out of step with the two inputs that produced them.
Both writers use this — ``leaderboard.py`` sets priors, ``eval_proficiency.py``
folds in measured runs — and each write re-blends the row it touched.
"""
from __future__ import annotations
import sqlite3
from datetime import datetime, timezone
from typing import Optional
from config import RouterConfig
from proficiency import accumulate, blend
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _read_row(
conn: sqlite3.Connection, model_id: str, provider: str, category: str
) -> Optional[sqlite3.Row]:
conn.row_factory = sqlite3.Row
return conn.execute(
"""
SELECT leaderboard_score, self_eval_score, self_eval_samples
FROM proficiency
WHERE model_id = ? AND provider = ? AND category = ?
""",
(model_id, provider, category),
).fetchone()
def _write(
conn: sqlite3.Connection,
cfg: RouterConfig,
model_id: str,
provider: str,
category: str,
leaderboard_score: Optional[float],
self_eval_score: Optional[float],
self_eval_samples: int,
) -> None:
blended, source = blend(
leaderboard_score,
self_eval_score,
self_eval_samples,
leaderboard_weight=cfg.proficiency.leaderboard_weight,
self_eval_weight=cfg.proficiency.self_eval_weight,
min_samples=cfg.proficiency.self_eval_min_samples,
)
conn.execute(
"""
INSERT INTO proficiency (
model_id, provider, category, leaderboard_score,
self_eval_score, self_eval_samples, blended_score, source, last_updated
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(model_id, provider, category) DO UPDATE SET
leaderboard_score = excluded.leaderboard_score,
self_eval_score = excluded.self_eval_score,
self_eval_samples = excluded.self_eval_samples,
blended_score = excluded.blended_score,
source = excluded.source,
last_updated = excluded.last_updated
""",
(
model_id,
provider,
category,
leaderboard_score,
self_eval_score,
self_eval_samples,
blended,
source,
_now(),
),
)
def set_leaderboard(
conn: sqlite3.Connection,
cfg: RouterConfig,
model_id: str,
provider: str,
category: str,
score: Optional[float],
) -> None:
"""Set a category's leaderboard prior, preserving any self-eval history."""
existing = _read_row(conn, model_id, provider, category)
_write(
conn,
cfg,
model_id,
provider,
category,
score,
existing["self_eval_score"] if existing else None,
existing["self_eval_samples"] if existing else 0,
)
def add_self_eval(
conn: sqlite3.Connection,
cfg: RouterConfig,
model_id: str,
provider: str,
category: str,
scores: list[float],
) -> None:
"""Fold an eval run's per-task scores into the running self-eval mean.
Accumulates rather than replaces, so re-running the harness tightens the
estimate instead of discarding what came before — and so
``self_eval_samples`` keeps meaning "how much evidence stands behind
this", which is what the blending threshold is gating on.
"""
existing = _read_row(conn, model_id, provider, category)
prev_score = existing["self_eval_score"] if existing else None
prev_samples = existing["self_eval_samples"] if existing else 0
new_score, new_samples = accumulate(prev_score, prev_samples, scores)
_write(
conn,
cfg,
model_id,
provider,
category,
existing["leaderboard_score"] if existing else None,
new_score,
new_samples,
)
def propagate_to_variants(
conn: sqlite3.Connection, cfg: RouterConfig, source_model_id: str, provider: str
) -> int:
"""Copy an evaluated row's scores onto its equivalent serving variants.
A ``-flex`` row is the same weights, same reasoning setting, same context
pool, on a different queue — so its answer quality IS the standard row's
and inheriting is correct. A ``-fast`` row is NOT equivalent (reasoning
off or capped), and neither is a ``-short`` row, so matching is on
(base_model_id, reasoning_mode, context_variant) rather than on family
alone. Inheriting across those would attribute reasoning-on quality to a
reasoning-off row.
Anything already measured directly keeps its own scores; inheritance is
the fallback, never an overwrite.
Returns the number of rows written.
"""
conn.row_factory = sqlite3.Row
source_rows = conn.execute(
"""
SELECT category, leaderboard_score, self_eval_score, self_eval_samples
FROM proficiency
WHERE model_id = ? AND provider = ?
""",
(source_model_id, provider),
).fetchall()
if not source_rows:
return 0
variants = conn.execute(
"""
SELECT v.model_id FROM models v
JOIN models src
ON src.base_model_id = v.base_model_id
AND src.provider = v.provider
AND src.reasoning_mode = v.reasoning_mode
AND src.context_variant = v.context_variant
WHERE src.model_id = ? AND v.provider = ? AND v.model_id != ?
""",
(source_model_id, provider, source_model_id),
).fetchall()
written = 0
for variant in variants:
for row in source_rows:
existing = _read_row(conn, variant["model_id"], provider, row["category"])
if existing and (existing["self_eval_samples"] or 0) > 0:
continue # measured directly; do not overwrite with the family's
_write(
conn,
cfg,
variant["model_id"],
provider,
row["category"],
row["leaderboard_score"],
row["self_eval_score"],
row["self_eval_samples"],
)
written += 1
return written

7
pyproject.toml Normal file
View File

@@ -0,0 +1,7 @@
[tool.pytest.ini_options]
# The router modules sit at the repo root rather than in a package, so the
# root has to be on sys.path for `from config import ...` to resolve. Running
# as `python -m pytest` happens to add the cwd and hides this; the `pytest`
# console script does not. Setting it here makes both invocations work.
pythonpath = ["."]
testpaths = ["tests"]

20
requirements.txt
View File

@@ -1,8 +1,12 @@
pyyaml>=6.0
pydantic>=2.0
requests>=2.31
fastapi>=0.110
uvicorn[standard]>=0.29
openai>=1.30
pytest>=8.0
pytest-cov>=5.0
# Pinned. Recreating the venv with >= constraints silently jumped openai
# 2.53 -> 3.0 and httpx -> httpx2; a service that restarts on boot should not
# change its dependency tree underneath itself. Bump deliberately.
pyyaml==6.0.3
pydantic==2.13.4
requests==2.34.2
fastapi==0.141.1
uvicorn[standard]==0.52.1
openai==3.0.0
python-dotenv==1.2.2
pytest==9.1.1
pytest-cov==7.1.0

235
routing.py Normal file
View File

@@ -0,0 +1,235 @@
"""Pure candidate selection and ranking for the local LLM model router.
Like ``scoring.py`` and ``tiering.py``, this module is free of I/O: rows come
in as dicts, thresholds come in as arguments. ``dispatcher.py`` owns the DB
reads and the provider call.
Two stages, in order:
1. **Hard filters** (``select_candidates``) — disqualify outright. A model
that cannot hold the context, is below the required tier, is stale, or is
not actually reachable by this account is not a cost tradeoff to be
weighed; it is not a candidate at all (design doc §3.2, §4).
2. **Weighted scoring** (``rank_candidates``) — order what survives by
``w_cost*cost + w_eco*eco + w_prof*proficiency``.
The flex filter is the one that is easy to get wrong. NeuralWatt's ``-flex``
rows are the same weights at the same advertised price, so on every scored
dimension they tie exactly with their standard sibling — but they are "held
server-side during peak until a capacity gap opens". Left to the scorer, a
coin flip decides whether an interactive request waits out a capacity gap.
Latency tolerance is therefore a filter, not a weight.
"""
from __future__ import annotations
from collections.abc import Sequence
from scoring import cost_score, proficiency_score
INTERACTIVE = "interactive"
BATCH = "batch"
def is_eligible(
row: dict,
*,
required_context_tokens: int,
required_tier: int,
latency_tolerance: str,
allowed_access_levels: Sequence[str],
exclude_stale: bool,
exclude_deprecated: bool,
) -> bool:
"""Whether a model row survives every hard filter.
A NULL ``effective_context_window`` fails the context filter: an unknown
window cannot be shown to be large enough, and silently truncating
mid-task is worse than routing elsewhere.
"""
eff_ctx = row.get("effective_context_window")
if eff_ctx is None or eff_ctx < required_context_tokens:
return False
tier = row.get("tier")
if tier is None or tier < required_tier:
return False
availability = row.get("availability")
if exclude_stale and availability == "stale":
return False
if exclude_deprecated and (availability == "deprecated" or row.get("deprecated")):
return False
if row.get("access_level", "public") not in allowed_access_levels:
return False
# Flex rows may be queued behind a capacity gap; only batch work opts in.
if latency_tolerance == INTERACTIVE and row.get("latency_class") == "flex":
return False
return True
def select_candidates(
rows: Sequence[dict],
*,
required_context_tokens: int,
required_tier: int,
latency_tolerance: str,
allowed_access_levels: Sequence[str],
exclude_stale: bool,
exclude_deprecated: bool,
) -> list[dict]:
"""Apply every hard filter, preserving input order."""
return [
row
for row in rows
if is_eligible(
row,
required_context_tokens=required_context_tokens,
required_tier=required_tier,
latency_tolerance=latency_tolerance,
allowed_access_levels=allowed_access_levels,
exclude_stale=exclude_stale,
exclude_deprecated=exclude_deprecated,
)
]
def estimated_cost(
row: dict,
prompt_tokens: int,
completion_tokens: int,
cache_rate: float,
) -> float | None:
"""What this request should cost on this model, from catalog prices.
Replaced a benchmark. The old cost signal came from a fixed 400-token
reference sweep, and it was measured to be WRONG for real traffic: on a
400-token prompt glm-5.2-fast looked 3.2x cheaper than deepseek-v4-flash,
but on a realistic 70k-token prompt deepseek is 5.0x cheaper. Attribution
inverts with prompt size — glm batches beautifully on toy prompts and badly
on real ones — so a fixed-shape benchmark cannot rank models for a workload
of a different shape.
Catalog prices, scaled to THIS request's shape, get the same answer as the
live measurement (7.8x vs 5.0x, same direction). They are also free, need
no sweep, and update whenever the poller runs.
They are not what gets billed — NeuralWatt charges per kWh — but billing is
capped at a multiple of token price, so this tracks the real ordering and
bounds it. Cheap and directionally right beats precise about the wrong
workload.
"""
prompt_price = row.get("cost_per_1m_prompt")
completion_price = row.get("cost_per_1m_completion")
if prompt_price is None or completion_price is None:
return None
# Agent traffic resends the conversation every turn, so most prompt tokens
# hit the provider's prefix cache and are billed at the cached rate.
cached_price = row.get("cost_per_1m_prompt_cached")
if cached_price is None:
cached_price = prompt_price
fresh = prompt_tokens * (1.0 - cache_rate) * prompt_price
cached = prompt_tokens * cache_rate * cached_price
return (fresh + cached + completion_tokens * completion_price) / 1_000_000
def within_budget(row: dict, max_energy_kwh: float | None) -> bool:
"""Whether a candidate's measured energy is inside the per-request ceiling.
Energy rather than dollars because the plan is a subscription with a fixed
kWh quota. Dollars accrue and can be reasoned about after the fact; a quota
is a wall you hit in the middle of a task.
A model with no energy measurement is admitted. Excluding the unmeasured
would mean a newly listed model could never be picked and so could never
acquire a measurement — the same trap the neutral-0.5 default avoids
elsewhere.
"""
if max_energy_kwh is None:
return True
energy = row.get("energy")
return energy is None or energy <= max_energy_kwh
def rank_candidates(
rows: Sequence[dict],
*,
quality_tolerance: float = 0.1,
max_energy_per_request: float | None = None,
prompt_tokens: int = 0,
completion_tokens: int = 500,
cache_rate: float = 0.84,
) -> list[dict]:
"""Order candidates: best quality first, cheapest among equals.
This replaced a weighted blend of cost, eco and proficiency, for reasons
measurement forced:
- **Eco is no longer an objective.** Carbon is still logged per request,
but it is not something this router optimizes; that judgement is made
outside it. Leaving it in meant 20% of every decision optimized an
unstated goal.
- **Cost is a constraint and a tiebreak, not a weight.** Under the blend,
60% of the decision adjudicated differences of fractions of a cent —
all real traffic to date totals $0.07 — and min-max normalization made
"expensive" relative to whoever else happened to be a candidate, so a
model could lose for being 2x a very cheap model when both round to
nothing. A ceiling states the cost mandate as a guarantee instead.
- **Quality is the objective**, which is what the blend obscured: turning
the cost weight from 0.4 all the way to zero changed the winner in only
2 of 6 categories, because the blend was never really steering on
quality at all.
``quality_tolerance`` is the band inside which two proficiency scores are
treated as equal. It is not a preference — it reflects measurement noise.
Proficiency currently rests on 2-3 samples per category, so a gap of 0.05
is indistinguishable from sampling variation, and paying more for it would
be buying noise. Widen it as confidence falls, narrow it as samples
accumulate.
Returns each row plus ``proficiency_score``, ``cost_score`` (reported for
visibility only, no longer part of the decision) and ``composite``, which
is now simply the effective quality after the tolerance band.
"""
affordable = [r for r in rows if within_budget(r, max_energy_per_request)]
# Priced for THIS request rather than for a benchmark, which is the whole
# point: the ordering depends on the workload's shape. A row the catalog
# has no price for keeps whatever measured cost it arrived with rather
# than losing the field — a missing list price is not free.
estimates = []
for r in affordable:
est = estimated_cost(r, prompt_tokens, completion_tokens, cache_rate)
estimates.append(r.get("cost") if est is None else est)
# cost_score is retained purely so callers can still see the spread; it
# does not enter the ordering.
cost_scores = cost_score(estimates)
ranked = []
for row, c_s, est in zip(affordable, cost_scores, estimates):
p_s = proficiency_score(row.get("proficiency"))
ranked.append({**row, "cost": est, "cost_score": c_s,
"proficiency_score": p_s, "composite": p_s})
if not ranked:
return []
# Quality first, but only differences larger than the tolerance count.
# Bucketing by band means a 0.01 edge cannot outrank a 10x cost saving,
# while a real gap (tool_use_agentic spans 0.67) still decides outright.
best = max(r["proficiency_score"] for r in ranked)
def band(r: dict) -> int:
return int((best - r["proficiency_score"]) / quality_tolerance)
ranked.sort(
key=lambda r: (
band(r),
r["cost"] if r.get("cost") is not None else float("inf"),
r["model_id"],
)
)
return ranked

125
schema.sql
View File

@@ -6,7 +6,12 @@ PRAGMA foreign_keys = ON;
-- One row per (model_id, provider). Refreshed by the pricing poller.
CREATE TABLE IF NOT EXISTS models (
model_id TEXT NOT NULL,
provider TEXT NOT NULL, -- 'openrouter' | 'neuralwatt'
provider TEXT NOT NULL, -- 'neuralwatt' (only provider today)
-- The model family under the serving suffixes: glm-5.2-short-fast-flex
-- and glm-5.2 share one. Proficiency and leaderboard priors are properties
-- of the weights, not the queue, so both key on this and every variant
-- inherits from its family.
base_model_id TEXT,
display_name TEXT,
cost_per_1m_prompt REAL, -- USD, null if pricing_tbd
cost_per_1m_completion REAL,
@@ -18,7 +23,29 @@ CREATE TABLE IF NOT EXISTS models (
supports_tools INTEGER DEFAULT 0, -- boolean 0/1
supports_json_mode INTEGER DEFAULT 0,
supports_vision INTEGER DEFAULT 0,
supports_reasoning INTEGER DEFAULT 0,
supports_reasoning INTEGER DEFAULT 0, -- capabilities.reasoning: "API accepts a
-- reasoning param", NOT a quality signal.
-- True for ~90% of the catalog; do not tier on it.
-- Whether reasoning is ON by default (metadata.reasoning.default_enabled), falling back to
-- supports_reasoning when the model exposes no reasoning block. This is the tier-bearing signal.
reasoning_default_enabled INTEGER DEFAULT 0,
-- Serving class. NeuralWatt ships one base model as several rows that differ only by suffix;
-- these three dimensions are orthogonal, hence ids like 'glm-5.2-short-fast-flex'. They carry
-- no price difference in the catalog, so without these columns such rows tie exactly and the
-- router picks between them arbitrarily.
latency_class TEXT DEFAULT 'standard', -- 'standard' | 'flex' (-flex: discounted
-- async, held server-side during peak)
reasoning_mode TEXT DEFAULT 'default', -- 'default' | 'reduced' (-fast: thinking
-- disabled or capped to a short budget)
context_variant TEXT DEFAULT 'full', -- 'full' | 'short' (-short: 200K pool with
-- a bounded reasoning budget)
-- Access gating is prose-only in the catalog ("Private preview (grant-gated)", "canary"), so
-- it is parsed from the description. Non-public rows are excluded from routing by default,
-- otherwise the dispatcher selects them and takes a 403.
access_level TEXT DEFAULT 'public', -- 'public' | 'preview' | 'canary'
pricing_tbd INTEGER DEFAULT 0,
deprecated INTEGER DEFAULT 0,
availability TEXT DEFAULT 'active', -- 'active' | 'deprecated' | 'stale'
@@ -51,16 +78,106 @@ CREATE TABLE IF NOT EXISTS energy_observations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
model_id TEXT NOT NULL,
provider TEXT NOT NULL,
-- The provider's completion id (chatcmpl-...). The client receives this in
-- the response body and in every stream chunk, so it is the join key that
-- lets a client report back later whether the answer actually worked.
request_id TEXT,
-- Fingerprint of the conversation this completion belongs to, derived from
-- its opening message. Stable across a session's turns and distinct
-- between sessions, so the router can tell whether two clients are active
-- WITHOUT the clients cooperating. Used to refuse ambiguous outcome
-- attribution rather than guess.
session_key TEXT,
-- Working directory, when the conversation reveals one. Agent clients
-- usually put the cwd in their system prompt, which makes an outcome
-- report from that directory attributable even under concurrency.
session_dir TEXT,
task_category TEXT,
prompt_tokens INTEGER,
completion_tokens INTEGER,
energy_kwh REAL, -- from response.energy.energy_kwh, when present
-- energy_kwh is what NeuralWatt BILLS, but it is not a measure of model
-- efficiency: it equals avg_power_watts * duration_seconds *
-- attribution_ratio, where attribution_ratio is this request's share of a
-- shared multi-tenant GPU pool. Eight identical calls to one model inside
-- one minute varied 20x in energy_kwh and 22x in attribution_ratio,
-- correlation +0.997, while duration and power held steady. Ranking models
-- on this ranks how busy the provider was, not how efficient the model is.
energy_kwh REAL, -- response.energy.energy_kwh (attributed, noisy)
energy_btu REAL, -- energy_kwh * 3412.14, purely for comedic dashboard value
cost_usd REAL, -- computed from tokens x pricing at call time
-- The pre-attribution terms. avg_power_watts * duration_seconds is the
-- pool's energy over the request, independent of how many other tenants
-- shared it, and it is ~10x more stable (1.85x spread vs 20x on the same
-- eight calls). That product is what scoring ranks on.
avg_power_watts REAL, -- response.energy.avg_power_watts
duration_seconds REAL, -- response.energy.duration_seconds
attribution_ratio REAL, -- kept so the noise term stays inspectable
-- Carbon is what design doc §4 actually scores eco on, and NeuralWatt
-- reports it per-request rather than making us derive it. Grid intensity
-- and region are stored alongside because the same energy in a different
-- region is a different carbon figure -- keeping them makes the open
-- question (real-time intensity vs per-model average) answerable later
-- from logged data instead of a re-run.
carbon_g_co2eq REAL, -- response.energy.carbon_g_co2eq
grid_carbon_intensity REAL, -- gCO2/kWh at call time
grid_id TEXT, -- e.g. 'FI'
-- How the carbon figure was obtained. 'static_fallback' means NeuralWatt
-- could not resolve live grid data and substituted a constant (475.0,
-- about a global average) while still reporting the original grid_id --
-- so the number is a placeholder, not a measurement. Routing on it would
-- penalize a model against a made-up figure, so eco scoring excludes it.
carbon_source TEXT, -- 'agent_cache' | 'static_fallback' | ...
-- The provider's own billed figure (response.cost.request_cost_usd), NOT
-- a tokens x list-price estimate. These disagree: flex rows bill roughly
-- 40% under their standard sibling while the catalog advertises both at
-- the same price, so the estimate would be wrong for every flex call.
cost_usd REAL,
allowance_remaining_usd REAL, -- response.cost.allowance_remaining_usd
service_tier TEXT, -- response.service_tier, as billed
observed_at TEXT NOT NULL
);
-- One row per verified completion. Structural checks are free, so every
-- response gets one; the point is to learn which models fail on REAL work
-- rather than only on the fixed 23-task benchmark in evals/tasks.yaml.
--
-- 'unverifiable' is recorded and is NOT a failure. Most prose lands there,
-- and counting "we could not check this" as "this was wrong" would penalize
-- models for the checker's limits.
CREATE TABLE IF NOT EXISTS verifications (
id INTEGER PRIMARY KEY AUTOINCREMENT,
model_id TEXT NOT NULL,
provider TEXT NOT NULL,
request_id TEXT, -- provider completion id, for client reports
task_category TEXT,
kind TEXT NOT NULL, -- 'structural' | 'local_llm' | 'client_outcome'
-- 'succeeded'/'failed' come only from client_outcome and are ground truth:
-- the client ran the code, or used the answer, and knows. Every other
-- verdict is a proxy for that.
verdict TEXT NOT NULL, -- ok | truncated | malformed | unverifiable
-- | succeeded | failed
detail TEXT,
completion_tokens INTEGER, -- what a wasted answer cost, for the payoff sum
observed_at TEXT NOT NULL,
-- Set once feedback.py has folded this row into proficiency, so re-running
-- cannot penalize a model repeatedly for the same bad response.
applied_at TEXT,
-- Whether this failure is the MODEL's fault. A client that sets a tight
-- max_tokens and gets a truncated answer caused that itself; counting it
-- against the model would let any agent with a small cap systematically
-- drag down whatever it routed to. Still recorded -- the response really
-- was unusable -- but excluded from proficiency feedback.
model_attributable INTEGER DEFAULT 1
);
CREATE INDEX IF NOT EXISTS idx_verifications_model ON verifications (model_id, provider);
CREATE INDEX IF NOT EXISTS idx_verifications_verdict ON verifications (verdict);
CREATE INDEX IF NOT EXISTS idx_observations_request ON energy_observations (request_id);
CREATE INDEX IF NOT EXISTS idx_models_provider ON models (provider);
CREATE INDEX IF NOT EXISTS idx_models_availability ON models (availability);
CREATE INDEX IF NOT EXISTS idx_models_routing ON models (access_level, latency_class, tier);
CREATE INDEX IF NOT EXISTS idx_proficiency_category ON proficiency (category);
CREATE INDEX IF NOT EXISTS idx_energy_model ON energy_observations (model_id, provider);

50
scoring.py
View File

@@ -16,43 +16,37 @@ from __future__ import annotations
from collections.abc import Sequence
def cost_score(costs: Sequence[float | None]) -> list[float]:
"""Inverted min-max cost scores over a candidate set.
def normalize_inverted(values: Sequence[float | None]) -> list[float]:
"""Inverted min-max over a candidate set: lower input = higher score.
score = (max_cost - cost) / (max_cost - min_cost); lower cost = higher
score. A $0-cheapest model legitimately scores 1.0 (free is cheapest);
paid models scale relative to max — this compression is intended, not a
bug (Metis finding #5).
score = (max - value) / (max - min). A cheapest/cleanest candidate scores
1.0; the rest scale relative to max — this compression is intended.
None costs are excluded from the min/max computation and get a neutral
0.5. If no candidate has a known cost, all get 0.5. If min == max
(all equal or a single candidate), present-data candidates get 1.0.
None values are excluded from the min/max computation and get a neutral
0.5, so a model is never penalized merely for having no data yet. If no
candidate has data, all get 0.5. If min == max (all equal, or a single
candidate), present-data candidates get 1.0.
"""
known = [c for c in costs if c is not None]
known = [v for v in values if v is not None]
if not known:
return [0.5 for _ in costs]
return [0.5 for _ in values]
lo, hi = min(known), max(known)
if lo == hi:
return [1.0 if c is not None else 0.5 for c in costs]
return [1.0 if v is not None else 0.5 for v in values]
span = hi - lo
return [0.5 if c is None else (hi - c) / span for c in costs]
return [0.5 if v is None else (hi - v) / span for v in values]
def eco_score(eco_values: Sequence[float | None]) -> list[float]:
"""Inverted min-max eco scores over candidates that have eco data.
Lower eco (carbon/energy) = higher score. Candidates without eco data
(None) get a neutral 0.5. If no candidate has eco data, all get 0.5.
If min == max among present-data candidates, they get 1.0.
"""
known = [e for e in eco_values if e is not None]
if not known:
return [0.5 for _ in eco_values]
lo, hi = min(known), max(known)
if lo == hi:
return [1.0 if e is not None else 0.5 for e in eco_values]
span = hi - lo
return [0.5 if e is None else (hi - e) / span for e in eco_values]
# Cost and eco normalize identically; they differ only in what is fed to
# them. Both are kept as named axes because they rank models DIFFERENTLY:
# cost tracks energy (NeuralWatt bills per kWh) while carbon is energy times
# the serving region's grid intensity, and that intensity spans 37 gCO2/kWh
# (FI) to 505 (US-MIDA-PJM) across the catalog. glm-5.2-fast is the second
# cheapest model and only the sixth cleanest; kimi-k3-flex draws 3.7x less
# energy than kimi-k2.7-code while emitting 3.6x more carbon. Collapsing
# these into one axis would silently pick a side.
cost_score = normalize_inverted
eco_score = normalize_inverted
def proficiency_score(blended: float | None) -> float:

225
seed_energy.py Normal file
View File

@@ -0,0 +1,225 @@
#!/usr/bin/env python3
"""Seed `energy_observations` by running a fixed reference task per model.
Why this exists: `eco_score` returns the neutral 0.5 for every candidate
until a model has observations, and observations only accrue from real
traffic. That is a cold start the router cannot route its way out of — every
model looks identical on eco, so eco contributes nothing to the very
decisions that would generate the data. This sweep breaks the cycle.
Method: one fixed prompt, one fixed `max_tokens`, `temperature=0`, run N
times per model. The prompt is held constant so the resulting figures are
comparable *across* models rather than reflecting who happened to be asked a
harder question.
N > 1 for a specific reason. The provider's billed `energy_kwh` is
`avg_power_watts * duration_seconds * attribution_ratio`, and that last term
is the request's share of a shared multi-tenant GPU pool — eight identical
calls to one model inside one minute spanned 20x, correlating +0.997 with
the attribution ratio while power and duration held steady. Two earlier
sweeps of these same 13 models disagreed by up to 36x on that basis.
So routing scores on `power * duration` (see `dispatcher.gross_energy_kwh`),
which held to 1.85x across those same eight calls. The summary prints both
spreads side by side; if the billed column is wildly wider than the gross
column, that is the artifact, not the models.
What lands in the table are real observations of real calls, identical in
kind to what the dispatcher logs; they are simply generated deliberately
rather than incidentally. They carry `task_category='seed_reference'` so
they can be identified or purged later.
Usage:
python seed_energy.py # 5 samples of every routable model
python seed_energy.py --samples 3
python seed_energy.py --models kimi-k3,gemma-4-31b
python seed_energy.py --dry-run # show the plan and cost, call nothing
"""
from __future__ import annotations
import argparse
import os
import sqlite3
import statistics
import sys
import time
import requests
from config import load_config
from dispatcher import extract_telemetry, gross_energy_kwh, log_observation
# Held constant across models so the energy numbers are comparable. Long
# enough that most models run to the token cap rather than stopping early
# (a short answer burns less energy and would look misleadingly efficient),
# and generic enough that no model is advantaged by domain fit.
REFERENCE_PROMPT = (
"Explain how a B-tree works, including its structure, how lookups "
"descend the tree, and how splits keep it balanced on insert."
)
REFERENCE_MAX_TOKENS = 400
SEED_CATEGORY = "seed_reference"
def routable_models(conn: sqlite3.Connection, allowed_levels: list[str]) -> list[dict]:
placeholders = ",".join("?" * len(allowed_levels))
rows = conn.execute(
f"""
SELECT model_id, provider, latency_class, tier, cost_per_1m_completion
FROM models
WHERE access_level IN ({placeholders})
AND availability = 'active'
ORDER BY model_id
""",
tuple(allowed_levels),
).fetchall()
return [dict(r) for r in rows]
def sample_once(base_url: str, api_key: str, model_id: str, timeout: int = 300) -> dict:
resp = requests.post(
f"{base_url}/chat/completions",
headers={"authorization": f"Bearer {api_key}"},
json={
"model": model_id,
"messages": [{"role": "user", "content": REFERENCE_PROMPT}],
"max_tokens": REFERENCE_MAX_TOKENS,
"temperature": 0,
},
timeout=timeout,
)
resp.raise_for_status()
return resp.json()
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--samples", type=int, default=5, help="samples per model (default 5)")
ap.add_argument("--models", help="comma-separated model_ids; default is all routable")
ap.add_argument("--max-tokens", type=int, default=REFERENCE_MAX_TOKENS)
ap.add_argument("--dry-run", action="store_true", help="print the plan, call nothing")
args = ap.parse_args()
cfg = load_config("config.yaml")
conn = sqlite3.connect(cfg.database.path)
conn.row_factory = sqlite3.Row
models = routable_models(conn, cfg.routing.allowed_access_levels)
conn.close()
if args.models:
wanted = {m.strip() for m in args.models.split(",")}
models = [m for m in models if m["model_id"] in wanted]
missing = wanted - {m["model_id"] for m in models}
if missing:
print(f"not routable / unknown: {', '.join(sorted(missing))}", file=sys.stderr)
if not models:
print("no models to sweep", file=sys.stderr)
return 1
total_calls = len(models) * args.samples
print(f"{len(models)} models x {args.samples} samples = {total_calls} calls")
print(f"prompt: {REFERENCE_PROMPT[:60]}... (max_tokens={args.max_tokens}, temperature=0)")
if args.dry_run:
for m in models:
print(f" {m['model_id']:26s} tier {m['tier']} {m['latency_class']}")
return 0
settings = cfg.dispatch_providers["neuralwatt"]
api_key = os.environ.get(settings.api_key_env)
if not api_key:
print(f"{settings.api_key_env} is not set", file=sys.stderr)
return 1
results: dict[str, list[dict]] = {}
allowance_start = allowance_end = None
for m in models:
model_id = m["model_id"]
results[model_id] = []
for i in range(args.samples):
try:
payload = sample_once(settings.base_url, api_key, model_id)
except requests.RequestException as e:
print(f" {model_id:26s} sample {i + 1}: FAILED {type(e).__name__}: {e}")
continue
usage = payload.get("usage") or {}
telemetry = extract_telemetry(payload)
log_observation(
model_id,
m["provider"],
SEED_CATEGORY,
usage.get("prompt_tokens"),
usage.get("completion_tokens"),
telemetry,
)
if telemetry.allowance_remaining_usd is not None:
if allowance_start is None:
allowance_start = telemetry.allowance_remaining_usd
allowance_end = telemetry.allowance_remaining_usd
gross = None
if telemetry.avg_power_watts and telemetry.duration_seconds:
gross = gross_energy_kwh(
telemetry.avg_power_watts, telemetry.duration_seconds
)
results[model_id].append(
{
"completion_tokens": usage.get("completion_tokens"),
"energy_kwh": telemetry.energy_kwh,
"gross_kwh": gross,
"carbon": telemetry.carbon_g_co2eq,
"cost": telemetry.cost_usd,
}
)
# Be a considerate neighbour on a shared endpoint.
time.sleep(0.3)
done = [d for d in results[model_id] if d["gross_kwh"]]
if done:
med = statistics.median(d["gross_kwh"] for d in done)
print(
f" {model_id:26s} {len(done)}/{args.samples} ok "
f"median gross {med:.3e} kWh"
)
# --- summary ---------------------------------------------------------
# Two spread columns, because the whole point of this sweep is that they
# differ: the billed figure carries a multi-tenancy attribution term that
# the pre-attribution product does not.
print()
print(
f"{'model':26s}{'n':>3}{'gross kWh':>12}{'gross spr':>11}"
f"{'billed kWh':>12}{'billed spr':>12}{'gCO2eq':>11}"
)
for model_id, rows in results.items():
rows = [r for r in rows if r["gross_kwh"]]
if not rows:
print(f"{model_id:26s} 0 (no successful samples)")
continue
gross = [r["gross_kwh"] for r in rows]
billed = [r["energy_kwh"] for r in rows if r["energy_kwh"]]
carbons = [r["carbon"] for r in rows if r["carbon"] is not None]
def spread(xs):
return max(xs) / min(xs) if xs and min(xs) else float("nan")
print(
f"{model_id:26s}{len(rows):>3}{statistics.median(gross):>12.3e}"
f"{spread(gross):>10.1f}x{statistics.median(billed) if billed else 0:>12.3e}"
f"{spread(billed):>11.1f}x"
f"{statistics.median(carbons) if carbons else 0:>11.2e}"
)
if allowance_start is not None and allowance_end is not None:
print()
print(
f"allowance: {allowance_start:.6f} -> {allowance_end:.6f} USD "
f"(spent {allowance_start - allowance_end:.6f})"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

56
tests/test_apply_tiering.py
View File

@@ -1,9 +1,9 @@
"""Integration tests for router-package/tier.py — apply_tiering DB upsert.
"""Integration tests for tier.py — apply_tiering DB upsert.
Seeds a temp sqlite DB from schema.sql, inserts synthetic model rows with
known expected tiers, and asserts the tiering pass writes the right tier
for every row (no NULLs), honors the override map, matches the heuristic,
is idempotent, and warns when supports_reasoning is uniformly False.
is idempotent, and warns when reasoning_default_enabled is uniformly False.
"""
import sqlite3
@@ -41,19 +41,27 @@ def db(tmp_path):
def _insert(conn: sqlite3.Connection, rows: list[dict]) -> None:
"""Insert synthetic model rows (only the columns the tiering pass reads)."""
"""Insert synthetic model rows (only the columns the tiering pass reads).
``supports_reasoning`` defaults to mirroring ``reasoning_default_enabled``
so callers only specify the signal under test; the fallback path is
exercised by setting them independently.
"""
for r in rows:
default_enabled = r["reasoning_default_enabled"]
conn.execute(
"""
INSERT INTO models (
model_id, provider, supports_reasoning,
cost_per_1m_completion, pricing_tbd, last_updated
) VALUES (?, ?, ?, ?, ?, ?)
model_id, provider, supports_reasoning, reasoning_default_enabled,
reasoning_mode, cost_per_1m_completion, pricing_tbd, last_updated
) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""",
(
r["model_id"],
r["provider"],
int(r["supports_reasoning"]),
int(r.get("supports_reasoning", default_enabled)),
int(default_enabled),
r.get("reasoning_mode", "default"),
r["cost_per_1m_completion"],
int(r["pricing_tbd"]),
"2026-08-07T00:00:00+00:00",
@@ -76,15 +84,15 @@ def _tiers(conn: sqlite3.Connection) -> dict[tuple[str, str], int | None]:
def test_every_row_gets_a_tier_in_1_2_3(db):
# Given: a mixed catalog covering every heuristic branch
_insert(db, [
{"model_id": "reasoner", "provider": "p", "supports_reasoning": True,
{"model_id": "reasoner", "provider": "p", "reasoning_default_enabled": True,
"cost_per_1m_completion": 5.0, "pricing_tbd": False},
{"model_id": "cheap", "provider": "p", "supports_reasoning": False,
{"model_id": "cheap", "provider": "p", "reasoning_default_enabled": False,
"cost_per_1m_completion": 0.25, "pricing_tbd": False},
{"model_id": "mid", "provider": "p", "supports_reasoning": False,
{"model_id": "mid", "provider": "p", "reasoning_default_enabled": False,
"cost_per_1m_completion": 5.0, "pricing_tbd": False},
{"model_id": "null-cost", "provider": "p", "supports_reasoning": False,
{"model_id": "null-cost", "provider": "p", "reasoning_default_enabled": False,
"cost_per_1m_completion": None, "pricing_tbd": False},
{"model_id": "tbd", "provider": "p", "supports_reasoning": False,
{"model_id": "tbd", "provider": "p", "reasoning_default_enabled": False,
"cost_per_1m_completion": 0.10, "pricing_tbd": True},
])
# When: applying the tiering pass
@@ -100,7 +108,7 @@ def test_every_row_gets_a_tier_in_1_2_3(db):
def test_override_map_row_matches_override(db):
# Given: a reasoning model (heuristic would say 3) overridden to tier 1
_insert(db, [
{"model_id": "deep-reasoner", "provider": "p", "supports_reasoning": True,
{"model_id": "deep-reasoner", "provider": "p", "reasoning_default_enabled": True,
"cost_per_1m_completion": 5.0, "pricing_tbd": False},
])
# When: applying tiering with an override map
@@ -114,11 +122,11 @@ def test_override_map_row_matches_override(db):
def test_heuristic_rows_match_expected_tiers(db):
# Given: reasoning -> 3, cheap non-reasoning -> 1, everything else -> 2
_insert(db, [
{"model_id": "reasoner", "provider": "p", "supports_reasoning": True,
{"model_id": "reasoner", "provider": "p", "reasoning_default_enabled": True,
"cost_per_1m_completion": 5.0, "pricing_tbd": False},
{"model_id": "cheap", "provider": "p", "supports_reasoning": False,
{"model_id": "cheap", "provider": "p", "reasoning_default_enabled": False,
"cost_per_1m_completion": 0.25, "pricing_tbd": False},
{"model_id": "mid", "provider": "p", "supports_reasoning": False,
{"model_id": "mid", "provider": "p", "reasoning_default_enabled": False,
"cost_per_1m_completion": 5.0, "pricing_tbd": False},
])
# When: applying the tiering pass
@@ -135,11 +143,11 @@ def test_heuristic_rows_match_expected_tiers(db):
def test_rerun_is_idempotent(db):
# Given: a mixed catalog
_insert(db, [
{"model_id": "reasoner", "provider": "p", "supports_reasoning": True,
{"model_id": "reasoner", "provider": "p", "reasoning_default_enabled": True,
"cost_per_1m_completion": 5.0, "pricing_tbd": False},
{"model_id": "cheap", "provider": "p", "supports_reasoning": False,
{"model_id": "cheap", "provider": "p", "reasoning_default_enabled": False,
"cost_per_1m_completion": 0.25, "pricing_tbd": False},
{"model_id": "mid", "provider": "p", "supports_reasoning": False,
{"model_id": "mid", "provider": "p", "reasoning_default_enabled": False,
"cost_per_1m_completion": 5.0, "pricing_tbd": False},
])
# When: applying the tiering pass twice
@@ -153,16 +161,16 @@ def test_rerun_is_idempotent(db):
# --- required test 5: sanity-guard warning path ---------------------------
def test_warns_when_supports_reasoning_uniformly_false(db, capsys):
# Given: a catalog where no row supports reasoning
def test_warns_when_reasoning_default_enabled_uniformly_false(db, capsys):
# Given: a catalog where no row reasons by default
_insert(db, [
{"model_id": "cheap", "provider": "p", "supports_reasoning": False,
{"model_id": "cheap", "provider": "p", "reasoning_default_enabled": False,
"cost_per_1m_completion": 0.25, "pricing_tbd": False},
{"model_id": "mid", "provider": "p", "supports_reasoning": False,
{"model_id": "mid", "provider": "p", "reasoning_default_enabled": False,
"cost_per_1m_completion": 5.0, "pricing_tbd": False},
])
# When: applying the tiering pass
apply_tiering(db, _make_config())
# Then: a clear warning is emitted (not a hard failure)
captured = capsys.readouterr()
assert "supports_reasoning is uniformly False" in captured.err
assert "reasoning_default_enabled is uniformly False" in captured.err

209
tests/test_eval_scoring.py Normal file
View File

@@ -0,0 +1,209 @@
"""Tests for eval_proficiency.py's scorers.
These decide what every proficiency number means, so the cases that matter
are the ones where a model did something *nearly* right — partial credit,
stray formatting, reasoning aloud before answering. Scoring those as zero
would make the whole table measure instruction-following rather than
competence.
No network: the code scorer really does execute a subprocess, which is the
point of testing it.
"""
import pytest
from eval_proficiency import (
extract_judge_json,
judge_for,
normalize_answer,
score_code,
score_exact,
score_tool,
strip_fences,
)
# --- code -----------------------------------------------------------------
def test_all_checks_passing_scores_one():
assert score_code("def f(x):\n return x * 2", ["f(2)==4", "f(0)==0"])[0] == 1.0
def test_partial_credit_is_proportional():
# A function right on one of two cases is genuinely better than one that
# fails both, and a binary score would throw that distinction away
score, detail = score_code(
"def f(x):\n return x * 2 if x else 99", ["f(2)==4", "f(0)==0"]
)
assert score == 0.5
assert "1/2" in detail
def test_code_that_does_not_parse_scores_zero_and_says_so():
score, detail = score_code("def f(x)\n return", ["f(1)==1"])
assert score == 0.0
# Distinguishable from "ran but failed the cases", which matters when
# reading results — one is a formatting problem, the other is competence
assert "did not run" in detail
def test_infinite_loop_is_bounded_not_hung():
score, detail = score_code("def f(x):\n while True: pass", ["f(1)==1"])
assert (score, detail) == (0.0, "timeout")
def test_markdown_fences_do_not_cost_the_model_the_task():
# Every code prompt says "no fences" and models add them anyway. Scoring
# that zero would measure instruction-following, not coding.
fenced = "```python\ndef f(x):\n return x * 2\n```"
assert score_code(fenced, ["f(2)==4"])[0] == 1.0
def test_raises_helper_is_available_to_checks():
# Checks express "should raise" through a helper the harness injects, so
# the sandbox needs no test framework of its own
code = "def g(n):\n if n < 1: raise ValueError()\n return n"
assert score_code(code, ["raises(ValueError, g, 0)", "g(3)==3"])[0] == 1.0
def test_common_imports_are_available():
# Ordinary solutions reach for re/math without importing them explicitly
code = "def f(s):\n return re.findall(r'\\d+', s)"
assert score_code(code, ["f('a1b22') == ['1','22']"])[0] == 1.0
def test_a_task_with_no_checks_cannot_score_credit():
assert score_code("def f(): pass", [])[0] == 0.0
# --- exact ----------------------------------------------------------------
def test_exact_ignores_surrounding_prose():
assert score_exact("The answer is 36.", "36")[0] == 1.0
def test_exact_ignores_thousands_separators():
assert score_exact("5,400", "5400")[0] == 1.0
def test_exact_takes_the_last_number_as_the_conclusion():
# A model that reasons aloud states intermediate values first; the answer
# is what it ends on
assert score_exact("1/60 + 1/90 = 1/36, so 36 minutes", "36")[0] == 1.0
def test_exact_wrong_answer_scores_zero_and_reports_both():
score, detail = score_exact("42", "36")
assert score == 0.0
assert "42" in detail and "36" in detail
def test_trailing_zeros_do_not_break_equality():
assert normalize_answer("36.0") == normalize_answer("36")
# --- tool use -------------------------------------------------------------
def _call(name, arguments="{}"):
return [{"function": {"name": name, "arguments": arguments}}]
def test_right_tool_and_arguments_scores_one():
calls = _call("get_weather", '{"location": "Reykjavik, Iceland"}')
task = {"expect_tool": "get_weather", "expect_args": {"location": "reykjavik"}}
assert score_tool(calls, task)[0] == 1.0
def test_right_tool_wrong_arguments_keeps_half_credit():
# Choosing correctly between tools is most of the skill; the arguments
# are the rest
calls = _call("get_weather", '{"location": "Paris"}')
task = {"expect_tool": "get_weather", "expect_args": {"location": "reykjavik"}}
assert score_tool(calls, task)[0] == 0.5
def test_wrong_tool_scores_zero():
task = {"expect_tool": "convert_currency"}
assert score_tool(_call("get_weather"), task)[0] == 0.0
def test_no_call_when_one_was_needed_scores_zero():
assert score_tool([], {"expect_tool": "get_weather"})[0] == 0.0
def test_abstaining_when_no_tool_applies_is_the_correct_answer():
# A model that reaches for a tool on every prompt is a real failure mode
# in an agent loop, so not calling one is scored, not just tolerated
assert score_tool([], {"expect_tool": None})[0] == 1.0
def test_calling_a_tool_when_none_applied_scores_zero():
score, detail = score_tool(_call("get_weather"), {"expect_tool": None})
assert score == 0.0
assert "when none applied" in detail
def test_unparseable_arguments_keep_the_tool_choice_credit():
calls = _call("get_weather", "{not json")
task = {"expect_tool": "get_weather", "expect_args": {"location": "x"}}
assert score_tool(calls, task)[0] == 0.5
# --- fence stripping ------------------------------------------------------
def test_strip_fences_leaves_bare_code_alone():
assert strip_fences("def f(): pass") == "def f(): pass"
def test_strip_fences_handles_an_unlabelled_block():
assert strip_fences("```\ndef f(): pass\n```").strip() == "def f(): pass"
# --- judge robustness -----------------------------------------------------
def test_judge_json_parses_a_bare_object():
assert extract_judge_json('{"score": 0.8, "reason": "ok"}')["score"] == 0.8
def test_judge_json_survives_a_thinking_preamble():
# Judges are reasoning models and leak their thinking into the content
# despite response_format. This was 44% of judge calls on the first run.
raw = 'Let me check the rubric:\n\n1. Distinguishes...\n\n{"score": 0.6, "reason": "partial"}'
assert extract_judge_json(raw)["score"] == 0.6
def test_truncated_judge_output_is_a_failure_not_a_guess():
# A cut-off object must not be salvaged into a number — better no sample
assert extract_judge_json('{"score": 0.8, "reason": "Accurate with') is None
def test_judge_prose_without_json_is_a_failure():
assert extract_judge_json("I think it is pretty good actually") is None
assert extract_judge_json("") is None
def test_object_without_a_score_is_not_a_verdict():
assert extract_judge_json('{"reason": "forgot the score"}') is None
def test_a_model_never_judges_its_own_family():
# Self-scoring is a known bias and the default judge is itself in the
# evaluated set, so its own family gets an alternate
assert judge_for("kimi-k3", "kimi-k3") != "kimi-k3"
assert judge_for("kimi-k3-fast", "kimi-k3") != "kimi-k3"
assert judge_for("kimi-k3-flex", "kimi-k3") != "kimi-k3"
# Everyone else is judged by the default
assert judge_for("gemma-4-31b", "kimi-k3") == "kimi-k3"
def test_a_single_leading_space_does_not_destroy_a_correct_answer():
# Observed: kimi-k2.7-code returns " def f(...)" with one leading space,
# which is an IndentationError once the harness prepends its imports. It
# scored 0.00 on every coding task for this and nothing else.
code = ' def f(x):\n return x * 2'
assert score_code(code, ["f(2)==4"])[0] == 1.0
def test_indented_fenced_block_is_still_recovered():
code = ' ```python\ndef f(x):\n return x * 2\n```'
assert score_code(code, ["f(2)==4"])[0] == 1.0

240
tests/test_feedback.py Normal file
View File

@@ -0,0 +1,240 @@
"""Tests for feedback.py — folding observed failures into proficiency.
The design decision under test is the asymmetry: failures are recorded,
passes are not. A structural 'ok' only means the code parsed, so treating it
as a quality sample would flood self_eval_score with 1.0s and wash out the
benchmark's discrimination — the coding categories already sit at 1.00 for
every model, and this would spread that flatness everywhere.
"""
import sqlite3
from pathlib import Path
import pytest
from config import load_config
from feedback import FAILURE_VERDICTS, apply_failures, summarize, unapplied_failures
ROOT = Path(__file__).resolve().parent.parent
SCHEMA_SQL = (ROOT / "schema.sql").read_text()
CFG = load_config(ROOT / "config.yaml")
@pytest.fixture
def db(tmp_path):
conn = sqlite3.connect(tmp_path / "t.db")
conn.row_factory = sqlite3.Row
conn.executescript(SCHEMA_SQL)
conn.execute(
"""
INSERT INTO models (model_id, provider, base_model_id, availability, last_updated)
VALUES ('m', 'nw', 'm', 'active', '2026-08-17T00:00:00+00:00')
"""
)
conn.commit()
yield conn
conn.close()
def _verify(conn, verdict, category="coding_general", kind="structural"):
conn.execute(
"""
INSERT INTO verifications (model_id, provider, task_category, kind, verdict, observed_at)
VALUES ('m', 'nw', ?, ?, ?, '2026-08-17T00:00:00+00:00')
""",
(category, kind, verdict),
)
conn.commit()
def _prof(conn):
return conn.execute(
"SELECT self_eval_score s, self_eval_samples n FROM proficiency WHERE model_id='m'"
).fetchone()
# --- only failures are folded in ------------------------------------------
def test_passes_are_not_recorded_as_samples(db):
# A structural 'ok' means "it parsed", not "it was correct". Recording it
# as a 1.0 would inflate every model toward the ceiling.
for _ in range(5):
_verify(db, "ok")
assert unapplied_failures(db) == []
def test_unverifiable_is_not_evidence(db):
# The checker had nothing to say; that says nothing about the model
for _ in range(5):
_verify(db, "unverifiable")
assert unapplied_failures(db) == []
@pytest.mark.parametrize("verdict", FAILURE_VERDICTS)
def test_failures_are_collected(db, verdict):
_verify(db, verdict)
assert len(unapplied_failures(db)) == 1
def test_a_failure_drags_the_score_down_proportionally(db):
from proficiency_store import add_self_eval
# Given: a benchmark score of 0.9 over 9 samples
add_self_eval(db, CFG, "m", "nw", "coding_general", [0.9] * 9)
_verify(db, "malformed")
apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False)
row = _prof(db)
# One 0.0 folded into the running mean: (0.9*9 + 0)/10
assert row["n"] == 10
assert row["s"] == pytest.approx(0.81)
def test_a_model_that_never_fails_keeps_its_benchmark_score(db):
from proficiency_store import add_self_eval
add_self_eval(db, CFG, "m", "nw", "coding_general", [0.9] * 9)
for _ in range(20):
_verify(db, "ok")
apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False)
row = _prof(db)
assert (row["n"], row["s"]) == (9, pytest.approx(0.9))
# --- idempotence ----------------------------------------------------------
def test_a_failure_is_applied_only_once(db):
_verify(db, "malformed")
apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False)
first = _prof(db)["n"]
# Re-running must not punish the model again for the same bad response
assert unapplied_failures(db) == []
apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False)
assert _prof(db)["n"] == first
def test_dry_run_changes_nothing(db):
_verify(db, "truncated")
apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=True)
assert _prof(db) is None
assert len(unapplied_failures(db)) == 1
# --- grouping -------------------------------------------------------------
def test_failures_group_by_model_and_category(db):
_verify(db, "malformed", category="coding_general")
_verify(db, "malformed", category="coding_general")
_verify(db, "truncated", category="debugging")
grouped = summarize(unapplied_failures(db))
assert {k[2]: len(v) for k, v in grouped.items()} == {
"coding_general": 2,
"debugging": 1,
}
def test_failures_without_a_category_are_skipped(db):
# Nothing to attribute them to — proficiency is per-category
db.execute(
"""
INSERT INTO verifications (model_id, provider, task_category, kind, verdict, observed_at)
VALUES ('m', 'nw', NULL, 'structural', 'malformed', '2026-08-17T00:00:00+00:00')
"""
)
db.commit()
assert unapplied_failures(db) == []
def test_both_check_kinds_count(db):
_verify(db, "malformed", kind="structural")
_verify(db, "malformed", kind="local_llm")
assert len(unapplied_failures(db)) == 2
# --- attribution: not every failure is the model's fault -------------------
def _verify_capped(conn, verdict="truncated", category="coding_general"):
conn.execute(
"""
INSERT INTO verifications (model_id, provider, task_category, kind, verdict,
observed_at, model_attributable)
VALUES ('m', 'nw', ?, 'structural', ?, '2026-08-17T00:00:00+00:00', 0)
""",
(category, verdict),
)
conn.commit()
def test_client_capped_truncation_is_not_the_models_fault(db):
# Found by forcing it: a request with max_tokens=40 truncates, and without
# this any agent using a tight cap would systematically drag down whatever
# model it routed to.
_verify_capped(db)
assert unapplied_failures(db) == []
def test_capped_failures_are_still_recorded_for_visibility(db):
# The response really was unusable — it just says nothing about the model
_verify_capped(db)
n = db.execute("SELECT COUNT(*) c FROM verifications WHERE verdict='truncated'").fetchone()["c"]
assert n == 1
def test_attributable_and_capped_failures_are_separated(db):
_verify(db, "malformed") # model's fault
_verify_capped(db) # client's cap
rows = unapplied_failures(db)
assert len(rows) == 1
assert rows[0]["verdict"] == "malformed"
# --- client outcomes: the only two-way evidence ----------------------------
def _outcome(conn, verdict, category="coding_general"):
conn.execute(
"""
INSERT INTO verifications (model_id, provider, task_category, kind, verdict, observed_at)
VALUES ('m', 'nw', ?, 'client_outcome', ?, '2026-08-17T00:00:00+00:00')
""",
(category, verdict),
)
conn.commit()
def test_a_client_reported_success_counts_as_a_positive_sample(db):
# Unlike a parser's 'ok' — which means the code parsed — this means the
# client ran it and the work worked. That is the only ground truth here.
from proficiency_store import add_self_eval
add_self_eval(db, CFG, "m", "nw", "coding_general", [0.5] * 3)
_outcome(db, "succeeded")
apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False)
row = _prof(db)
assert row["n"] == 4
assert row["s"] == pytest.approx((0.5 * 3 + 1.0) / 4)
def test_a_client_reported_failure_counts_against(db):
from proficiency_store import add_self_eval
add_self_eval(db, CFG, "m", "nw", "coding_general", [1.0] * 3)
_outcome(db, "failed")
apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False)
assert _prof(db)["s"] == pytest.approx(0.75)
def test_a_structural_pass_still_contributes_nothing(db):
# The asymmetry that matters: 'ok' from a parser is weak evidence and
# would inflate every score toward the ceiling
for _ in range(10):
_verify(db, "ok")
assert unapplied_failures(db) == []
def test_successes_and_failures_mix_into_one_rate(db):
for _ in range(3):
_outcome(db, "succeeded")
_outcome(db, "failed")
apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False)
row = _prof(db)
assert (row["n"], row["s"]) == (4, pytest.approx(0.75))

113
tests/test_iteration.py Normal file
View File

@@ -0,0 +1,113 @@
"""Tests for iteration.py — spending a tier's retry budget.
Two ideas under test. First, that a retry is matched to the failure: a
truncated answer needs a bigger budget, not a different model, and a malformed
one needs a different model, not a bigger budget. Second, that declining to
retry is a real outcome — every wasted attempt burns energy against a fixed
kWh quota.
"""
import pytest
from iteration import attempts_allowed, plan_retry
BY_TIER = {1: 0, 2: 1, 3: 2}
# --- how much iteration a tier buys ---------------------------------------
def test_tier_one_buys_no_retries():
# Cheap/simple work: one shot. Iterating on it costs more than it is worth.
assert attempts_allowed(1, "batch", BY_TIER, 1) == 0
def test_higher_tiers_buy_more_attempts():
assert attempts_allowed(2, "batch", BY_TIER, 5) == 1
assert attempts_allowed(3, "batch", BY_TIER, 5) == 2
def test_interactive_work_is_capped_below_its_tier_budget():
# Every retry doubles time-to-answer, and in interactive use latency IS a
# quality loss — a slow correct answer can be worth less than a fast one
# the user can judge themselves.
assert attempts_allowed(3, "interactive", BY_TIER, 1) == 1
assert attempts_allowed(3, "batch", BY_TIER, 1) == 2
def test_an_unknown_tier_buys_nothing():
assert attempts_allowed(9, "batch", BY_TIER, 5) == 0
# --- truncation: more budget, same model ----------------------------------
def test_truncation_retries_the_same_model_with_more_tokens():
# A different model would also run out. The budget is the problem.
plan = plan_retry("truncated", "qwen3.6-35b", 1000, 16384, [("kimi-k3", 65536)], False)
assert plan.model_id == "qwen3.6-35b"
assert plan.max_tokens == 2000
def test_truncation_retry_respects_the_model_ceiling():
plan = plan_retry("truncated", "gemma-4-31b", 10000, 16384, [], False)
assert plan.max_tokens == 16384
def test_no_retry_when_already_at_the_model_ceiling():
# Doubling would change nothing, and the attempt costs quota
assert plan_retry("truncated", "gemma-4-31b", 16384, 16384, [], False) is None
def test_a_client_chosen_cap_is_not_overridden():
# The caller explicitly asked for a short answer. Ignoring that would
# override an instruction, and they may well want it short.
assert plan_retry("truncated", "qwen3.6-35b", 40, 16384, [("kimi-k3", 65536)], True) is None
def test_no_cap_set_escalates_to_a_model_that_can_emit_more():
# The model's own output ceiling is the wall, so retrying IT changes
# nothing — but a roomier candidate might finish the answer. Without this
# the truncation branch was unreachable in practice: via /v1 the cap is
# either the client's (not ours to override) or absent.
plan = plan_retry("truncated", "gemma-4-31b", None, 16384, [("kimi-k3", 65536)], False)
assert plan.model_id == "kimi-k3"
def test_no_roomier_candidate_means_no_retry():
plan = plan_retry("truncated", "deepseek-v4-flash", None, 65536,
[("gemma-4-31b", 16384)], False)
assert plan is None
def test_candidates_with_unknown_ceilings_are_not_assumed_roomier():
# 11 of 19 catalog rows report no max_output_tokens; guessing they are
# bigger would waste the attempt
assert plan_retry("truncated", "gemma-4-31b", None, 16384,
[("qwen3.6-35b", None)], False) is None
# --- malformed: different model, same budget ------------------------------
def test_malformed_escalates_to_the_next_candidate():
# More tokens will not make unparseable output parse
plan = plan_retry("malformed", "qwen3.6-35b", 1000, 16384, [("kimi-k3", 65536), ("gemma-4-31b", 16384)], False)
assert plan.model_id == "kimi-k3"
assert plan.max_tokens == 1000
def test_malformed_with_no_alternative_declines():
assert plan_retry("malformed", "qwen3.6-35b", 1000, 16384, [], False) is None
def test_malformed_ignores_the_client_cap_question():
# The cap is irrelevant here — the answer did not parse, it was not cut off
plan = plan_retry("malformed", "qwen3.6-35b", 40, 16384, [("kimi-k3", 65536)], True)
assert plan.model_id == "kimi-k3"
# --- what must NOT buy a retry --------------------------------------------
@pytest.mark.parametrize("verdict", ["ok", "unverifiable"])
def test_non_failures_buy_no_retry(verdict):
# 'unverifiable' is most prose traffic. Retrying it would burn quota
# across the majority of requests for no signal at all.
assert plan_retry(verdict, "qwen3.6-35b", 1000, 16384, [("kimi-k3", 65536)], False) is None

172
tests/test_load_candidates.py Normal file
View File

@@ -0,0 +1,172 @@
"""Tests for dispatcher.load_candidates — how measured data reaches scoring.
Every behaviour here was found by a wrong routing decision rather than by
reasoning, so each gets explicit coverage:
- scoring reads the provider's ATTRIBUTED figures, because the attribution
ratio turned out to be a stable per-model property (750x between models,
~1.8x within one) rather than the noise it resembles up close
- the median, not the mean, because a few models still throw 50-90x outliers
- carbon reported as a static fallback is not data and must not rank anything
"""
import sqlite3
from pathlib import Path
import pytest
from dispatcher import (
FALLBACK_CARBON_SOURCE,
SEED_CATEGORY,
USD_PER_KWH,
gross_energy_kwh,
load_candidates,
)
ROOT = Path(__file__).resolve().parent.parent
SCHEMA_SQL = (ROOT / "schema.sql").read_text()
# 1000 W for 3.6 s = 1e-3 kWh, so the arithmetic below stays readable.
WATTS = 1000.0
SECONDS = 3.6
GROSS_KWH = 1e-3
@pytest.fixture
def db(tmp_path):
conn = sqlite3.connect(tmp_path / "test.db")
conn.row_factory = sqlite3.Row
conn.executescript(SCHEMA_SQL)
conn.execute(
"""
INSERT INTO models (model_id, provider, tier, availability, last_updated)
VALUES ('m', 'neuralwatt', 2, 'active', '2026-08-12T00:00:00+00:00')
"""
)
conn.commit()
yield conn
conn.close()
def _observe(conn, *, cost=1.0, carbon=37.0, source="agent_cache", category=SEED_CATEGORY):
conn.execute(
"""
INSERT INTO energy_observations (
model_id, provider, task_category, cost_usd,
carbon_g_co2eq, carbon_source, observed_at
) VALUES ('m', 'neuralwatt', ?, ?, ?, ?, '2026-08-12T00:00:00+00:00')
""",
(category, cost, carbon, source),
)
conn.commit()
def _only(conn):
return load_candidates(conn, "coding_general")[0]
# --- attribution is signal, not noise -------------------------------------
def test_gross_energy_identity_holds():
# The decomposition behind the finding: billed = power x duration x
# attribution. 1260.9 W x 1.523 s x 0.0158 = 8.43e-06 kWh, and the API
# reported 8.438e-06. gross_energy_kwh is a diagnostic on that identity.
assert gross_energy_kwh(1000.0, 3.6) == pytest.approx(1e-3)
assert gross_energy_kwh(1260.9, 1.523) * 0.0158 == pytest.approx(8.43e-06, rel=1e-2)
def test_scoring_reads_billed_cost_not_pre_attribution_energy(db):
# Given: a model whose billed cost is far below its share of pool gross,
# because many concurrent requests share its GPUs — deepseek-v4-flash
# bills ~1000x under gross. That discount is real money, so it must reach
# the score rather than being normalized away.
_observe(db, cost=1.154e-06)
_observe(db, cost=1.154e-06)
assert _only(db)["cost"] == pytest.approx(1.154e-06)
# --- median, not mean -----------------------------------------------------
def test_cost_uses_the_median_so_one_spike_cannot_dominate(db):
# Given: kimi-k3's real shape — ordinary samples plus a ~90x outlier
for c in (1.0, 1.0, 1.0, 1.0, 90.0):
_observe(db, cost=c)
assert _only(db)["cost"] == 1.0
def test_eco_uses_the_median_too(db):
for carbon in (1.0, 2.0, 3.0, 4.0, 500.0):
_observe(db, cost=1.0, carbon=carbon)
assert _only(db)["eco"] == 3.0
# --- fabricated carbon is not data ----------------------------------------
def test_static_fallback_carbon_is_excluded(db):
# Given: the glm-5.2-fast case. NeuralWatt could not resolve live grid
# data and substituted 475.0 while still reporting grid_id 'FI'.
for _ in range(3):
_observe(db, cost=1.0, carbon=8.5e-03, source=FALLBACK_CARBON_SOURCE)
row = _only(db)
# Then: no eco figure at all rather than a placeholder one. scoring turns
# that into a neutral 0.5, which is honest; ranking a model against a
# constant nobody measured would not be.
assert row["eco"] is None
# Cost is unaffected — it comes from the billing block, not the estimate.
assert row["cost"] == 1.0
def test_measured_carbon_survives_alongside_fallback_rows(db):
_observe(db, cost=1.0, carbon=100.0, source=FALLBACK_CARBON_SOURCE)
_observe(db, cost=1.0, carbon=2.0)
_observe(db, cost=1.0, carbon=4.0)
assert _only(db)["eco"] == 3.0
# --- scope ----------------------------------------------------------------
def test_only_reference_workload_observations_are_used(db):
# Given: organic traffic, whose energy tracks request shape far more than
# model efficiency (19x spread on one model), alongside the sweep
_observe(db, cost=1.0)
_observe(db, cost=1.0)
_observe(db, cost=999.0, category="coding_general")
assert _only(db)["cost"] == 1.0
def test_unswept_model_has_no_measurements(db):
row = _only(db)
assert row["cost"] is None
assert row["eco"] is None
assert row["samples"] == 0
# --- context estimation ---------------------------------------------------
def test_context_estimate_is_conservative_for_agent_traffic():
# Real measurement: opencode sent a prompt that tokenized to 106,158 while
# the chars/4 estimate put it under 94,196 — so it was routed to a model
# that could not hold it within its effective window. Agent traffic is
# code, JSON and tool schemas, which pack denser than prose.
from dispatcher import estimate_prompt_tokens
dense = "x" * 376_784 # the char count implied by that failure
# At the old divisor of 4 this estimated 94,196 and passed the filter.
assert estimate_prompt_tokens([{"role": "user", "content": dense}]) > 94_196
def test_context_estimate_counts_every_message():
from dispatcher import estimate_prompt_tokens
msgs = [{"role": "user", "content": "a" * 300}, {"role": "assistant", "content": "b" * 300}]
assert estimate_prompt_tokens(msgs) == 200
def test_context_estimate_reads_multimodal_text_parts():
from dispatcher import estimate_prompt_tokens
msgs = [{"role": "user", "content": [
{"type": "text", "text": "c" * 300},
{"type": "image_url", "image_url": {"url": "data:..."}},
]}]
assert estimate_prompt_tokens(msgs) == 100

107
tests/test_poller_parsing.py Normal file
View File

@@ -0,0 +1,107 @@
"""Tests for poller.py's pure catalog-parsing helpers.
The NeuralWatt catalog encodes serving class in the model id and access
gating only in prose, so both are parsed rather than read from a field.
Cases below are taken from the live catalog.
"""
import pytest
from poller import parse_access_level, parse_base_model_id, parse_serving_class
# --- serving class --------------------------------------------------------
@pytest.mark.parametrize(
"model_id,expected",
[
# base rows take the schema defaults
("gemma-4-31b", ("standard", "default", "full")),
("kimi-k3", ("standard", "default", "full")),
# one dimension at a time
("kimi-k3-fast", ("standard", "reduced", "full")),
("kimi-k3-flex", ("flex", "default", "full")),
("glm-5.2-short", ("standard", "default", "short")),
# the dimensions are orthogonal and combine freely
("glm-5.2-short-fast", ("standard", "reduced", "short")),
("glm-5.2-short-flex", ("flex", "default", "short")),
("glm-5.2-short-fast-flex", ("flex", "reduced", "short")),
],
)
def test_parse_serving_class(model_id, expected):
assert parse_serving_class(model_id) == expected
def test_flash_is_not_read_as_fast():
# Given: a base model whose name ends in 'flash' — a near-miss for the
# '-fast' suffix that a substring match would misclassify, wrongly
# demoting the cheapest model in the catalog out of tier 1
# Then: whole-segment matching leaves it a plain standard row
assert parse_serving_class("deepseek-v4-flash") == ("standard", "default", "full")
assert parse_serving_class("deepseek-v4-flash-flex") == ("flex", "default", "full")
def test_namespaced_and_mixed_case_ids():
# Given: the HF-style duplicate row, which carries a namespace and caps
assert parse_serving_class("deepseek-ai/DeepSeek-V4-Flash") == (
"standard",
"default",
"full",
)
def test_suffix_only_id_is_not_stripped_to_nothing():
# Given: a degenerate id consisting solely of a suffix token, the loop
# must leave at least one segment rather than consuming the whole id
assert parse_serving_class("fast") == ("standard", "default", "full")
# --- access level ---------------------------------------------------------
@pytest.mark.parametrize(
"display_name,description,expected",
[
("GLM-5.2 (short)", "... Private preview (grant-gated).", "preview"),
("GLM-5.2", "Private GLM-5.2 test canary", "canary"),
# gating can appear in the display name rather than the description
("DeepSeek V4 Flash 0731 (Canary)", "1M context window.", "canary"),
("Gemma 4 31B", "Google Gemma 4 31B — multimodal with tool calling.", "public"),
(None, None, "public"),
],
)
def test_parse_access_level(display_name, description, expected):
assert parse_access_level(display_name, description) == expected
def test_preview_takes_precedence_over_canary():
# Given: prose mentioning both, the more restrictive label wins
assert parse_access_level("x", "Private preview (grant-gated) canary") == "preview"
# --- model family ---------------------------------------------------------
@pytest.mark.parametrize(
"model_id,expected",
[
("gemma-4-31b", "gemma-4-31b"),
("kimi-k3", "kimi-k3"),
("kimi-k3-fast", "kimi-k3"),
("kimi-k3-flex", "kimi-k3"),
# every serving dimension collapses, in any combination
("glm-5.2-short-fast-flex", "glm-5.2"),
("glm-5.2-short", "glm-5.2"),
# the HF-style duplicate row folds into the same family
("deepseek-ai/DeepSeek-V4-Flash", "deepseek-v4-flash"),
# ...without 'flash' being mistaken for the '-fast' suffix
("deepseek-v4-flash", "deepseek-v4-flash"),
("deepseek-v4-flash-flex", "deepseek-v4-flash"),
],
)
def test_parse_base_model_id(model_id, expected):
assert parse_base_model_id(model_id) == expected
def test_family_never_collapses_to_empty():
# A degenerate id made only of suffix tokens must keep a segment, or every
# such row would share one meaningless family
assert parse_base_model_id("flex") == "flex"

179
tests/test_proficiency.py Normal file
View File

@@ -0,0 +1,179 @@
"""Tests for proficiency.py — blending leaderboard priors with self-eval.
The interesting cases are the ones the design doc's one-line rule does not
cover: what happens when a model has no leaderboard prior (common, because
NeuralWatt ships models faster than benchmarks cover them) and when a run
produces no usable scores at all.
"""
import pytest
from proficiency import accumulate, blend
BLEND_KW = {"leaderboard_weight": 0.3, "self_eval_weight": 0.7, "min_samples": 10}
def _blend(lb, se, n, **over):
return blend(lb, se, n, **{**BLEND_KW, **over})
# --- the documented rule --------------------------------------------------
def test_blends_both_sources_once_samples_suffice():
score, source = _blend(0.4, 0.9, 10)
assert score == pytest.approx(0.3 * 0.4 + 0.7 * 0.9)
assert source == "blended"
def test_leaderboard_alone_below_the_sample_threshold():
# Thin self-eval must not dominate a published benchmark early
score, source = _blend(0.4, 0.9, 9)
assert (score, source) == (0.4, "leaderboard")
def test_threshold_is_inclusive():
assert _blend(0.4, 0.9, 10)[1] == "blended"
assert _blend(0.4, 0.9, 9)[1] == "leaderboard"
# --- no leaderboard prior, which is the common case -----------------------
def test_self_eval_alone_when_no_prior_exists():
# Given: a model NeuralWatt added that no public benchmark covers yet.
# Read literally, the design doc's rule would fall back to a leaderboard
# score that does not exist and yield nothing — discarding real evidence.
score, source = _blend(None, 0.8, 10)
assert (score, source) == (0.8, "self_eval")
def test_thin_self_eval_still_beats_nothing():
# Nine real samples are worse than twelve, but far better than the
# neutral 0.5 a model gets for being unmeasured. Flagged so a caller can
# tell it apart from a score that cleared the threshold.
score, source = _blend(None, 0.8, 9)
assert (score, source) == (0.8, "self_eval_thin")
def test_unmeasured_model_returns_none_not_zero():
# None leaves the candidate on the neutral 0.5 downstream. Zero would
# rank it below every measured model for the crime of being new.
assert _blend(None, None, 0) == (None, None)
def test_zero_samples_is_not_evidence():
# A score with no samples behind it is a leftover, not a measurement
assert _blend(None, 0.9, 0) == (None, None)
assert _blend(0.4, 0.9, 0) == (0.4, "leaderboard")
def test_a_genuine_zero_score_is_kept():
# 0.0 means "measured, and it failed everything" — distinct from unmeasured
score, source = _blend(None, 0.0, 12)
assert (score, source) == (0.0, "self_eval")
# --- accumulation ---------------------------------------------------------
def test_first_run_sets_the_mean():
assert accumulate(None, 0, [1.0, 0.5, 0.0]) == (0.5, 3)
def test_later_runs_tighten_rather_than_replace():
# Given: 0.8 over 10 samples, then a run of 2 perfect scores
score, samples = accumulate(0.8, 10, [1.0, 1.0])
assert samples == 12
assert score == pytest.approx((0.8 * 10 + 2.0) / 12)
def test_an_all_errors_run_changes_nothing():
# Given: every task errored, so there are no scores. Resetting a model's
# history to zero on a bad run would silently erase months of evidence.
assert accumulate(0.8, 10, []) == (0.8, 10)
def test_no_history_and_no_scores_stays_empty():
assert accumulate(None, 0, []) == (None, 0)
def test_stale_score_with_zero_samples_is_overwritten():
# A score with no samples behind it carries no weight in the average
assert accumulate(0.9, 0, [0.1, 0.3]) == (pytest.approx(0.2), 2)
# --- variant inheritance --------------------------------------------------
def _models_db(tmp_path, rows):
import sqlite3
from pathlib import Path
schema = (Path(__file__).resolve().parent.parent / "schema.sql").read_text()
conn = sqlite3.connect(tmp_path / "t.db")
conn.row_factory = sqlite3.Row
conn.executescript(schema)
for model_id, base, latency, reasoning, ctx in rows:
conn.execute(
"""
INSERT INTO models (model_id, provider, base_model_id, latency_class,
reasoning_mode, context_variant, availability, last_updated)
VALUES (?, 'nw', ?, ?, ?, ?, 'active', '2026-08-17T00:00:00+00:00')
""",
(model_id, base, latency, reasoning, ctx),
)
conn.commit()
return conn
def test_flex_inherits_from_its_standard_equivalent(tmp_path):
from config import load_config
from pathlib import Path
from proficiency_store import add_self_eval, propagate_to_variants
cfg = load_config(Path(__file__).resolve().parent.parent / "config.yaml")
conn = _models_db(tmp_path, [
("kimi-k3", "kimi-k3", "standard", "default", "full"),
("kimi-k3-flex", "kimi-k3", "flex", "default", "full"),
])
add_self_eval(conn, cfg, "kimi-k3", "nw", "coding_general", [1.0, 1.0])
assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 1
got = conn.execute(
"SELECT self_eval_score FROM proficiency WHERE model_id='kimi-k3-flex'"
).fetchone()
assert got["self_eval_score"] == 1.0
def test_fast_does_not_inherit_reasoning_on_quality(tmp_path):
# A '-fast' row runs with reasoning off or capped, so it is NOT the same
# model for quality purposes. Inheriting across that would credit it with
# its reasoning-enabled sibling's answers.
from config import load_config
from pathlib import Path
from proficiency_store import add_self_eval, propagate_to_variants
cfg = load_config(Path(__file__).resolve().parent.parent / "config.yaml")
conn = _models_db(tmp_path, [
("kimi-k3", "kimi-k3", "standard", "default", "full"),
("kimi-k3-fast", "kimi-k3", "standard", "reduced", "full"),
])
add_self_eval(conn, cfg, "kimi-k3", "nw", "reasoning_math", [1.0])
assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 0
assert conn.execute(
"SELECT COUNT(*) c FROM proficiency WHERE model_id='kimi-k3-fast'"
).fetchone()["c"] == 0
def test_a_directly_measured_variant_is_never_overwritten(tmp_path):
from config import load_config
from pathlib import Path
from proficiency_store import add_self_eval, propagate_to_variants
cfg = load_config(Path(__file__).resolve().parent.parent / "config.yaml")
conn = _models_db(tmp_path, [
("kimi-k3", "kimi-k3", "standard", "default", "full"),
("kimi-k3-flex", "kimi-k3", "flex", "default", "full"),
])
add_self_eval(conn, cfg, "kimi-k3", "nw", "coding_general", [1.0])
add_self_eval(conn, cfg, "kimi-k3-flex", "nw", "coding_general", [0.2])
propagate_to_variants(conn, cfg, "kimi-k3", "nw")
got = conn.execute(
"SELECT self_eval_score FROM proficiency WHERE model_id='kimi-k3-flex'"
).fetchone()
assert got["self_eval_score"] == 0.2

235
tests/test_routing.py Normal file
View File

@@ -0,0 +1,235 @@
"""Tests for routing.py — hard filters and weighted ranking.
The filters are the part that disqualifies outright, so each one gets a case
proving it rejects and a case proving it does not over-reject.
"""
import pytest
from routing import is_eligible, rank_candidates, select_candidates
def _row(**overrides) -> dict:
"""A routable model row; override one field per test."""
row = {
"model_id": "m",
"provider": "neuralwatt",
"tier": 2,
"cost": 1.0,
"energy": 1.0e-5,
"effective_context_window": 100_000,
"availability": "active",
"deprecated": 0,
"access_level": "public",
"latency_class": "standard",
"reasoning_mode": "default",
"context_variant": "full",
}
row.update(overrides)
return row
def _eligible(row, **overrides) -> bool:
kwargs = {
"required_context_tokens": 10_000,
"required_tier": 2,
"latency_tolerance": "interactive",
"allowed_access_levels": ["public"],
"exclude_stale": True,
"exclude_deprecated": True,
}
kwargs.update(overrides)
return is_eligible(row, **kwargs)
# --- context window -------------------------------------------------------
def test_context_window_too_small_is_rejected():
assert _eligible(_row(effective_context_window=5_000)) is False
def test_context_window_exactly_equal_is_accepted():
# The filter is >=, so a model that exactly fits is still a candidate
assert _eligible(_row(effective_context_window=10_000)) is True
def test_null_context_window_is_rejected():
# Given: a row whose window never got derived. An unknown window cannot be
# shown to fit, and truncating mid-task is worse than routing elsewhere.
assert _eligible(_row(effective_context_window=None)) is False
# --- tier floor -----------------------------------------------------------
def test_tier_below_required_is_rejected():
assert _eligible(_row(tier=1), required_tier=2) is False
def test_tier_above_required_is_accepted():
# tier is a floor, not an equality match
assert _eligible(_row(tier=3), required_tier=2) is True
def test_null_tier_is_rejected():
assert _eligible(_row(tier=None)) is False
# --- freshness ------------------------------------------------------------
def test_stale_row_is_rejected_when_configured():
assert _eligible(_row(availability="stale")) is False
def test_stale_row_is_kept_when_not_excluded():
assert _eligible(_row(availability="stale"), exclude_stale=False) is True
def test_deprecated_row_is_rejected():
assert _eligible(_row(availability="deprecated")) is False
assert _eligible(_row(deprecated=1)) is False
# --- access gating --------------------------------------------------------
def test_grant_gated_rows_are_rejected_by_default():
# Given: the glm-5.2-short* rows, which are private preview. Routing to
# one earns a 403 at dispatch, so they are excluded before scoring.
assert _eligible(_row(access_level="preview")) is False
assert _eligible(_row(access_level="canary")) is False
def test_gated_rows_are_admitted_when_the_account_holds_the_grant():
assert (
_eligible(_row(access_level="preview"), allowed_access_levels=["public", "preview"])
is True
)
# --- latency tolerance ----------------------------------------------------
def test_flex_row_is_rejected_for_interactive_work():
# Given: a flex row, held server-side during peak until capacity frees up
assert _eligible(_row(latency_class="flex"), latency_tolerance="interactive") is False
def test_flex_row_is_admitted_for_batch_work():
assert _eligible(_row(latency_class="flex"), latency_tolerance="batch") is True
def test_standard_row_is_admitted_for_batch_work():
# Batch tolerates flex; it does not require it
assert _eligible(_row(latency_class="standard"), latency_tolerance="batch") is True
# --- select_candidates ----------------------------------------------------
def test_select_candidates_filters_and_preserves_order():
rows = [
_row(model_id="keep-1"),
_row(model_id="drop-flex", latency_class="flex"),
_row(model_id="keep-2", tier=3),
_row(model_id="drop-gated", access_level="preview"),
]
selected = select_candidates(
rows,
required_context_tokens=10_000,
required_tier=2,
latency_tolerance="interactive",
allowed_access_levels=["public"],
exclude_stale=True,
exclude_deprecated=True,
)
assert [r["model_id"] for r in selected] == ["keep-1", "keep-2"]
# --- measured cost --------------------------------------------------------
# --- ranking: quality first, cheapest among equals -------------------------
def test_a_real_quality_gap_decides_outright():
# tool_use_agentic spans 0.67 across the catalog; a gap that size must
# beat any cost saving
rows = [_row(model_id="cheap-bad", cost=1e-6, proficiency=0.33),
_row(model_id="dear-good", cost=1e-3, proficiency=1.00)]
assert rank_candidates(rows)[0]["model_id"] == "dear-good"
def test_within_tolerance_the_cheaper_model_wins():
# 0.02 apart on 2-3 samples is sampling noise, not a quality difference.
# Paying 100x for it would be buying noise.
rows = [_row(model_id="cheap", cost=1e-5, proficiency=0.98),
_row(model_id="dear", cost=1e-3, proficiency=1.00)]
assert rank_candidates(rows, quality_tolerance=0.1)[0]["model_id"] == "cheap"
def test_narrowing_the_tolerance_makes_small_gaps_count():
# As samples accumulate and confidence rises, the band should shrink
rows = [_row(model_id="cheap", cost=1e-5, proficiency=0.98),
_row(model_id="dear", cost=1e-3, proficiency=1.00)]
assert rank_candidates(rows, quality_tolerance=0.001)[0]["model_id"] == "dear"
def test_eco_no_longer_influences_the_decision():
# Carbon is still logged; it is simply not what this router optimizes
rows = [_row(model_id="clean", cost=1e-3, proficiency=0.5, eco=1e-6),
_row(model_id="dirty", cost=1e-5, proficiency=0.5, eco=1e9)]
assert rank_candidates(rows)[0]["model_id"] == "dirty"
def test_missing_proficiency_is_neutral_not_penalized():
ranked = rank_candidates([_row()])
assert ranked[0]["proficiency_score"] == 0.5
def test_unknown_cost_does_not_disqualify(db_free=None):
# A model never swept must stay pickable, or it can never acquire a
# measurement — the same trap the neutral-0.5 default avoids
ranked = rank_candidates([_row(model_id="unswept", cost=None)])
assert ranked[0]["model_id"] == "unswept"
def test_ties_break_deterministically():
rows = [_row(model_id="glm-b"), _row(model_id="glm-a"), _row(model_id="glm-c")]
first = [r["model_id"] for r in rank_candidates(rows)]
second = [r["model_id"] for r in rank_candidates(list(reversed(rows)))]
assert first == ["glm-a", "glm-b", "glm-c"] == second
def test_empty_candidate_set_ranks_to_empty():
assert rank_candidates([]) == []
# --- the budget ceiling: the quota mandate as a guarantee ------------------
def test_ceiling_excludes_models_over_budget():
# Denominated in kWh: the plan is a fixed quota, and a quota is a wall you
# hit mid-task rather than a bill that accrues
rows = [_row(model_id="affordable", energy=1e-6, proficiency=0.5),
_row(model_id="expensive", energy=1e-3, proficiency=1.0)]
ranked = rank_candidates(rows, max_energy_per_request=1e-4)
assert [r["model_id"] for r in ranked] == ["affordable"]
def test_ceiling_binds_even_against_the_best_model():
# This is the point of a constraint rather than a weight: no amount of
# quality buys past the ceiling
rows = [_row(model_id="expensive", energy=1e-3, proficiency=1.0)]
assert rank_candidates(rows, max_energy_per_request=1e-6) == []
def test_no_ceiling_admits_everything():
rows = [_row(model_id="expensive", energy=1e9, proficiency=1.0)]
assert len(rank_candidates(rows, max_energy_per_request=None)) == 1
def test_unmeasured_cost_is_admitted_under_a_ceiling():
# Excluding the unmeasured would mean a new model could never be picked
# and so could never acquire a measurement
rows = [_row(model_id="unswept", energy=None, proficiency=1.0)]
assert len(rank_candidates(rows, max_energy_per_request=1e-9)) == 1
def test_ceiling_at_exactly_the_cost_admits():
rows = [_row(model_id="borderline", energy=1e-6, proficiency=0.5)]
assert len(rank_candidates(rows, max_energy_per_request=1e-6)) == 1

2
tests/test_scoring.py
View File

@@ -1,4 +1,4 @@
"""Tests for router-package/scoring.py — pure scoring functions.
"""Tests for scoring.py — pure scoring functions.
Covers the required edge cases from the router-scoring-tiering plan:
cost min-max inversion (incl. $0-cheapest and None handling), eco

99
tests/test_session_identity.py Normal file
View File

@@ -0,0 +1,99 @@
"""Tests for session identity — telling concurrent clients apart.
The router has to attribute an outcome report to the right conversation
without the client cooperating. Getting this wrong is worse than getting no
report at all: a misattributed test failure penalizes a model for work it
never did, and this project has already recorded false failures twice from
harness bugs that took measurement to catch.
So the rule under test is: identify exactly when possible, refuse when not.
"""
from dispatcher import session_directory, session_fingerprint
SYS_A = "You are opencode. Project root: /home/alee/Sources/6krrt"
SYS_B = "You are opencode. Project root: /tmp/otherproj"
def _conv(system, *turns):
return [{"role": "system", "content": system}] + [
{"role": "user", "content": t} for t in turns
]
# --- fingerprints ---------------------------------------------------------
def test_two_sessions_are_distinguishable():
assert session_fingerprint(_conv(SYS_A)) != session_fingerprint(_conv(SYS_B))
def test_a_fingerprint_is_stable_as_a_session_grows():
# It must survive the conversation accumulating turns, or every message
# would look like a new session
short = session_fingerprint(_conv(SYS_A, "first"))
long = session_fingerprint(_conv(SYS_A, "first", "second", "third"))
assert short == long
def test_an_empty_conversation_has_no_fingerprint():
assert session_fingerprint([]) is None
assert session_fingerprint([{"role": "user", "content": " "}]) is None
def test_multimodal_content_still_fingerprints():
msgs = [{"role": "system", "content": [{"type": "text", "text": SYS_A}]}]
assert session_fingerprint(msgs) is not None
# --- directory extraction -------------------------------------------------
def test_a_single_mention_is_not_enough_to_claim_a_directory():
# One stray path proves nothing. Requiring corroboration keeps an
# unrelated path in a chat message from hijacking attribution.
assert session_directory(_conv("see /home/alee/Sources/6krrt once")) is None
def test_no_directory_when_the_prompt_names_none():
assert session_directory(_conv("You are a helpful assistant.")) is None
def test_only_the_opening_messages_are_scanned():
# A path mentioned deep in a long conversation is not the project root
msgs = _conv("You are an assistant.", "x", "y") + [
{"role": "user", "content": "check /tmp/some/other/path"}
]
assert session_directory(msgs) is None
# --- directory from the whole conversation --------------------------------
def test_directory_comes_from_the_files_the_agent_touches():
# opencode does not state its project root up front — verified by
# capturing a real request — but a coding agent names files inside its
# project constantly.
conv = [
{"role": "assistant", "content": "reading /tmp/proj/inventory.py"},
{"role": "user", "content": "ran /tmp/proj/test_inventory.py"},
{"role": "assistant", "content": "editing /tmp/proj/inventory.py"},
]
assert session_directory(conv) == "/tmp/proj"
def test_a_filename_is_never_mistaken_for_a_directory():
conv = [{"role": "assistant", "content": "/tmp/proj/only.py " * 3}]
assert session_directory(conv) == "/tmp/proj"
def test_the_shared_root_wins_over_any_one_subdirectory():
conv = [{"role": "assistant", "content":
"/home/alee/Sources/6krrt/routing.py "
"/home/alee/Sources/6krrt/tests/test_routing.py "
"/home/alee/Sources/6krrt/scoring.py"}]
assert session_directory(conv) == "/home/alee/Sources/6krrt"
def test_two_projects_stay_distinct():
a = [{"role": "assistant", "content": "/tmp/projA/x.py /tmp/projA/y.py /tmp/projA/z.py"}]
b = [{"role": "assistant", "content": "/tmp/projB/x.py /tmp/projB/y.py /tmp/projB/z.py"}]
assert session_directory(a) != session_directory(b)

232
tests/test_task_set.py Normal file
View File

@@ -0,0 +1,232 @@
"""Validate the eval task set against reference solutions.
A check the task author cannot satisfy is a broken check, and it scores the
task set rather than the model. That has already happened here: a sequence
task asked for "the next number" and "the 11th term" in one breath, and every
model that read it correctly scored zero.
So every `code` task gets a reference implementation below, and every check
must pass against it. Every `exact` answer is recomputed rather than trusted.
These tests run offline and take milliseconds — they are the cheap guard
against spending an hour of API calls measuring a typo.
"""
from pathlib import Path
import pytest
import yaml
from eval_proficiency import score_code, score_exact
TASKS = yaml.safe_load((Path(__file__).resolve().parent.parent / "evals" / "tasks.yaml").read_text())["tasks"]
BY_ID = {t["id"]: t for t in TASKS}
# --- reference solutions --------------------------------------------------
REFERENCES = {
"merge_intervals": '''
def merge_intervals(intervals):
if not intervals:
return []
ordered = sorted(intervals, key=lambda iv: iv[0])
out = [list(ordered[0])]
for start, end in ordered[1:]:
if start <= out[-1][1]:
out[-1][1] = max(out[-1][1], end)
else:
out.append([start, end])
return out
''',
"parse_semver": r'''
import re
_SEMVER = re.compile(
r"^(?P<major>0|[1-9]\d*)\.(?P<minor>0|[1-9]\d*)\.(?P<patch>0|[1-9]\d*)"
r"(?:-(?P<prerelease>(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)"
r"(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
r"(?:\+(?P<build>[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$"
)
def parse_semver(version):
m = _SEMVER.match(version or "")
if not m:
raise ValueError(version)
d = m.groupdict()
return {
"major": int(d["major"]),
"minor": int(d["minor"]),
"patch": int(d["patch"]),
"prerelease": d["prerelease"],
"build": d["build"],
}
''',
"word_wrap": '''
def word_wrap(text, width):
words = text.split()
if not words:
return []
lines, current = [], words[0]
for word in words[1:]:
if len(current) + 1 + len(word) <= width:
current += " " + word
else:
lines.append(current)
current = word
lines.append(current)
return lines
''',
"refactor_falsy_defaults": '''
_DEFAULTS = {"retries": 3, "timeout": 30, "verbose": False}
def apply_settings(overrides):
return {k: overrides.get(k, v) for k, v in _DEFAULTS.items()}
''',
"refactor_first_match": '''
def first_match(items, predicates):
return next(
(item for item in items if any(p(item) for p in predicates)), None
)
''',
"refactor_dispatch": '''
_TABLE = {200: "ok", 201: "created", 404: "not found", 500: "server error"}
def describe(code):
return _TABLE.get(code, "unknown")
''',
"debug_late_binding": '''
def make_multipliers(factors):
return [lambda x, f=f: x * f for f in factors]
''',
"debug_binary_search": '''
def bsearch(items, target):
lo, hi = 0, len(items)
while lo < hi:
mid = (lo + hi) // 2
if items[mid] == target:
return mid
elif items[mid] < target:
lo = mid + 1
else:
hi = mid
return -1
''',
"debug_greedy_regex": '''
import re
def extract_tags(text):
return re.findall(r"<([^<>]+)>", text)
''',
}
@pytest.mark.parametrize("task_id", sorted(REFERENCES))
def test_reference_solution_passes_every_check(task_id):
task = BY_ID[task_id]
score, detail = score_code(REFERENCES[task_id], task["checks"])
assert score == 1.0, f"{task_id}: reference scored {score} ({detail})"
def test_every_code_task_has_a_reference():
# A code task with no reference has never been validated, so its checks
# could be wrong in exactly the way that costs an hour of API calls
code_tasks = {t["id"] for t in TASKS if t["kind"] == "code"}
assert code_tasks == set(REFERENCES), (
f"unvalidated: {sorted(code_tasks - set(REFERENCES))}"
)
# --- the buggy code really is buggy ---------------------------------------
BUGGY = {
"refactor_first_match": '''
def first_match(items, predicates):
found = None
done = False
for item in items:
if done:
break
for p in predicates:
if p(item):
found = item
done = True
break
return found
''',
"debug_late_binding": '''
def make_multipliers(factors):
out = []
for f in factors:
out.append(lambda x: x * f)
return out
''',
"debug_greedy_regex": '''
import re
def extract_tags(text):
return re.findall(r"<(.+)>", text)
''',
}
def test_refactor_target_already_passes_its_own_checks():
# A refactor task's ORIGINAL code must pass, or the task is secretly a
# debugging task and "behaviour must not change" is a lie
score, detail = score_code(BUGGY["refactor_first_match"], BY_ID["refactor_first_match"]["checks"])
assert score == 1.0, f"refactor target fails its own checks: {detail}"
@pytest.mark.parametrize("task_id", ["debug_late_binding", "debug_greedy_regex"])
def test_debugging_tasks_start_broken(task_id):
# The whole point is that the given code fails. If it passes, the task
# measures nothing — a model could return the input unchanged.
score, _ = score_code(BUGGY[task_id], BY_ID[task_id]["checks"])
assert score < 1.0
# --- exact answers, recomputed --------------------------------------------
def test_percent_trap_answer():
# 20% up then 20% down is a 4% net loss, not a wash
original = 96 / (1.20 * 0.80)
assert score_exact(str(original), BY_ID["math_percent_trap"]["answer"])[0] == 1.0
def test_rate_trap_answer():
# 3 machines / 3 widgets / 3 min => one machine makes one widget in 3 min,
# so 100 machines make 100 widgets in the same 3 minutes
per_machine_minutes = 3
assert score_exact(str(per_machine_minutes), BY_ID["math_rate_trap"]["answer"])[0] == 1.0
def test_counting_answer():
# first digit 9 choices (not 0), then 9, 8, 7 for distinctness
assert score_exact(str(9 * 9 * 8 * 7), BY_ID["math_counting"]["answer"])[0] == 1.0
def test_counting_answer_matches_brute_force():
count = sum(
1 for n in range(1000, 10000) if len(set(str(n))) == 4
)
assert str(count) == BY_ID["math_counting"]["answer"]
# --- task set hygiene -----------------------------------------------------
def test_every_task_has_the_fields_its_kind_needs():
for task in TASKS:
kind = task["kind"]
assert task.get("prompt"), f"{task['id']} has no prompt"
if kind == "code":
assert task.get("checks"), f"{task['id']} has no checks"
elif kind == "exact":
assert task.get("answer") is not None, f"{task['id']} has no answer"
elif kind == "judge":
assert task.get("rubric"), f"{task['id']} has no rubric"
elif kind == "tool":
assert "expect_tool" in task, f"{task['id']} has no expect_tool"
assert task.get("tools"), f"{task['id']} has no tools"
def test_task_ids_are_unique():
ids = [t["id"] for t in TASKS]
assert len(ids) == len(set(ids))

178
tests/test_tiering.py
View File

@@ -1,35 +1,49 @@
"""Tests for router-package/tiering.py — pure tier resolver.
"""Tests for tiering.py — pure tier resolver.
Covers the required cases from the router-scoring-tiering plan:
override-wins, reasoning -> 3, cheap non-reasoning -> 1, and every
fallback path -> 2 (missing/NULL cost, pricing_tbd, cost >= threshold,
including the strict-< boundary at exactly the threshold).
Covers override-wins, effective-reasoning -> 3, cheap non-reasoning -> 1,
and every fallback path -> 2 (missing/NULL cost, pricing_tbd, cost >= the
threshold, including the strict-< boundary at exactly the threshold).
Also covers the two signals that replaced ``supports_reasoning``: a
``-fast`` row (``reasoning_mode='reduced'``) does not earn tier 3, and
``supports_reasoning`` is consulted only when ``reasoning_default_enabled``
is absent.
"""
from tiering import resolve_tier
from tiering import reasoning_effectively_on, resolve_tier
def _row(
*,
model_id: str = "m",
supports_reasoning: bool = False,
reasoning_default_enabled: bool = False,
reasoning_mode: str = "default",
cost_per_1m_completion: float | None = 1.0,
pricing_tbd: bool = False,
supports_reasoning: bool = False,
context_window: int | None = None,
) -> dict:
"""Build a minimal models-table row dict (ModelRow-shaped)."""
"""Build a minimal models-table row dict (ModelRow-shaped).
``context_window`` defaults to None — absent, not small — so the tier-1
context gate is inert unless a test opts in.
"""
return {
"model_id": model_id,
"supports_reasoning": supports_reasoning,
"reasoning_default_enabled": reasoning_default_enabled,
"reasoning_mode": reasoning_mode,
"cost_per_1m_completion": cost_per_1m_completion,
"pricing_tbd": pricing_tbd,
"supports_reasoning": supports_reasoning,
"context_window": context_window,
}
# --- rule 1: override wins ------------------------------------------------
def test_override_wins_over_heuristic():
# Given: a reasoning-capable model (heuristic would say 3) with an override to 1
row = _row(model_id="deep-reasoner", supports_reasoning=True)
# Given: a reasoning model (heuristic would say 3) with an override to 1
row = _row(model_id="deep-reasoner", reasoning_default_enabled=True)
override_map = {"deep-reasoner": 1}
# When: resolving the tier
tier = resolve_tier(row, cheap_completion_max=1.00, override_map=override_map)
@@ -40,8 +54,8 @@ def test_override_wins_over_heuristic():
def test_override_applies_to_all_providers_for_same_model_id():
# Given: the same model_id on two providers, override keyed by model_id only
override_map = {"shared-model": 2}
row_a = _row(model_id="shared-model", supports_reasoning=True)
row_b = _row(model_id="shared-model", supports_reasoning=True)
row_a = _row(model_id="shared-model", reasoning_default_enabled=True)
row_b = _row(model_id="shared-model", reasoning_default_enabled=True)
# When: resolving both rows
tier_a = resolve_tier(row_a, cheap_completion_max=1.00, override_map=override_map)
tier_b = resolve_tier(row_b, cheap_completion_max=1.00, override_map=override_map)
@@ -50,17 +64,67 @@ def test_override_applies_to_all_providers_for_same_model_id():
assert tier_b == 2
# --- rule 2: reasoning -> 3 ----------------------------------------------
# --- rule 2: effective reasoning -> 3 -------------------------------------
def test_reasoning_capable_tiers_to_3():
# Given: a reasoning-capable model with any cost
row = _row(model_id="deep-reasoner", supports_reasoning=True, cost_per_1m_completion=5.0)
def test_reasoning_default_enabled_tiers_to_3():
# Given: a model that reasons by default, with any cost
row = _row(
model_id="deep-reasoner",
reasoning_default_enabled=True,
cost_per_1m_completion=5.0,
)
# When: resolving the tier
tier = resolve_tier(row, cheap_completion_max=1.00, override_map={})
# Then: it is tier 3 (frontier / high-stakes)
assert tier == 3
def test_supports_reasoning_alone_does_not_tier_to_3():
# Given: a model that merely ACCEPTS a reasoning param but does not use it
# by default — true for all but two rows of the live catalog
row = _row(
model_id="accepts-param",
supports_reasoning=True,
reasoning_default_enabled=False,
cost_per_1m_completion=5.0,
)
# When: resolving the tier
tier = resolve_tier(row, cheap_completion_max=1.00, override_map={})
# Then: it is not promoted to tier 3 on capability alone
assert tier == 2
def test_fast_variant_does_not_inherit_tier_3():
# Given: a '-fast' row — same weights, thinking disabled or capped
row = _row(
model_id="glm-5.2-fast",
reasoning_default_enabled=True,
reasoning_mode="reduced",
cost_per_1m_completion=4.5,
)
# When: resolving the tier
tier = resolve_tier(row, cheap_completion_max=1.00, override_map={})
# Then: reduced reasoning drops it out of tier 3
assert tier == 2
def test_missing_default_enabled_falls_back_to_supports_reasoning():
# Given: a model exposing no reasoning block (the kimi-k2.7-code family),
# so reasoning_default_enabled is absent entirely
row = {
"model_id": "kimi-k2.7-code",
"reasoning_mode": "default",
"cost_per_1m_completion": 4.0,
"pricing_tbd": False,
"supports_reasoning": True,
}
# When: resolving the tier
tier = resolve_tier(row, cheap_completion_max=1.00, override_map={})
# Then: the capability flag is used as the fallback signal
assert reasoning_effectively_on(row) is True
assert tier == 3
# --- rule 3: cheap non-reasoning -> 1 -------------------------------------
def test_no_reasoning_cheap_cost_tiers_to_1():
@@ -72,6 +136,22 @@ def test_no_reasoning_cheap_cost_tiers_to_1():
assert tier == 1
def test_cheap_reasoning_capable_but_off_by_default_reaches_tier_1():
# Given: deepseek-v4-flash's real shape — accepts a reasoning param, does
# not reason by default, and costs $0.28/1M. The previous heuristic put
# this in tier 3 because it checked the capability flag before cost.
row = _row(
model_id="deepseek-v4-flash",
supports_reasoning=True,
reasoning_default_enabled=False,
cost_per_1m_completion=0.28,
)
# When: resolving the tier
tier = resolve_tier(row, cheap_completion_max=1.00, override_map={})
# Then: it lands in tier 1 where it belongs
assert tier == 1
# --- rule 4: fallback -> 2 ------------------------------------------------
def test_no_reasoning_cost_at_or_above_threshold_tiers_to_2():
@@ -108,3 +188,67 @@ def test_cost_exactly_at_threshold_tiers_to_2():
tier = resolve_tier(row, cheap_completion_max=1.00, override_map={})
# Then: tier 1 requires strictly-below, so it is tier 2
assert tier == 2
# --- cheapness is not a capability ceiling --------------------------------
def test_a_cheap_model_with_a_huge_window_is_not_tier_1():
# deepseek-v4-flash: $0.28/1M completion, 1M advertised window, 1.00 on all
# three coding categories. Tiering on price alone capped it at tier 1,
# which — because tier is a FLOOR — excluded it outright from every tier-2
# request. Being inexpensive is not evidence of being incapable.
row = _row(
model_id="deepseek-v4-flash",
reasoning_default_enabled=False,
cost_per_1m_completion=0.28,
context_window=1_048_560,
)
assert resolve_tier(row, 1.00, {}, 512_000) == 2
def test_a_cheap_model_with_a_small_window_stays_tier_1():
# gemma-4-31b: same rule, opposite side. 256K is genuinely a small model.
row = _row(
model_id="gemma-4-31b",
reasoning_default_enabled=False,
cost_per_1m_completion=0.42,
context_window=262_128,
)
assert resolve_tier(row, 1.00, {}, 512_000) == 1
def test_the_context_gate_is_inclusive_at_the_threshold():
row = _row(reasoning_default_enabled=False, cost_per_1m_completion=0.1,
context_window=512_000)
assert resolve_tier(row, 1.00, {}, 512_000) == 2
row["context_window"] = 511_999
assert resolve_tier(row, 1.00, {}, 512_000) == 1
def test_a_missing_window_does_not_block_tier_1():
# Absent capability evidence must not promote a model — a row with no
# advertised window falls through to the cost rule as before.
row = _row(reasoning_default_enabled=False, cost_per_1m_completion=0.1,
context_window=None)
assert resolve_tier(row, 1.00, {}, 512_000) == 1
def test_the_gate_defaults_to_off_for_callers_that_do_not_pass_it():
# Omitting the argument keeps the old cost-only behaviour rather than
# silently re-tiering a caller that has not opted in.
row = _row(reasoning_default_enabled=False, cost_per_1m_completion=0.28,
context_window=1_048_560)
assert resolve_tier(row, 1.00, {}) == 1
def test_a_huge_window_does_not_rescue_an_expensive_model_into_tier_1():
# The gate only ever demotes; it is not a second route into tier 1.
row = _row(reasoning_default_enabled=False, cost_per_1m_completion=15.0,
context_window=131_056)
assert resolve_tier(row, 1.00, {}, 512_000) == 2
def test_override_still_beats_the_context_gate():
row = _row(model_id="pinned", reasoning_default_enabled=False,
cost_per_1m_completion=0.28, context_window=1_048_560)
assert resolve_tier(row, 1.00, {"pinned": 1}, 512_000) == 1

299
tests/test_verification.py Normal file
View File

@@ -0,0 +1,299 @@
"""Tests for verification.py — structural checks on model responses.
The load-bearing distinction here is between "this is wrong" and "we could
not check this". Conflating them would penalize a model for the checker's
limits, which is the same mistake as scoring a judge malfunction against a
model — something this project has already done once.
Nothing here executes model output, and the tests assert that.
"""
import pytest
from verification import (
Block,
check_block,
check_json,
check_python,
check_yaml,
extract_blocks,
verify_response,
)
# --- truncation, the failure that looks like success ----------------------
def test_finish_reason_length_is_decisive():
# A truncated answer that happens to parse is the dangerous case: it looks
# complete to the client, which then acts on a fragment.
v = verify_response("```python\nx = 1\n```", finish_reason="length")
assert v.verdict == "truncated"
assert v.failed
def test_unterminated_fence_is_truncated_even_if_it_parses():
# "x = 1" is valid Python; the missing closing fence is the signal
v = verify_response("here you go:\n```python\nx = 1\n")
assert v.verdict == "truncated"
def test_complete_response_is_ok():
v = verify_response("```python\ndef f(x):\n return x * 2\n```")
assert v.verdict == "ok"
assert not v.failed
# --- malformed content ----------------------------------------------------
def test_broken_python_is_malformed_with_a_line_number():
v = verify_response("```python\ndef f(x)\n return x\n```")
assert v.verdict == "malformed"
assert "line" in v.detail
def test_broken_json_is_malformed():
assert check_json('{"a": 1,}').verdict == "malformed"
assert check_json('{"a": 1}').verdict == "ok"
def test_broken_yaml_is_malformed():
assert check_yaml("a: [1, 2\nb: 3").verdict == "malformed"
assert check_yaml("a: 1\nb: two").verdict == "ok"
def test_prose_left_inside_a_code_fence_is_caught():
# Models sometimes trail off into explanation without closing the fence
v = verify_response("```python\ndef f():\n return 1\nThis function returns one.\n```")
assert v.verdict == "malformed"
# --- unverifiable is not failure -----------------------------------------
def test_plain_prose_is_unverifiable_not_failed():
v = verify_response("A B-tree keeps itself balanced by splitting nodes.")
assert v.verdict == "unverifiable"
assert not v.failed
def test_unknown_language_is_unverifiable_not_failed():
v = verify_response("```rust\nfn main() { let x = ; }\n```")
assert v.verdict == "unverifiable"
assert not v.failed
def test_shell_is_deferred_to_the_caller_not_guessed():
# bash -n is a subprocess, so this module declines rather than pretending
c = check_block(Block(lang="bash", code="echo hi", closed=True))
assert c.verdict == "unverifiable"
def test_empty_block_is_unverifiable():
assert check_block(Block(lang="python", code=" \n", closed=True)).verdict == "unverifiable"
def test_response_with_no_blocks_at_all_is_unverifiable():
# Prose, not empty — an empty response is a failure, decided separately
assert verify_response("A B-tree splits nodes to stay balanced.").verdict == "unverifiable"
# --- multiple blocks ------------------------------------------------------
def test_one_bad_block_fails_the_response():
text = "```python\nx = 1\n```\nand\n```json\n{bad\n```"
v = verify_response(text)
assert v.verdict == "malformed"
assert len(v.checks) == 2
def test_truncation_outranks_malformation_in_reporting():
# If a response is both cut off and broken, the truncation is the cause
text = "```json\n{bad\n```\n```python\nx = (\n"
v = verify_response(text)
assert v.verdict == "truncated"
def test_all_good_blocks_pass():
text = "```python\nx = 1\n```\ntext\n```json\n{\"a\": 1}\n```"
assert verify_response(text).verdict == "ok"
def test_checkable_block_alongside_an_uncheckable_one_still_passes():
text = "```rust\nfn main() {}\n```\n```python\nx = 1\n```"
assert verify_response(text).verdict == "ok"
# --- extraction -----------------------------------------------------------
def test_extract_records_language_and_closure():
blocks = extract_blocks("```python\na\n```\n```\nb\n")
assert [(b.lang, b.closed) for b in blocks] == [("python", True), ("", False)]
def test_language_tag_is_case_insensitive():
assert verify_response("```PYTHON\nx = 1\n```").verdict == "ok"
# --- safety ---------------------------------------------------------------
def test_verification_never_executes_the_code_it_checks(tmp_path):
# If verification ran this, the file would exist. It must not.
canary = tmp_path / "canary.txt"
payload = f"```python\nopen({str(canary)!r}, 'w').write('executed')\n```"
v = verify_response(payload)
assert v.verdict == "ok" # it parses fine
assert not canary.exists() # ...and was never run
def test_a_syntactically_valid_destructive_snippet_is_only_parsed(tmp_path):
victim = tmp_path / "important.txt"
victim.write_text("still here")
payload = f"```python\nimport os\nos.remove({str(victim)!r})\n```"
verify_response(payload)
assert victim.read_text() == "still here"
# --- local LLM verification -----------------------------------------------
def test_local_verdict_survives_a_thinking_preamble():
from verification import parse_verdict_json
raw = 'Let me consider the answer...\n\n{"ok": false, "reason": "cut off mid-sentence"}'
assert parse_verdict_json(raw)["ok"] is False
def test_unusable_local_output_records_no_verdict():
# A checker that malfunctions must not produce a failure verdict — that
# charges the model for the checker's problem
from verification import interpret_local_verdict, parse_verdict_json
assert interpret_local_verdict(parse_verdict_json("I think it's fine")) is None
assert interpret_local_verdict(None) is None
def test_non_boolean_ok_is_rejected():
from verification import interpret_local_verdict
assert interpret_local_verdict({"ok": "yes"}) is None
assert interpret_local_verdict({"reason": "no verdict"}) is None
def test_local_verdict_maps_to_a_check():
from verification import interpret_local_verdict
assert interpret_local_verdict({"ok": True, "reason": "fine"}).verdict == "ok"
assert interpret_local_verdict({"ok": False, "reason": "empty"}).verdict == "malformed"
# --- the size gate is economics, not taste --------------------------------
def test_small_answers_are_not_worth_checking():
# A local check costs ~15% of a median 193-token answer, so it only pays
# if such answers fail more than ~15% of the time
from verification import worth_local_check
assert worth_local_check("unverifiable", 193, 600) is False
assert worth_local_check("unverifiable", 599, 600) is False
def test_large_answers_are_worth_checking():
from verification import worth_local_check
assert worth_local_check("unverifiable", 600, 600) is True
assert worth_local_check("unverifiable", 4000, 600) is True
def test_structurally_decided_responses_skip_the_fuzzy_check():
# If the code already parsed (or failed to), that verdict is exact and
# free; a fuzzy second opinion adds nothing and costs real time
from verification import worth_local_check
for verdict in ("ok", "malformed", "truncated"):
assert worth_local_check(verdict, 5000, 600) is False
def test_missing_token_count_does_not_trigger_a_check():
from verification import worth_local_check
assert worth_local_check("unverifiable", None, 600) is False
# --- excerpting must not manufacture truncation ---------------------------
def test_excerpt_leaves_short_text_alone():
from verification import excerpt
assert excerpt("short", 4000) == "short"
def test_excerpt_preserves_the_real_ending():
# The bug this guards: head-truncating a complete 6,637-char answer to
# 4,000 made it end mid-sentence, and the local checker correctly reported
# "cut off mid-thought" — a false failure created by the harness.
from verification import excerpt
text = "START" + ("x" * 10_000) + "THE ACTUAL ENDING."
out = excerpt(text, 4000)
assert out.startswith("START")
assert out.endswith("THE ACTUAL ENDING.")
assert "elided" in out
def test_excerpt_labels_the_cut_so_the_checker_knows():
from verification import excerpt
out = excerpt("a" * 10_000, 1000)
assert "characters elided" in out
# Budget is respected apart from the marker itself
assert len(out) < 1000 + 100
def test_prompt_carries_both_sides_with_the_real_ending():
from verification import build_local_verify_prompt
answer = "A" * 9000 + "FINAL SENTENCE."
prompt = build_local_verify_prompt("do a thing", answer, limit=2000)
assert "FINAL SENTENCE." in prompt
assert "do a thing" in prompt
def test_empty_response_is_decided_in_code_not_by_a_model():
# Asking the local 9.7B checker about an empty answer returned ok=true
# with the reason "Answer is too short" — a self-contradiction. This is
# settled by an if-statement instead.
assert verify_response("").verdict == "malformed"
assert verify_response(" \n\t ").verdict == "malformed"
assert verify_response("").failed
def test_whitespace_only_beats_the_no_blocks_path():
# It must not fall through to 'unverifiable' just for lacking code fences
assert verify_response("\n\n").verdict != "unverifiable"
# --- agent turns are not prose answers ------------------------------------
def test_a_tool_call_turn_is_not_a_failure():
# Real measurement: 29 of 63 completions in one agent session were
# recorded as "malformed: empty response" purely for making tool calls.
# Folding those in would have penalized the models that shipped a working
# feature with 349 passing tests.
v = verify_response("", has_tool_calls=True)
assert v.verdict == "unverifiable"
assert not v.failed
def test_a_half_sentence_before_a_tool_call_is_not_a_failure():
v = verify_response("Let me check the config", has_tool_calls=True)
assert not v.failed
def test_tool_call_outranks_even_truncation():
# An agent turn ending in a tool call routinely reports finish_reason
# length; that is the call boundary, not a cut-off answer
assert verify_response("x", "length", has_tool_calls=True).verdict == "unverifiable"
def test_a_genuinely_empty_answer_is_still_a_failure():
# The fix must not blind the checker to real emptiness
assert verify_response("", has_tool_calls=False).verdict == "malformed"
def test_broken_code_is_still_caught_when_a_tool_was_not_called():
assert verify_response("```python\ndef f(\n```").verdict == "malformed"
def test_the_local_checker_skips_tool_turns_too():
# Mirror-image blind spot: "cuts off mid-sentence" is exactly what a turn
# looks like when it ends by calling a tool. Observed producing 8 false
# failures in 9 checks on real agent traffic.
from verification import worth_local_check
assert worth_local_check("unverifiable", 5000, 600, has_tool_calls=True) is False
assert worth_local_check("unverifiable", 5000, 600, has_tool_calls=False) is True

36
tier.py
View File

@@ -22,10 +22,8 @@ import sys
from config import RouterConfig, load_config
from tiering import resolve_tier
DB_PATH = "router.db"
WARNING_UNIFORMLY_FALSE = (
"WARNING: supports_reasoning is uniformly False — verify poller field mapping"
"WARNING: reasoning_default_enabled is uniformly False — verify poller field mapping"
)
@@ -36,29 +34,43 @@ def apply_tiering(conn: sqlite3.Connection, config: RouterConfig) -> None:
``UPDATE models SET tier = ? WHERE model_id = ? AND provider = ?`` per
row, then commits. Only the ``tier`` column is touched.
Sanity guard (Metis finding #3): if ``supports_reasoning`` is True for
ZERO rows across the whole table, emit a warning to stderr — a uniformly
False field suggests a poller field-mapping bug that would silently zero
Sanity guard: if ``reasoning_default_enabled`` is True for ZERO rows
across the whole table, emit a warning to stderr — a uniformly False
field suggests a poller field-mapping bug that would silently zero
tier-3. This is a warning, not a hard failure.
"""
rows = conn.execute(
"SELECT model_id, provider, supports_reasoning, "
"cost_per_1m_completion, pricing_tbd FROM models"
"SELECT model_id, provider, supports_reasoning, reasoning_default_enabled, "
"reasoning_mode, cost_per_1m_completion, pricing_tbd, context_window "
"FROM models"
).fetchall()
if not any(row[2] for row in rows):
if not any(row[3] for row in rows):
print(WARNING_UNIFORMLY_FALSE, file=sys.stderr)
for model_id, provider, supports_reasoning, cost, pricing_tbd in rows:
for (
model_id,
provider,
supports_reasoning,
reasoning_default_enabled,
reasoning_mode,
cost,
pricing_tbd,
context_window,
) in rows:
tier = resolve_tier(
{
"model_id": model_id,
"supports_reasoning": bool(supports_reasoning),
"reasoning_default_enabled": bool(reasoning_default_enabled),
"reasoning_mode": reasoning_mode,
"cost_per_1m_completion": cost,
"pricing_tbd": bool(pricing_tbd),
"context_window": context_window,
},
config.tiering.cheap_completion_max,
config.tiering.model_tiers,
config.tiering.tier1_context_max,
)
conn.execute(
"UPDATE models SET tier = ? WHERE model_id = ? AND provider = ?",
@@ -68,9 +80,9 @@ def apply_tiering(conn: sqlite3.Connection, config: RouterConfig) -> None:
def main() -> int:
conn = sqlite3.connect(DB_PATH)
conn.execute("PRAGMA foreign_keys = ON")
cfg = load_config("config.yaml")
conn = sqlite3.connect(cfg.database.path)
conn.execute("PRAGMA foreign_keys = ON")
apply_tiering(conn, cfg)
conn.close()
print("tiering applied")

111
tiering.py
View File

@@ -3,41 +3,104 @@
This module is deliberately free of I/O: no DB reads, no file reads, no
config import at runtime. The model row and thresholds are passed as
arguments so the resolver stays testable and reusable. The DB upsert
(``apply_tiering``) is a separate concern (Todo 4) and lives elsewhere.
(``apply_tiering``) is a separate concern and lives in ``tier.py``.
Tier semantics (design doc §4 / config.yaml ``tiers`` labels):
1 = cheap / simple
2 = mid / general
3 = frontier / high-stakes
Why not ``supports_reasoning``: that column mirrors the catalog's
``capabilities.reasoning``, which only means "this endpoint accepts a
reasoning parameter". It is true for all but two rows in the NeuralWatt
catalog, so tiering on it collapsed 17 of 19 models into tier 3 and left
tier 1 empty. The discriminating signals are ``reasoning_default_enabled``
(whether the model actually thinks unless told otherwise) and
``reasoning_mode`` (whether this is a ``-fast`` row with thinking disabled
or capped).
Tier is a capability FLOOR: ``routing.py`` excludes any row whose
``tier < required_tier``, so tier 1 does not mean "cheap", it means "only
suitable for simple work". That distinction is what rule 3 below turns on.
Resolution precedence (EXACT order):
1. If ``override_map`` has this model's ``model_id`` -> that value wins.
2. ``supports_reasoning`` True -> 3.
3. ``cost_per_1m_completion`` is not None AND < ``cheap_completion_max``
-> 1.
4. Otherwise (missing/NULL completion cost, OR ``pricing_tbd`` True, OR
cost >= threshold) -> 2.
2. Reasoning is "effectively on" when ``reasoning_default_enabled`` is
True AND ``reasoning_mode`` is not ``'reduced'``. A ``-fast`` row is
the same weights served without chain-of-thought, so it does not earn
tier 3 on its sibling's behalf.
3. Reasoning effectively OFF, ``cost_per_1m_completion`` is not None and
not ``pricing_tbd`` and < ``cheap_completion_max``, AND
``context_window`` is below ``tier1_context_max`` -> 1.
4. Reasoning effectively ON -> 3.
5. Otherwise (missing/NULL cost, ``pricing_tbd``, cost >= threshold, or a
large context window) -> 2.
Rules 3 and 4 are mutually exclusive, so their relative order is immaterial;
cost is stated first because the previous version's reasoning-before-cost
ordering is what pushed $0.28/1M models into tier 3.
Why context window gates tier 1: price is a market signal, not a capability
measurement, and this project already learned that once in the cost domain —
list price ranks these models backwards. Tiering on price alone made the same
substitution and it capped ``deepseek-v4-flash`` at tier 1 on nothing but its
$0.28/1M completion price, which excluded it OUTRIGHT from every tier-2
request. It has a 1M advertised window and scores 1.00 on all three coding
categories in the eval set; "simple tasks only" is not a defensible reading of
that row. A model that can hold a million tokens of context is not a small
model however little it charges, so tier 1 now requires the model to be small
AND cheap rather than merely cheap.
``context_window`` (advertised) is used rather than
``effective_context_window`` because the advertised figure is the clean market
class — the catalog's values are 131056, 199984, 262128 and 1048560, i.e.
128K/200K/256K/1M — while the effective figures are provider-derived and vary
within a class (a 256K row reports 180212 or 192500 depending on family).
``tier1_context_max`` therefore sits in the wide empty band between the 256K
and 1M classes rather than being fitted to any one model.
NULL-cost semantics: tier 1 REQUIRES a non-NULL completion cost strictly
below the threshold. A strong-cheap non-reasoning model being mis-tiered to
1 is ACCEPTED — the override map is the escape hatch (Metis finding #9).
No second capability dimension is added to the heuristic.
below the threshold. A missing/None ``context_window`` does NOT block tier 1 —
absent capability evidence should not promote a model.
The override map remains the escape hatch for a row the heuristic still gets
wrong.
"""
from __future__ import annotations
def reasoning_effectively_on(model_row: dict) -> bool:
"""Whether this row actually reasons by default.
Falls back to ``supports_reasoning`` when ``reasoning_default_enabled``
is absent, matching the poller's own fallback for models that expose no
reasoning block (the kimi-k2.7-code family).
"""
default_enabled = model_row.get("reasoning_default_enabled")
if default_enabled is None:
default_enabled = model_row.get("supports_reasoning", False)
return bool(default_enabled) and model_row.get("reasoning_mode") != "reduced"
def resolve_tier(
model_row: dict,
cheap_completion_max: float,
override_map: dict[str, int],
tier1_context_max: float = float("inf"),
) -> int:
"""Resolve a model row to a tier in {1, 2, 3}.
``model_row`` is a row-like dict with at least these keys (from the
``models`` table / ``ModelRow``): ``model_id`` (str),
``supports_reasoning`` (bool), ``cost_per_1m_completion`` (float or
None), ``pricing_tbd`` (bool).
``reasoning_default_enabled`` (bool), ``reasoning_mode`` (str),
``cost_per_1m_completion`` (float or None), ``pricing_tbd`` (bool) and
``context_window`` (int or None, the ADVERTISED window).
``supports_reasoning`` is consulted only as a fallback.
``tier1_context_max`` defaults to infinity so an omitted argument keeps
the pre-existing cost-only behaviour rather than silently re-tiering a
caller that has not opted in.
``override_map`` is keyed by ``model_id`` only and therefore applies to
ALL providers serving that model (documented limitation vs the
@@ -48,15 +111,19 @@ def resolve_tier(
if model_id in override_map:
return override_map[model_id]
if model_row["supports_reasoning"]:
return 3
if not reasoning_effectively_on(model_row):
cost = model_row["cost_per_1m_completion"]
# A missing window is not evidence of a small model, so it does not
# block tier 1 — only a window we can see and that is large does.
window = model_row.get("context_window")
large_context = window is not None and window >= tier1_context_max
if (
not model_row["pricing_tbd"]
and cost is not None
and cost < cheap_completion_max
and not large_context
):
return 1
return 2
cost = model_row["cost_per_1m_completion"]
if (
not model_row["pricing_tbd"]
and cost is not None
and cost < cheap_completion_max
):
return 1
return 2
return 3

345
verification.py Normal file
View File

@@ -0,0 +1,345 @@
"""Pure verification of model responses.
Free, exact checks on what a model just returned, before the answer is
accepted. Like ``scoring.py`` and ``routing.py`` this module does no I/O and
makes no network calls, so it is testable in milliseconds and costs nothing
to run on every request.
**This module never executes model output.** ``eval_proficiency.py`` does run
generated code, but there the prompts are ones this project authored, so what
comes back is bounded. Here the code is whatever the user asked a model to
write — it could delete files, make requests, anything — and running it as a
side effect of *routing* would be indefensible. So the checks are structural:
parse it, don't run it. ``ast.parse`` builds a tree without evaluating,
``json.loads`` and ``yaml.safe_load`` never construct arbitrary objects, and
shell is checked with ``bash -n`` (parse-only) by the caller.
What this catches is the failure mode that has actually bitten this project
repeatedly: truncated and malformed output. A response cut off at the token
limit looks like a normal answer to a client and is silently broken.
Economics, measured on real traffic: a wasted cloud completion costs about 52
local checks at 1,500 tokens and 139 at 4,000, so a check that costs nothing
at all is worth running unconditionally.
"""
from __future__ import annotations
import ast
import json
import re
from dataclasses import dataclass, field
from typing import Literal, Optional
Verdict = Literal["ok", "truncated", "malformed", "unverifiable"]
# Languages whose syntax can be checked in-process without executing anything.
PYTHON_LANGS = frozenset({"python", "py", "python3"})
JSON_LANGS = frozenset({"json", "jsonc"})
YAML_LANGS = frozenset({"yaml", "yml"})
SHELL_LANGS = frozenset({"bash", "sh", "shell", "zsh"})
FENCE_RE = re.compile(
r"```[ \t]*([A-Za-z0-9_+-]*)[ \t]*\r?\n(.*?)(?:```|\Z)", re.DOTALL
)
@dataclass
class Block:
"""One fenced block from a response."""
lang: str
code: str
closed: bool
@dataclass
class Check:
"""The outcome of validating one block."""
lang: str
verdict: Verdict
detail: str = ""
@dataclass
class Verification:
"""What we can say about a whole response."""
verdict: Verdict
checks: list[Check] = field(default_factory=list)
detail: str = ""
@property
def failed(self) -> bool:
return self.verdict in ("truncated", "malformed")
def extract_blocks(text: str) -> list[Block]:
"""Pull fenced blocks out of a response.
An unterminated fence is kept and flagged rather than dropped — a response
that runs out of tokens mid-block leaves exactly that, and it is the
signal worth catching.
"""
blocks = []
for match in FENCE_RE.finditer(text or ""):
lang = (match.group(1) or "").lower()
body = match.group(2)
closed = match.group(0).rstrip().endswith("```")
blocks.append(Block(lang=lang, code=body, closed=closed))
return blocks
def check_python(code: str) -> Check:
"""Parse Python without running it.
``ast.parse`` builds a syntax tree and evaluates nothing, so this is safe
on arbitrary output. It catches truncation, unbalanced brackets, and the
stray prose that models sometimes leave inside a fence.
"""
try:
ast.parse(code)
except SyntaxError as e:
return Check("python", "malformed", f"line {e.lineno}: {e.msg}")
except (ValueError, MemoryError, RecursionError) as e:
return Check("python", "malformed", f"{type(e).__name__}: {e}")
return Check("python", "ok")
def check_json(code: str) -> Check:
try:
json.loads(code)
except json.JSONDecodeError as e:
return Check("json", "malformed", f"line {e.lineno}: {e.msg}")
return Check("json", "ok")
def check_yaml(code: str) -> Check:
# Imported lazily so the module stays dependency-free for callers that
# only need the Python and JSON checks.
try:
import yaml
except ImportError: # pragma: no cover
return Check("yaml", "unverifiable", "pyyaml not installed")
try:
yaml.safe_load(code)
except yaml.YAMLError as e:
return Check("yaml", "malformed", str(e).splitlines()[0][:120])
return Check("yaml", "ok")
def check_block(block: Block) -> Check:
"""Validate one block, or report that nothing structural applies.
An unclosed fence is reported as truncated regardless of language: the
content may happen to parse, but the response was cut off mid-thought and
the client is about to act on a fragment.
"""
if not block.closed:
return Check(block.lang or "?", "truncated", "unterminated code fence")
if not block.code.strip():
return Check(block.lang or "?", "unverifiable", "empty block")
if block.lang in PYTHON_LANGS:
return check_python(block.code)
if block.lang in JSON_LANGS:
return check_json(block.code)
if block.lang in YAML_LANGS:
return check_yaml(block.code)
if block.lang in SHELL_LANGS:
# Shell needs `bash -n`, which is a subprocess and therefore the
# caller's job; this module stays I/O-free.
return Check(block.lang, "unverifiable", "shell needs an external parse")
return Check(block.lang or "?", "unverifiable", "no structural check for this language")
def verify_response(
text: str,
finish_reason: Optional[str] = None,
has_tool_calls: bool = False,
) -> Verification:
"""Structurally verify a completion.
``has_tool_calls`` short-circuits everything below it: an agent turn that
calls a tool is not a prose answer and cannot be judged as one. It outranks
even truncation, because a turn routinely stops mid-sentence at the call
boundary and that is correct behaviour, not a budget overrun.
``finish_reason == 'length'`` is otherwise decisive on its own: the model
ran out of budget mid-answer, so whatever came back is a fragment even if
it happens to parse. That is checked before the content rules because a
truncated response that parses is the most dangerous case — it looks fine.
A response with nothing checkable returns ``unverifiable``, which is NOT a
failure. Most prose answers land here, and treating "we could not check
this" as "this is wrong" would penalize models for the checker's limits —
the same mistake as scoring a judge malfunction against a model.
"""
# A turn that calls a tool is not a prose answer and must not be judged as
# one. Its text is routinely empty, or a half-sentence before the call, and
# both are correct behaviour.
#
# This was not theoretical. On a real agent session, 29 of 63 completions
# were recorded as "malformed: empty response" purely for making tool
# calls, and folding those in would have penalized the models that shipped
# a working feature with 349 passing tests, clean mypy and clean ruff.
if has_tool_calls:
return Verification(
"unverifiable", detail="tool-call turn; no prose answer to check"
)
if finish_reason == "length":
return Verification("truncated", detail="finish_reason=length")
# Decided in code, not by a model. An empty answer is unambiguously a
# failure, and asking a 9.7B model about it produced a verdict of "ok"
# with the reason "Answer is too short" — self-contradictory. Never ask a
# model what an if-statement can settle.
if not (text or "").strip():
return Verification("malformed", detail="empty response")
blocks = extract_blocks(text)
if not blocks:
return Verification("unverifiable", detail="no fenced blocks in response")
checks = [check_block(b) for b in blocks]
for verdict in ("truncated", "malformed"):
bad = [c for c in checks if c.verdict == verdict]
if bad:
return Verification(
verdict,
checks=checks,
detail=f"{len(bad)}/{len(checks)} blocks: {bad[0].lang}: {bad[0].detail}",
)
if any(c.verdict == "ok" for c in checks):
return Verification("ok", checks=checks, detail=f"{len(checks)} block(s) parsed")
return Verification(
"unverifiable", checks=checks, detail="no block had a structural check"
)
# --- local LLM verification (for responses nothing structural can check) ---
VERDICT_JSON_RE = re.compile(r"\{.*\}", re.DOTALL)
LOCAL_VERIFY_SYSTEM = (
"You check whether an assistant's answer actually addresses the user's "
"request. Reply with ONLY a JSON object: "
'{"ok": true|false, "reason": "<at most 12 words>"}. '
"Answer false ONLY for a clear failure: the answer is empty, refuses "
"without cause, ENDS mid-sentence, contradicts itself, or responds to a "
"different question. Style, brevity and debatable choices are NOT "
"failures. When unsure, answer true.\n"
"IMPORTANT: a marker reading '[... N characters elided from the middle "
"...]' is this harness shortening a long answer so you can read it. It is "
"NOT a defect. Ignore it and judge only the beginning and the ending you "
"were given."
)
ELISION = "\n\n[... {n} characters elided from the middle ...]\n\n"
def excerpt(text: str, limit: int) -> str:
"""Shorten text for the checker WITHOUT making it look truncated.
Naive head-truncation is not safe here. A complete 6,637-character answer
cut to 4,000 reaches the checker ending mid-sentence, and it duly reports
"cut off mid-thought" — a false failure caused entirely by the harness.
That happened on the first live run, and had it been wired into
proficiency it would have recorded every long answer as a failure.
So the middle is dropped instead of the tail, and the cut is labelled, so
the ending the checker judges is the real ending.
"""
if not text or len(text) <= limit:
return text
head = limit // 2
tail = limit - head
dropped = len(text) - limit
return text[:head] + ELISION.format(n=dropped) + text[-tail:]
def build_local_verify_prompt(request_text: str, answer: str, limit: int = 4000) -> str:
"""Frame one answer for the local checker.
Both sides are shortened — the local model is small and slow, and a check
that reads 100k tokens costs more than the answer it guards — but via
``excerpt``, so an elision is never mistaken for a truncated answer.
"""
return (
f"USER'S REQUEST:\n{excerpt(request_text, limit)}\n\n"
f"ASSISTANT'S ANSWER:\n{excerpt(answer, limit)}"
)
def parse_verdict_json(raw: str) -> Optional[dict]:
"""Pull the verdict object out of a local model's reply.
Same defence as the eval judge: local models are reasoning models and leak
their thinking into the content despite response_format, so the object
usually arrives wrapped in prose. Returns None when nothing usable came
back, so the caller records no sample rather than a false verdict.
"""
if not raw:
return None
candidates = [raw]
match = VERDICT_JSON_RE.search(raw)
if match:
candidates.append(match.group(0))
for candidate in candidates:
try:
parsed = json.loads(candidate)
except json.JSONDecodeError:
continue
if isinstance(parsed, dict) and "ok" in parsed:
return parsed
return None
def interpret_local_verdict(parsed: Optional[dict]) -> Optional[Check]:
"""Turn a parsed local verdict into a Check, or None if unusable.
A malfunctioning checker must never produce a failure verdict: that would
charge the model for the checker's problem, which is the mistake this
project already made once with the eval judge.
"""
if parsed is None:
return None
ok = parsed.get("ok")
if not isinstance(ok, bool):
return None
reason = str(parsed.get("reason", ""))[:120]
return Check("local_llm", "ok" if ok else "malformed", reason)
def worth_local_check(
verdict: Verdict,
completion_tokens: Optional[int],
min_tokens: int,
has_tool_calls: bool = False,
) -> bool:
"""Whether a local LLM check earns its cost on this response.
Only for responses nothing structural could judge — if the code already
parsed, or failed to, that verdict is exact and free and a fuzzy second
opinion adds nothing.
The size gate is economics, measured on real traffic: a local check costs
about 15% of a median 193-token answer, so it pays only if such answers
fail more than ~15% of the time. On a 1,500-token answer the break-even
failure rate drops to ~1.9%, which is plausible. Small answers are simply
not worth checking.
"""
if verdict != "unverifiable":
return False
if has_tool_calls:
# Same reason the structural check declines: an agent turn ending in a
# tool call reads as "cuts off mid-sentence" to a checker expecting a
# finished answer. Observed producing 8 false failures in 9 checks on
# real agent traffic.
return False
return bool(completion_tokens and completion_tokens >= min_tokens)