alee/6krrt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

neuralwatt-router-service #2

Merged
alee merged 26 commits from neuralwatt-router-service into main 2026-08-23 16:20:48 +00:00
Conversation 0 Commits 26 Files Changed 32 +5193 -344
32 changed files with 5193 additions and 344 deletions
Download Patch File Download Diff File Expand all files Collapse all files

587
CLAUDE.md
View File

@@ -7,12 +7,17 @@ next steps, and is the one to trust on what is currently true.
## What this is ## What this is
A router that uses a local model (served via Ollama on an RTX 6000, 24GB) to A router that uses a local model (served via Ollama) to classify incoming
classify incoming coding/documentation tasks — category, tier, required coding/documentation tasks — category, tier, required context size — and
context size — and dispatch each task to the cheapest/best-fit open-weight dispatch each task to the cheapest/best-fit open-weight model on **Neuralwatt
model on **Neuralwatt Cloud**, weighted by cost, ecological impact Cloud**, weighted by cost, per-category proficiency, and a per-request energy
(Neuralwatt exposes real energy-per-request data), and per-category ceiling.
proficiency.
Every measurement in this file was taken on one deployment against one
provider account. They are recorded because the reasoning is worth more than
the conclusion, but treat them as observations with a date on them, not as
constants — the catalog, prices, grid intensity and pool load all move. When
a number here decides something, re-run the measurement before trusting it.
Neuralwatt is the only provider. OpenRouter was removed — the `provider` Neuralwatt is the only provider. OpenRouter was removed — the `provider`
column and the `(model_id, provider)` primary key stay so a second provider column and the `(model_id, provider)` primary key stay so a second provider
@@ -24,8 +29,9 @@ can be added later without a migration.
CPU-bound; iteration speed on the scoring/weighting logic matters more than CPU-bound; iteration speed on the scoring/weighting logic matters more than
raw execution speed at this scale) raw execution speed at this scale)
- SQLite for the decision table - SQLite for the decision table
- Ollama (OpenAI-compatible endpoint at `localhost:11434/v1`) for local - Ollama for local classification, via any OpenAI-compatible endpoint —
classification `localhost:11434/v1`, or an Ollama on another machine across a VPN
(`classifier.base_url`)
- FastAPI for the dispatcher service - FastAPI for the dispatcher service
## Billing is per-kWh, not per-token — and neither is what scoring uses ## Billing is per-kWh, not per-token — and neither is what scoring uses
@@ -85,8 +91,9 @@ is load-bearing, and `tests/test_routing.py` pins it.
for real traffic, because the reference workload is the wrong shape. for real traffic, because the reference workload is the wrong shape.
The sweep sends a 400-token prompt with a 400-token completion. Real agent The sweep sends a 400-token prompt with a 400-token completion. Real agent
traffic is a 150,000-token prompt with a ~400-token completion and 84% cache traffic is a 150,000-token prompt with a ~400-token completion and ~92% cache
hits. The attribution ratio moves with prompt size, so the ranking inverts: hits (token-weighted over 50 sessions and 40.7M tokens on 2026-08-23; the
figure was 84% when measured on 2.2M tokens of earlier traffic). The attribution ratio moves with prompt size, so the ranking inverts:
| workload | winner | | workload | winner |
|---|---| |---|---|
@@ -176,8 +183,16 @@ median over ALL `seed_reference` rows, so repeated sweeps accumulate into a
median-across-time for free — hence `llm-router-seed.timer`, which runs a median-across-time for free — hence `llm-router-seed.timer`, which runs a
small sweep every 6 hours. small sweep every 6 hours.
Until several sweeps have accumulated, treat the cost and eco ordering as Until several sweeps have accumulated, treat the eco ordering as provisional.
provisional. A single sweep's ranking is one sample of a moving quantity. A single sweep's ranking is one sample of a moving quantity.
**And none have accumulated since 6e729ad.** That commit moved
`log_observation`'s trailing arguments to keyword-only without updating
`seed_energy.py`, so every timer run since spent one billed completion and then
died on `TypeError` — which is not a `RequestException`, so the per-sample
`except` did not catch it. Fixed, and the sweep now has an offline end-to-end
test, but the accumulation this section describes starts from the next run
rather than from months of history.
## What's built and working ## What's built and working
@@ -215,7 +230,77 @@ provisional. A single sweep's ranking is one sample of a moving quantity.
`code` executes the model's Python against checks, `exact` compares a `code` executes the model's Python against checks, `exact` compares a
normalized answer, `tool` inspects the tool call structurally, and only the normalized answer, `tool` inspects the tool call structurally, and only the
four prose categories fall back to a `judge`. four prose categories fall back to a `judge`.
- `tests/` — 207 tests across 11 files, all passing. - `capabilities.py` — request-side capability detection. `detect_capabilities`
reads an OpenAI-format body and returns `RequestCapabilities` (`has_images`,
`require_json_mode`, `tools_present`, `has_reasoning_request`). It scans all
messages for `image_url` parts and reads `response_format.type` against
`{json_object, json_schema}`. Detection is read from the request body, not
inferred by a classifier.
- `logs.py` — the service's structured logging: a per-request trace id in a
ContextVar, logfmt rendering, and journald priority prefixes emitted only
when systemd actually owns stderr. `logs.bind()` exists because a
StreamingResponse's generator is resumed in a fresh copy of the caller's
context, so the ContextVar cannot reach it — the path all agent traffic
takes. Level from `logging.level`, overridden by `LLM_ROUTER_LOG_LEVEL`.
- `tests/` — 356 tests across 19 files, all passing, all offline. Verified on
Python 3.10 and 3.14; nothing declares `requires-python`, so 3.10 is the
tested floor rather than a promised one.
### Request-side capability gates
The router treats some capabilities as hard filters, read directly from the
request body. Two new gates live in `routing.rejection_reason` and
`routing.select_candidates`:
- `require_vision` — active when the request carries `image_url` parts and
`routing.require_vision` is true. A model passes only if its catalog row says
`supports_vision = 1`. `supports_vision = NULL` fails closed: an unknown flag
means the capability cannot be confirmed, and routing an image request to a
model that might lack vision is a guaranteed provider 400.
- `require_json_mode` — active when `response_format.type` is `json_object` or
`json_schema` and `routing.require_json_mode` is true. A model passes only if
`supports_json_mode = 1`; `NULL` also fails closed, for the same reason.
Both gates default to **on** in `config.yaml`, because a wrong guess produces a
400. This is a deliberate asymmetry against the tool-proficiency gate below:
capability **flags** fail closed on unknown, while quality **measurements**
(tool proficiency, energy) admit on absent evidence ("unproven, not bad").
`tools_present` remains a measurement gate, not a flag gate. Every routable
catalog row has `supports_tools = 1`, so that flag would be inert. What matters
is the measured `tool_use_agentic` proficiency. A model with no measured tool
score is admitted; a measured score below `routing.min_tool_proficiency` is
dropped — but only when the request carries a `tools` array. This gated filter
ships disabled (`null`).
`has_reasoning_request` is detected in `capabilities.py` but is informational
only. Reasoning stays on tiering: `reasoning_default_enabled` decides the tier,
not `supports_reasoning` (which only means "the endpoint accepts a reasoning
param" and is true for 17 of 19 rows). There is no reasoning capability gate.
### Pass-through capability check
When a client pins a real model id in `/v1/chat/completions`, the router
dispatches as asked. Before spending a provider call, `_check_pinned_capabilities`
reads the model's `supports_vision` and `supports_json_mode` flags and returns a
clear 422 if the pin cannot satisfy the request. The pin could never have worked,
so failing early is better than an opaque provider 400.
### Local Ollama vision fallback
`local_vision:` in `config.yaml` configures a fallback path for image requests
that find no cloud vision candidate. It ships **disabled** (`enabled: false`).
When enabled and routing returns no selected model, `_run_local_vision` sends
the original messages, with `image_url` parts intact, to a local Ollama model
(`qwen3-vl:4b` by default). The local answer then **replaces** the completion:
`_local_vision_response` returns a normal OpenAI-shaped response, including a
stream-wrapped version for `stream: true`. It does not inject a caption into a
cloud call, because the streaming proxy cannot rewrite bytes mid-stream.
Budget guards refuse images too numerous or too large before the local call is
made. If the local call fails for any reason, it falls through to the ordinary
`422 No model satisfies the hard filters` so the failure is visible rather than a
silent empty response.
### Serving class: one base model, many rows ### Serving class: one base model, many rows
@@ -279,47 +364,145 @@ and is still correctly absent from `tool_use_agentic`, where its measured 0.33
drops it out of the quality band. That is the eval data earning it the slot drops it out of the quality band. That is the eval data earning it the slot
rather than a thumb on the scale — no `model_tiers` override was needed. rather than a thumb on the scale — no `model_tiers` override was needed.
### Why tools and reasoning stay on their existing signals
Tools stay on the measured `tool_use_agentic` proficiency gate, not a
`supports_tools` flag gate. Every routable catalog row already has
`supports_tools = 1`, so a flag gate would be inert. The real signal is the
measured proficiency, because the observed failure is a model over-reaching for
tools on a non-agentic prompt. `routing.min_tool_proficiency` captures that
measurement and only applies when the request carries a `tools` array.
Reasoning stays on tiering (`reasoning_default_enabled`), not on a new flag gate.
`supports_reasoning` only means the endpoint accepts a reasoning parameter, and
that is true for 17 of 19 rows — nearly the whole catalog. `has_reasoning_request`
is detected purely for observation. Making it a gate would add no useful
filtering, because the decision of whether a request needs reasoning is already
encoded in the requested tier.
The fail-closed asymmetry, stated plainly: **capability flags fail closed on
unknown; quality measurements admit on absent evidence.** A missing
`supports_vision` or `supports_json_mode` flag means "cannot confirm", so the
model is dropped. A missing `tool_use_agentic` score or energy measurement means
"unproven, not bad", so the model is admitted. The first wrong guess is a
guaranteed 400; the second is just an empty data point that the neutral default
handles.
## Proficiency: category now changes routing ## Proficiency: category now changes routing
`proficiency_score` is the ONLY category-dependent term in the composite, so `proficiency_score` is the ONLY category-dependent term in the ranking, so
until this table had data, `task_category` could not change a decision at until this table had data, `task_category` could not change a decision at
all — the classifier computed it, the router paid ~10s for it, and then it all — the classifier computed it, the router paid ~10s for it, and then it
made no difference. It does now: made no difference. It does now: sweeping 9 categories x 3 tiers currently
returns **5 distinct winners** at both 50k and 120k of context.
| scoring | distinct models across 27 decisions (9 categories x 3 tiers) | | context | winners over 27 decisions |
|---|---| |---|---|
| catalog list price | 2 | | 50k | `qwen3.6-35b` (10), `gemma-4-31b` (7), `deepseek-v4-flash` (5), `kimi-k3` (3), `kimi-k3-fast` (2) |
| measured cost + eco | 3 | | 120k | `kimi-k2.7-code` (10), `gemma-4-31b` (7), `deepseek-v4-flash` (5), `kimi-k3` (3), `kimi-k3-fast` (2) |
| proficiency only (cost/eco data missing) | 7 |
| **all three axes live** | **1** |
That last row is not a regression, and it is worth understanding before **This spread is recent, and how it got here is the useful part.** For a long
tuning anything. With cost, eco and proficiency all populated, time all 27 decisions returned ONE model, and that was the correct answer at
`qwen3.6-35b` is **Pareto-dominant**: cheapest AND cleanest in the routable the time rather than a bug: with cost and eco both populated, `qwen3.6-35b`
set, while scoring within 0.15 of the best model on proficiency. On was Pareto-dominant — cheapest AND cleanest in the routable set, while
`summarization` it costs 7x less and emits 65x less carbon than `kimi-k3`, scoring within `quality_tolerance` of the best. No defensible weighting picks
which beats it 1.00 to 0.85 on quality — no defensible weighting picks anything else out of that.
`kimi-k3` from that. One model winning is the correct answer to the question
the weights ask.
Proficiency is still doing work underneath. The runner-up ordering reorders Two corrections widened it, and neither was a tuning change:
by category — `kimi-k3-fast` climbs to 3rd on `docs_writing` and
`summarization` (proficiency 1.00) and drops out of the top four elsewhere.
So the axis is live; it just cannot overturn a leader that wins on two axes
at once.
The 7-model spread appeared only while cost and eco were missing and scoring - **Cost stopped being a benchmark average.** It is now priced per request
neutral 0.5. If a genuinely different balance is wanted, the lever is from catalog prices scaled to the request's shape, so the ranking depends
`weights` in config.yaml, not more data. on the workload instead of on a 400-token reference sweep that no real
traffic resembles.
- **Tier stopped being inferred from price.** `deepseek-v4-flash` was pinned
to tier 1 for being cheap, which excluded it from every tier-2 request
regardless of what any score said.
Note what changes between the two rows above: only the leader, and only
because of the hard context filter. That is the filter working, not the
scoring disagreeing with itself.
**If you see one model win everything again, check for dominance before
reaching for config.** One winner is a legitimate outcome. The lever, if a
genuinely different balance is wanted, is `objective.quality_tolerance` (how
large a quality gap must be before it outranks a cost saving) or
`objective.max_energy_per_request` (a hard ceiling). There is no weight to
tune — quality is the objective and cost is the tiebreak.
### What the task set actually found ### What the task set actually found
**Coding does not discriminate these models.** All 13 rows score exactly **The benchmark could not discriminate these models on coding.** Every row
1.00 on `coding_general`, `coding_refactor` and `debugging` — and that is scored exactly 1.00 on `coding_general`, `coding_refactor` and `debugging` —
after the tasks were deliberately hardened with touching intervals, full and that is after the tasks were deliberately hardened with touching
semver, present-but-falsy defaults, late-binding closures and a binary search intervals, full semver, present-but-falsy defaults, late-binding closures and
that infinite-loops. Every model in this catalog is simply good at that class a binary search that infinite-loops. Every model in this catalog is simply
of problem, so cost and eco decide coding routes, which is the right outcome. good at that class of problem, so cost decides coding routes, which is the
right outcome.
**Real traffic broke one of those ties, which the benchmark never could.**
`coding_general` now spans 0.86-1.00: `glm-5.2-fast` fell to 0.862 over 29
samples folded in by `feedback.py` from an actual agent session, and crossed
`self_eval_min_samples` on the way, so it reads `self_eval` rather than
`self_eval_thin`. That is the intended shape of this system — the 23-task
benchmark establishes a floor, and your own traffic is what refines it.
`coding_refactor` and `debugging` are still flat at 1.00, awaiting the same
treatment.
**A 1.00 can also be a sampling artifact, and `docs_writing` was one.** At 2
samples per model the category read 0.70-1.00 with a model at the ceiling, and
the router paid for that ceiling: `kimi-k3-fast` won every docs route. Six more
benchmark passes moved every score and left NOTHING at 1.00:
| model | n=2 | n=11-14 |
|---|---|---|
| `kimi-k3` | 0.85 | **0.973** |
| `kimi-k2.7-code` | 0.85 | 0.886 |
| `deepseek-v4-flash` | 0.80 | 0.864 |
| `kimi-k3-fast` | **1.00** | 0.864 |
| `qwen3.6-35b` | 0.85 | 0.800 |
| `gemma-4-31b` | 0.85 | 0.786 |
The winner moved to `kimi-k2.7-code`, **3.2x cheaper** at 50k of context
($0.0441 -> $0.0136), with no config change — `kimi-k3` scores higher but sits
inside `quality_tolerance`, so cost breaks the tie. `deepseek-v4-flash`
($0.0024) misses the band by 0.009, which is the kind of margin the tolerance
exists to describe rather than a verdict.
**The whole spread rests on one rubric line, though.** `docs_function` is
effectively saturated — 1.00 on nine of every ten samples — and nearly every
`docs_gotcha` deduction is the same omission: the model documents that order is
preserved, that the first occurrence is kept, and what `key` does, then never
says elements must be hashable. That is real discrimination, since it is a real
property of the function, but one sentence is deciding a category. Treat this
ordering as thinner than n=14 makes it look.
**The self-judging guard costs sample density, and it shows up here.** Most
models reached n=14; `kimi-k3` and `kimi-k3-fast` reached only 11, because
those two are the ones diverted to the alternate judge `qwen3.6-35b`, which
returns unparseable JSON more often than `kimi-k3` does. The guard is still
right — a model grading its own family is worse than a thinner sample — but
the alternate judges should be picked for parseability, not just for being
someone else.
The reflex when a category looks flat is to reach for `quality_tolerance`.
Neither tie broken so far was broken that way: `coding_general` opened up when
`feedback.py` folded in real traffic, and `docs_writing` opened up on six more
benchmark passes. Both were samples, not settings. `coding_refactor` and
`debugging` are still flat at 1.00 on 2-3 samples each — which is now a state
this project has mistaken for a measurement once.
Current spread by category, widest first:
| category | spread |
|---|---|
| `tool_use_agentic` | 0.33 - 1.00 |
| `summarization` | 0.60 - 1.00 |
| `reasoning_math` | 0.67 - 1.00 |
| `docs_writing` | 0.66 - 0.97 |
| `general_chat` | 0.80 - 1.00 |
| `translation` | 0.85 - 1.00 |
| `coding_general` | 0.86 - 1.00 |
| `coding_refactor`, `debugging` | flat at 1.00 |
**What does discriminate is tool use, arithmetic traps, and prose.** **What does discriminate is tool use, arithmetic traps, and prose.**
`deepseek-v4-flash` scores 1.00 on all three coding categories yet **0.33 on `deepseek-v4-flash` scores 1.00 on all three coding categories yet **0.33 on
@@ -330,10 +513,42 @@ It over-reaches for tools, which is exactly the failure mode that matters in
an agent loop. The router now avoids it for those categories while still an agent loop. The router now avoids it for those categories while still
picking it for coding. picking it for coding.
Everything currently reads `source='self_eval_thin'`: real measurement, but Most rows still read `source='self_eval_thin'` (118 of 132): real
below `self_eval_min_samples` (2-3 tasks per category per run). Re-run measurement, but below `self_eval_min_samples` at 2-3 tasks per category per
`eval_proficiency.py` to accumulate — scores fold into a running mean rather run. The 14 that have crossed it are all `docs_writing`, from the six extra
than replacing, so samples add up across runs. passes above. Two paths thicken it, and they are complementary — re-run
`eval_proficiency.py` to accumulate benchmark samples, or just use the router
and let `feedback.py` fold in real outcomes. Both fold into a running mean
rather than replacing, so samples add up across runs.
### A score is only as fresh as the row it was copied to
Proficiency is a property of the weights, not the queue, so the eval harness
scores one row per family and `propagate_to_variants` copies the result onto
the serving variants — `kimi-k3-flex` gets `kimi-k3`'s number, because no
benchmark rates a `-flex` row separately.
That copy used to happen **exactly once per variant, ever.** The guard skipped
any row with `self_eval_samples > 0`, meaning "measured directly, do not
overwrite" — but inheritance copies the sample count too, so after the first
propagation an inherited row was indistinguishable from a measured one and was
never refreshed again. `kimi-k3-flex` sat at 0.85/n=2 while `kimi-k3` moved to
0.973/n=11.
`proficiency.inherited_from` records the provenance that was missing, and the
**migration** was the delicate half, not the fix: `ADD COLUMN` gives every
existing row NULL, which reads as "measured here", so shipping the guard alone
would have permanently frozen the exact rows it exists to unfreeze. The
backfill infers provenance from the harness's own selection rule rather than
guessing — `eval_identities` only ever evaluates standard rows plus flex rows
with **no** standard equivalent, so a flex row that has one was never a
candidate for direct evaluation, whatever its sample count claims. Everything
else keeps NULL, which fails safe: NULL means "do not overwrite", so no real
measurement can be lost to a wrong guess.
Confirmed on the live database, and on the catalog's one genuine exception —
`glm-5.2` is canary, so `glm-5.2-flex` is the routable row the harness scores
directly, and its NULL is correct.
### Harness bugs this shook out ### Harness bugs this shook out
@@ -361,6 +576,66 @@ fails. It immediately caught a check where the expected value was simply
wrong — which would have docked every model on a task and been wrong — which would have docked every model on a task and been
indistinguishable from genuine difficulty. indistinguishable from genuine difficulty.
## Tool competence is read from the request, not guessed at
Neither local classifier can identify agentic work. Asked to label six
unambiguous tool-use prompts ("read the config then update the manifest",
"run the tests and fix what fails"), `qwen3.5` got 2/6 and `mistral-nemo`
1/6 — and `mistral-nemo`'s misses collapse to `general_chat`, which is also
the configured `fallback_category`, so qwen3.5's crashes land in the same
place.
That mattered because `tool_use_agentic` has the widest proficiency spread in
the table (0.33-1.00) and `deepseek-v4-flash` — the current winner on coding —
sits at the bottom of it.
**The fix was not a better classifier.** Whether tools are on the table is
stated in the request: every agent client sends a `tools` array, and
`chat_completions` never looked at it. Reading it is exact and free.
It is applied as a **hard filter**, not a category override, and the
distinction is load-bearing. The question is not "is this task agentic" but
"can this model be trusted with tools that exist". The recorded failure is
precisely the second one: `deepseek-v4-flash` was given a *non*-agentic prompt
("it is 1:20pm and my meeting is at 3pm, how many minutes away?", both times
supplied) and called two tools rather than subtracting. A model that
over-reaches is a hazard on every request where tools are available, whatever
a classifier would have labelled the task.
So `routing.min_tool_proficiency` drops any candidate whose measured
`tool_use_agentic` score is below it, but only when the request carries tools:
| request | winner on `coding_general` @ 50k |
|---|---|
| no tools | `deepseek-v4-flash` ($0.0024) |
| tools present | `qwen3.6-35b` ($0.0041) |
Verified live through `/v1/chat/completions` with identical bodies differing
only by the `tools` array. The cost of safety here is 1.7x on that route,
paid only where tools exist.
**It is currently set to `null`, i.e. OFF**, deliberately and pending
experiment. opencode sends `tools` on essentially every request, so with the
filter on, `deepseek-v4-flash` is excluded from ordinary agent traffic and its
~7x cost advantage goes unused; with it off, that advantage applies and a
model measured at 0.33 on tool use handles requests where tools are on the
table. Which is right is an empirical question and the benchmark cannot
answer it — the 0.33 comes from 3 tasks.
What settles it is `POST /outcome`: run with the filter off, let real pass/fail
reports accumulate, and compare `deepseek-v4-flash`'s `tool_use_agentic`
proficiency before and after. That is the one signal here that knows whether
the work actually worked, and `feedback.py` folds client outcomes in both
directions, so success counts too.
0.5 sits in the empty band between the only two values the catalog holds
(0.33 and 1.00), so it is not fitted to either. A model with **no** measured
tool score is unproven rather than proven bad and is not dropped — the same
rule as the tier-1 context gate. Config load refuses a
`routing.tool_use_category` that is not a real category, because a name
matching nothing yields NULL for every row and NULL means "do not
disqualify": the filter would silently stop filtering.
## Tier is an iteration budget, not just a floor ## Tier is an iteration budget, not just a floor
A tier used to mean only "do not route below this". It now also buys A tier used to mean only "do not route below this". It now also buys
@@ -477,25 +752,54 @@ truncating the answer — are recorded but excluded.
## The classifier is the latency floor ## The classifier is the latency floor
Every routed request pays a full local classification round-trip before a Every routed request pays a full local classification round-trip before a
single upstream token is requested. Measured on `qwen3.5:latest`: single upstream token is requested, so the classifier model is the single
biggest lever on interactive latency.
| | latency | **The default is `mistral-nemo:12b`, and it replaced `qwen3.5:latest` on
|---|---| measurement.** Same prompts, same system prompt, `temperature: 0`, cold load
| cold (model not resident) | **43s+** — a reload can exceed even the 120s ceiling | excluded — 14 unambiguous category cases and 5 tier probes:
| warm, simple prompt | **~4s** |
| warm, prompt that triggers a long thinking trace | **up to ~25s** | | | `qwen3.5` | `mistral-nemo:12b` |
|---|---|---|
| category correct | 10/14 | 9/14 |
| tier correct | 1/5 (2 of them crashed) | **3/5** |
| **hard failures** | **4 of 19 calls (21%)** | **0** |
| latency mean / median / max | 6.6 / 5.5 / **15.6s** | 1.7 / 1.8 / **1.9s** |
Category accuracy is a wash. What decides it is the failure column and the
tail. Every one of those 4 failures is the runaway-thinking-trace mode below:
~15s spent to produce no JSON, which degrades to `source: "fallback"` — tier
2, `general_chat`. A reasoning model is the wrong tool for a job whose entire
output is ~45 tokens of JSON.
`mistral-nemo` does not reason by default, which is why its tail is flat: its
slowest call (1.94s) is faster than qwen3.5's median. End-to-end `/route`
went ~10s -> **~1.7s**. Both scored 5/5 on the local verification task, so
the verifier moved with it and only one model stays resident.
Four settings keep this usable, each fixing a failure seen in practice: Four settings keep this usable, each fixing a failure seen in practice:
- **`max_retries=0` on the classifier client.** The OpenAI SDK retries twice - **`max_retries=0` on the classifier client.** The OpenAI SDK retries twice
by default, so `timeout_seconds` silently became a 3x wall-clock bound — a by default, so `timeout_seconds` silently became a 3x wall-clock bound — a
request hung past 250s on a 120s setting and logged nothing. request hung past 250s on a 120s setting and logged nothing.
- **`max_output_tokens: 1024`.** qwen3.5 is a reasoning model and will - **`max_output_tokens: 1024`.** Bounds a REASONING model's chain of thought.
otherwise emit an unbounded chain of thought. That cascades: Ollama keeps Unbounded, it cascades: Ollama keeps generating after the client gives up
generating after the client gives up *and* serializes per model, so one *and* serializes per model, so one runaway request queues every later
runaway request queues every later request behind it and the timeouts request behind it and the timeouts spread. 256 was too tight — the trace
spread. 256 was too tight — the trace consumed the budget and the model was consumed the budget and the model was truncated before emitting any JSON,
truncated before emitting any JSON, which surfaced as an empty response. which surfaced as an empty response. Does not bind for `mistral-nemo`; kept
because it costs nothing unused and is the only guard if a reasoning model
is swapped back in.
- **`max_input_chars: 8000`.** The classifier decides a category and a tier;
it does not need the document. Feeding it one is harmful, not merely
wasteful — on a ~20k-token prompt `qwen3.5` spent 28.7s and returned empty
while `mistral-nemo` spent 41.8s echoing the input back inside its JSON.
Both land on `source: "fallback"`, i.e. 30-40s of local inference buying the
answer an instant failure would have given. Clamped to head + tail (the
instruction sits at one end or the other; never the middle), the same
prompts classify correctly in ~2.2s. Nothing is lost:
`chat_completions` measures the real conversation with
`estimate_prompt_tokens` and takes the larger value.
- **`fallback_tier` / `fallback_category`.** A classifier that times out, - **`fallback_tier` / `fallback_category`.** A classifier that times out,
errors, or returns garbage now degrades to a configured mid tier flagged errors, or returns garbage now degrades to a configured mid tier flagged
`source: "fallback"` instead of returning 502/503. The caller is a coding `source: "fallback"` instead of returning 502/503. The caller is a coding
@@ -506,10 +810,71 @@ Four settings keep this usable, each fixing a failure seen in practice:
then tier 1 on consecutive calls and routed to two different models. then tier 1 on consecutive calls and routed to two different models.
But ~10s of local overhead on every message is a real tax for an interactive But ~10s of local overhead on every message is a real tax for an interactive
agent, where the upstream answer itself may take 2s. Unaddressed options: agent, where the upstream answer itself may take 2s. Still unaddressed:
keep Ollama resident (`OLLAMA_KEEP_ALIVE`), classify once per session rather classify once per session rather than per message, cache by prompt hash, or
than per message, cache by prompt hash, use a smaller classifier, or skip skip classification for short prompts.
classification for short prompts. This is a design call, not a tuning one.
### "Local" means your hardware, not this machine
The premise is that a local LLM classifies the task before a cloud model
answers it. That does not require the GPU to be in the machine you are typing
on, and usually it isn't — most developers already have WireGuard or a VPN
back to a home lab. So the normal shape is router and editor on the laptop,
Ollama on the workstation, `classifier.base_url` pointing across the tunnel.
`classifier.base_url` / `api_key_env` / `model` take any OpenAI-compatible
endpoint. Verified end to end against a non-loopback address, classifier and
verifier both.
Ollama binds `127.0.0.1` by default, so this fails with connection-refused
until the serving host applies `deploy/ollama-over-vpn.conf`. Bind it to the
VPN address, not `0.0.0.0`: Ollama has no auth of any kind, so anything that
can reach the port can run inference and enumerate your models, and `0.0.0.0`
publishes it on whatever wifi the laptop is sitting on.
A cloud endpoint also satisfies that interface, and measured better than the
local one — worth knowing before assuming local is the cheap side. Five
prompts, same system prompt, `temperature: 0`:
| | local `qwen3.5` (RTX 6000) | NeuralWatt `deepseek-v4-flash` |
|---|---|---|
| mean latency | **11.58s** (4.95-15.76) | **1.02s** |
| categories agreed with the label | 2 of 4 | **5 of 5** |
| hard failures | 1 of 5 (empty after 15.76s) | 0 |
| energy per call | ~7e-05 kWh, on your meter | 1.17e-05 kWh attributed |
| cost per 1,000 calls | electricity + 6.6GB resident | **$0.093** (0.19% of quota) |
11x faster, more accurate, and less attributed energy, on a machine with a
24GB card sitting idle. The one hard failure is the documented
runaway-thinking-trace mode — `qwen3.5` spent 15.76s and emitted no JSON,
degrading to `source: "fallback"`, i.e. a silent tier-2 guess.
`deepseek-v4-flash` does not reason by default, so that mode does not exist
for it. End-to-end `/route` goes ~10s -> **~1.0s**.
Local classification was assumed to be the cheap option because local compute
felt free. It is not free, it is just unbilled — the same substitution the
cost axis had to unlearn. The shipped default is still local Ollama, because
switching spends quota and that is a deployment choice, not a code one.
### The verifier follows, but only to another Ollama
The local LLM check speaks Ollama's **native** `/api/chat` (the only way to
set `think: False`), so it follows the classifier across a VPN but not to a
cloud provider. It used to derive its URL by stripping `/v1` off
`classifier.base_url`, which meant moving the classifier at all would have
pointed it at `<that host>/api/chat`. It now has its own
`verification.base_url` and `verification.model`.
`verification.model` may be null only while both run on one host. Config load
**refuses** the null once the hostnames differ, because the failure is silent:
observed directly, with the classifier on NeuralWatt the verifier POSTed
`deepseek-v4-flash` to `localhost:11434`, 404'd, caught it, logged "local
verification unavailable" and recorded no sample. Verification would have
looked enabled while producing nothing.
Structural verification needs no model at all — it is pure Python — so
`verification.local_llm_enabled: false` leaves a host with no local inference
fully functional, minus the refusal/incoherence class of failure.
Note also that opencode sends **~32K prompt tokens** of system prompt and Note also that opencode sends **~32K prompt tokens** of system prompt and
tool definitions on a trivial request, so the measured-size floor in tool definitions on a trivial request, so the measured-size floor in
@@ -543,13 +908,25 @@ that request was two orders of magnitude low.
4. **Session-directory attribution picks the wrong directory.** The opencode 4. **Session-directory attribution picks the wrong directory.** The opencode
plugin now genuinely reports — 31 accepted, 3 refused as ambiguous, 22 plugin now genuinely reports — 31 accepted, 3 refused as ambiguous, 22
`succeeded` / 4 `failed` from real test runs, all folded in. But the `succeeded` / 4 `failed` from real test runs, all folded in. But the
"most frequent path" heuristic resolved the session to "most frequent path" heuristic resolved one session to a **dependency's
`.venv/lib/python3.14/site-packages/c2pa` 24 times versus source directory** inside the project's own virtualenv (24 mentions)
`~/Sources/fieldwitness` 22, because reading a dependency's source rather than the project being edited (22), because reading a library's
outweighed editing the project. It degrades safely — an ambiguous session source outweighed writing the code under test. It degrades safely — an
is refused rather than misattributed, which is why the 3 × 409 — but the ambiguous session is refused rather than misattributed, which is what the
heuristic needs to weight *writes* over reads, or anchor on the client's 3 × 409 were — but the heuristic needs to weight *writes* over reads, or
cwd instead of a path histogram. anchor on the client's cwd instead of a path histogram.
5. **Local energy is not on the ledger.** The router meters what NeuralWatt
bills and reports, but the electricity its own classifier and verifier burn
on local hardware is invisible to it. The whole premise is spending cheap
local compute to avoid wasting expensive cloud compute, and right now only
one side of that trade is measured — which is how "local is free" survived
as an assumption long enough to be wrong (a hosted classifier measured
faster, more accurate and lower attributed energy than the local one).
Closing it means metering local draw (nvidia-smi / RAPL / a smart plug) and
pricing it against a real tariff — utility rate data, ideally the user's own
plan, including time-of-use bands. That would also make the local-vs-cloud
comparison an actual number rather than a shrug.
## Known open questions ## Known open questions
@@ -563,19 +940,49 @@ that request was two orders of magnitude low.
- Three models still fail a split-half stability check at 7 samples. Is the - Three models still fail a split-half stability check at 7 samples. Is the
instability real (variable serving conditions) or an artifact of when the instability real (variable serving conditions) or an artifact of when the
sweep ran? Re-sweeping at a different hour would tell. sweep ran? Re-sweeping at a different hour would tell.
- The tier-1 composites sit within 0.009 of each other across a 40x cost - Answered, and the question no longer parses: tier-1 composites used to sit
range, because min-max compresses once one candidate is far cheaper than within 0.009 of each other because min-max normalization compressed them.
the rest. Ranking is right, but margins are thin — proficiency data will There is no composite any more — ranking is quality first, cost as the
swing these easily, which is the intent. tiebreak inside `quality_tolerance` — so nothing normalizes and nothing
compresses.
- Answered: the eval set exists (`evals/tasks.yaml`, 23 tasks, four scoring
kinds) and `tests/test_task_set.py` keeps it honest. The open part is
narrower now — `coding_refactor` and `debugging` are still flat at 1.00
across every model, so those tasks discriminate nothing and either need
hardening again or should be conceded as non-discriminating. **Try samples
before hardening.** `docs_writing` looked flat at the top too, and six more
passes spread it 0.66-0.97 without touching a task; two samples per model is
not enough to tell a saturated task from an unsampled one.
- How much context-assembly (RAG-style retrieval) belongs in the classifier - How much context-assembly (RAG-style retrieval) belongs in the classifier
step vs. a separate pre-step? Leaning decoupled, undecided. step vs. a separate pre-step? Leaning decoupled, undecided.
- Self-run eval set: minimum viable task set to start proficiency scoring
without it becoming its own maintenance burden?
- Should `eco_score` use real-time grid carbon intensity per request or a - Should `eco_score` use real-time grid carbon intensity per request or a
stable per-model average? Currently the latter, from the reference sweep. stable per-model average? Currently the latter, from the reference sweep.
`grid_carbon_intensity` and `grid_id` are logged per observation, so this `grid_carbon_intensity` and `grid_id` are logged per observation, so this
stays answerable from data without a re-run. stays answerable from data without a re-run.
## Config is strict: an unknown key is an error
Pydantic ignores extra keys by default, which means a typo or a misplaced
setting loads cleanly, does nothing, and still looks configured. Every config
model now inherits `StrictModel` (`extra="forbid"`), so both of these fail at
load rather than silently:
```
verification.max_input_chars # right key, wrong section
routing.min_tool_proficency # sic
```
This is not hypothetical. `max_input_chars` shipped into the `verification:`
block instead of `classifier:` and was accepted and discarded — it happened to
match the code default, so behaviour was correct and the file was a lie.
Editing it would have done nothing.
The corollary worth keeping: **every knob belongs in `config.yaml`, not only
in a Pydantic default.** A default the file never mentions is invisible to
anyone tuning it. `classifier.outcome_attribution_window_seconds` was removed
in the same pass — it was declared, never read, and shadowed the
`verification` one that actually is.
## Setup ## Setup
```bash ```bash
@@ -589,12 +996,22 @@ python config.py # sanity-check config loads
python -m uvicorn dispatcher:app --reload python -m uvicorn dispatcher:app --reload
``` ```
Ollama must be running locally with the classifier model pulled — the name Then set what is deployment-specific in `config.yaml`: `classifier.model` and
must match `classifier.model` in `config.yaml`: `classifier.base_url` for your Ollama, and `objective.plan_kwh_per_period` to
your own plan's quota (it is reported in `/health` as burn against the
allowance; it does not gate anything).
Ollama must be reachable with the classifier model pulled — the name must
match `classifier.model` in `config.yaml`:
```bash ```bash
ollama pull qwen3.5:latest ollama pull mistral-nemo:12b
``` ```
It does not have to be on this machine. To use one across a VPN, point
`classifier.base_url` and `verification.base_url` at it and apply
`deploy/ollama-over-vpn.conf` on the serving host — Ollama binds loopback-only
by default and will otherwise refuse.
`requirements.txt` is pinned. Recreating the venv with the old `>=` ranges `requirements.txt` is pinned. Recreating the venv with the old `>=` ranges
jumped openai 2.53 -> 3.0 and httpx -> httpx2 without warning; a service that jumped openai 2.53 -> 3.0 and httpx -> httpx2 without warning; a service that
restarts on boot shouldn't change its dependency tree underneath itself. restarts on boot shouldn't change its dependency tree underneath itself.
@@ -607,8 +1024,9 @@ resolves.
## Run as a service ## Run as a service
`deploy/` holds three systemd **user** units — see `deploy/README.md` for `deploy/` holds five systemd **user** units (dispatcher, plus a timer and a
install and operation. In short: oneshot service each for the poller and the seed sweep) and one drop-in for a
*system* Ollama — see `deploy/README.md` for install and operation. In short:
```bash ```bash
echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env
@@ -626,12 +1044,17 @@ The service holds a billable API key and has **no auth of its own**. Loopback
bind is the only thing standing between the open internet and your allowance; bind is the only thing standing between the open internet and your allowance;
add auth before widening `--host`. add auth before widening `--host`.
The same applies to an Ollama shared over a VPN — it has no auth either, so
`deploy/ollama-over-vpn.conf` binds it to the VPN address rather than
`0.0.0.0`, which would publish it on whatever network the client happens to
be on.
## Pointing a coding agent at it ## Pointing a coding agent at it
The `/v1` endpoints are OpenAI-compatible, so any normal client works — The `/v1` endpoints are OpenAI-compatible, so any normal client works —
opencode, an SDK, plain curl. Repo-local `opencode.json` is already wired up, opencode, an SDK, plain curl. Repo-local `opencode.json` is already wired up,
so `cd ~/Sources/6krrt && opencode` routes by default. For global use, merge so running `opencode` from a clone of this repo routes by default. For global
`provider.llm-router` into `~/.config/opencode/opencode.json`. use, merge `provider.llm-router` into `~/.config/opencode/opencode.json`.
| model name | behavior | | model name | behavior |
|---|---| |---|---|

377
README.md
View File

@@ -1,11 +1,21 @@
# Local LLM Model Router # Local LLM Model Router
A router that uses a local model (served via Ollama on an RTX 6000, 24GB) to A router that uses a local model (served via Ollama) to classify incoming
classify incoming coding/documentation tasks — category, tier, required coding/documentation tasks — category, tier, required context size — and
context size — and dispatch each task to the cheapest/best-fit open-weight dispatch each task to the cheapest/best-fit open-weight model on **Neuralwatt
model on **Neuralwatt Cloud**, weighted by cost, ecological impact Cloud**, weighted by cost, per-category proficiency, and a per-request energy
(Neuralwatt exposes real energy-per-request data), and per-category ceiling.
proficiency.
The local model does the classifying, so it wants a GPU, but not necessarily
*your* GPU — `classifier.base_url` takes any OpenAI-compatible endpoint, so
the usual shape is the router on your laptop and Ollama on a workstation
across a VPN. See [Setup](#setup).
**Numbers in this README are measurements, not specifications.** They come
from one deployment against one provider account, and the catalog, prices,
grid intensity and pool load all move. They are here because the reasoning
behind a design choice is worth more than the choice, and re-running the
measurement is how you check whether it still holds for you.
Neuralwatt is the only provider. The `provider` column and the Neuralwatt is the only provider. The `provider` column and the
`(model_id, provider)` primary key stay so a second provider can be added `(model_id, provider)` primary key stay so a second provider can be added
@@ -16,10 +26,10 @@ later without a migration.
| Dimension | Detail | | Dimension | Detail |
|---|---| |---|---|
| **Cost model** | Per-kWh, not per-token. Flat $8.00/kWh measured across the catalog. | | **Cost model** | Per-kWh, not per-token. Flat $8.00/kWh measured across the catalog. |
| **Latency** | ~4 s warm classification, up to ~120 s on cold start. Dispatcher response is unaffected — verification runs async. | | **Latency** | Classification is the floor: ~5-15 s warm on a local reasoning model, up to ~120 s cold. A cloud classifier measured ~1 s. Verification is async and never blocks the response. |
| **Quality** | Quality is the objective; cost is a per-request kWh ceiling plus a tiebreak. Eco is logged but no longer optimized. Proficiency blends external benchmarks with a self-run eval harness. | | **Quality** | Quality is the objective; cost is a per-request kWh ceiling plus a tiebreak. Eco is logged but no longer optimized. Proficiency blends external benchmarks with a self-run eval harness. |
| **Verification** | Every response structurally checked (free) + local LLM spot-check on long prose answers (~6 s, never blocks). Failures fold back into proficiency via `feedback.py`. | | **Verification** | Every response structurally checked (free) + local LLM spot-check on long prose answers (~6 s, never blocks). Failures fold back into proficiency via `feedback.py`. |
| **Fault tolerance** | Classifier failure degrades to a mid-tier fallback rather than 502/503. Retry count is zero to prevent silent 3× time-outs. | | **Fault tolerance** | Classifier failure degrades to a mid-tier fallback rather than 502/503. SDK retry count is zero, to prevent `timeout_seconds` silently becoming a 3× wall-clock bound. |
| **API surface** | OpenAI-compatible `/v1` endpoints, streaming chunk proxy with SSE telemetry scraping. | | **API surface** | OpenAI-compatible `/v1` endpoints, streaming chunk proxy with SSE telemetry scraping. |
## Verification Pipeline ## Verification Pipeline
@@ -109,7 +119,7 @@ Key behaviors:
``` ```
┌─────────────────────┐ ┌─────────────────────┐
incoming task ───▶│ Local Classifier │ Ollama (qwen3.5:latest) incoming task ───▶│ Local Classifier │ Ollama (mistral-nemo:12b)
│ - task_category │ ~4s warm, ~120s cold cap │ - task_category │ ~4s warm, ~120s cold cap
│ - task_tier │ temperature: 0, max 1024 tokens │ - task_tier │ temperature: 0, max 1024 tokens
│ - required_context │ max_retries: 0 (silent 3× cap guard) │ - required_context │ max_retries: 0 (silent 3× cap guard)
@@ -163,19 +173,19 @@ Key behaviors:
| Layer | Technology | | Layer | Technology |
|---|---| |---|---|
| **Language** | Python 3 (IO-bound provider APIs; iteration speed matters more than raw speed) | | **Language** | Python 3.10+ (IO-bound provider APIs; iteration speed matters more than raw speed) |
| **Framework** | FastAPI + uvicorn | | **Framework** | FastAPI + uvicorn |
| **Database** | SQLite (`router.db`) — decision table, energy observations, proficiency, verifications | | **Database** | SQLite (`router.db`) — decision table, energy observations, proficiency, verifications |
| **Local Classification** | Ollama (OpenAI-compatible at `localhost:11434/v1`) | | **Local Classification** | Ollama, OpenAI-compatible — `localhost:11434/v1` or an Ollama across your VPN |
| **Local Model** | `qwen3.5:latest` | | **Local Model** | `classifier.model` — `mistral-nemo:12b` by default; any Ollama model works |
| **Cloud Provider** | Neuralwatt only | | **Cloud Provider** | Neuralwatt only |
| **Config** | `config.yaml` loaded & validated by Pydantic (`config.py`) | | **Config** | `config.yaml` loaded & validated by Pydantic (`config.py`) |
| **OpenAI Client** | `openai==3.0.0` (official SDK) | | **OpenAI Client** | `openai==3.0.0` (official SDK) |
| **HTTP** | `requests` for poller, `httpx` (via openai/uvicorn) | | **HTTP** | `requests` for poller, `httpx` (via openai/uvicorn) |
| **Testing** | `pytest` — 207 tests across 11 files | | **Testing** | `pytest` — 356 tests across 19 files, all offline |
| **Config Files** | `config.yaml`, `leaderboards.yaml`, `evals/tasks.yaml` | | **Config Files** | `config.yaml`, `leaderboards.yaml`, `evals/tasks.yaml` |
| **Deployment** | systemd user units (`.service` + `.timer` files in `deploy/`) | | **Deployment** | systemd user units (`.service` + `.timer` files in `deploy/`) |
| **Integration** | Pre-configured in `opencode.json` — `cd` into repo uses router by default | | **Integration** | `opencode.json` in the repo routes through it by default; any OpenAI-compatible client works |
## Pinned Dependencies (requirements.txt) ## Pinned Dependencies (requirements.txt)
@@ -212,7 +222,8 @@ restarts on boot shouldn't change its dependency tree underneath itself.
| **`verification.py`** | Two-layer check: structural parse (always) + local LLM spot-check (async, size-gated). Never executes model output | Pure | | **`verification.py`** | Two-layer check: structural parse (always) + local LLM spot-check (async, size-gated). Never executes model output | Pure |
| **`feedback.py`** | Folds observed verification failures into `proficiency`; routing learns from real traffic | Yes (DB) | | **`feedback.py`** | Folds observed verification failures into `proficiency`; routing learns from real traffic | Yes (DB) |
| **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) | | **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) |
| **`config.py`** | YAML loader + Pydantic validators (weights sum to 1, valid tiers, etc.) | Yes (file) | | **`iteration.py`** | Retry budget per tier, and matching the retry to the failure kind | Pure |
| **`config.py`** | YAML loader + Pydantic validators (blend weights sum to 1, valid tiers, endpoints separately addressable) | Yes (file) |
## Decision Table Schema (SQLite) ## Decision Table Schema (SQLite)
@@ -270,6 +281,7 @@ parses them into `access_level` and `routing.allowed_access_levels` (default
| `self_eval_samples` | INTEGER | Evidence count for blending threshold | | `self_eval_samples` | INTEGER | Evidence count for blending threshold |
| `blended_score` | REAL | Computed: `w_lb × lb + w_se × se` | | `blended_score` | REAL | Computed: `w_lb × lb + w_se × se` |
| `source` | TEXT | `blended` \| `self_eval` \| `self_eval_thin` \| `leaderboard` | | `source` | TEXT | `blended` \| `self_eval` \| `self_eval_thin` \| `leaderboard` |
| `inherited_from` | TEXT | Model this row was copied from, NULL if measured directly |
| `last_updated` | TEXT | ISO8601 | | `last_updated` | TEXT | ISO8601 |
**Category set** (9 categories, defined in `config.yaml`): **Category set** (9 categories, defined in `config.yaml`):
@@ -302,8 +314,14 @@ exclude it can.
**Proficiency inheritance:** `propagate_to_variants` copies evaluated **Proficiency inheritance:** `propagate_to_variants` copies evaluated
`scores` to equivalent serving variants (same weights, same reasoning `scores` to equivalent serving variants (same weights, same reasoning
setting, same context pool), but only when the variant has no measured setting, same context pool), but never over a row that was measured
data itself. A `-fast` row is **not** equivalent to its `-standard` sibling. directly. A `-fast` row is **not** equivalent to its `-standard` sibling.
"Measured directly" is `inherited_from IS NULL`, not `self_eval_samples > 0`
— inheritance copies the sample count too, so sample count alone cannot tell
an inherited row from a measured one, and using it meant a variant inherited
exactly once and then froze forever. `ensure_columns()` adds the column and
backfills provenance on databases that predate it.
### `energy_observations` — per-request telemetry ### `energy_observations` — per-request telemetry
@@ -355,11 +373,43 @@ failure — it means the checker had nothing to say, not that the model failed.
## Weighted Scoring ## Weighted Scoring
Three hard filters are applied **before** scoring (not weighted — outright disqualification): Four hard filters are applied **before** scoring (not weighted — outright disqualification):
1. `effective_context_window ≥ required_context_tokens` 1. `effective_context_window ≥ required_context_tokens`
2. `tier ≥ required_tier` (from classifier) 2. `tier ≥ required_tier` (from classifier)
3. Serving class compatible with request's `latency_tolerance`; `access_level` reachable 3. Serving class compatible with request's `latency_tolerance`; `access_level` reachable
4. `tool_use_agentic` proficiency ≥ `routing.min_tool_proficiency`, **only when
the request carries a `tools` array** (ships disabled — `null`)
5. `supports_vision = 1` when the request carries `image_url` parts; `NULL`
fails closed (an unknown flag means the capability cannot be confirmed)
6. `supports_json_mode = 1` when `response_format.type` is `json_object` or
`json_schema`; `NULL` also fails closed
Filters 4–6 are read from the request body rather than inferred. A `tools`
array states whether tool definitions are on the table, `image_url` parts state
whether vision is needed, and `response_format` states whether JSON mode is
needed. Local classifiers identified unambiguous tool-use prompts only 1-2
times in 6, so inferring capability requirements does not work; the request
states them exactly and for free.
The tool gate is a **quality measurement** filter: a model with no measured
`tool_use_agentic` score is admitted, because "unproven" is not "proven bad".
Only a measured score below `routing.min_tool_proficiency` is dropped. The
vision and JSON-mode gates are **capability flag** filters. There every
routable catalog row has `supports_tools = 1`, so a flag gate would be inert;
vision and JSON mode are not universal, and an absent or `NULL` flag means
"cannot confirm the capability", so the model is dropped. A wrong guess on a
capability flag is a guaranteed provider 400.
The tool filter ships **off** (`null`), because agent clients send `tools` on
nearly every request, so enabling it excludes the cheapest model from ordinary
agent traffic. Whether that trade is worth it is an empirical question best
settled with `POST /outcome` data rather than a 3-task benchmark score. Set it
to `0.5` to turn it on. The vision and JSON-mode gates ship **on**, because a
wrong guess is a guaranteed 400.
Config is strict (`extra="forbid"`): a misspelled or misplaced key fails at
load instead of being silently ignored.
``` ```
1. drop candidates whose measured energy exceeds objective.max_energy_per_request 1. drop candidates whose measured energy exceeds objective.max_energy_per_request
@@ -371,53 +421,78 @@ Three hard filters are applied **before** scoring (not weighted — outright dis
| Setting | Value | Notes | | Setting | Value | Notes |
|---|---|---| |---|---|---|
| `quality_tolerance` | 0.10 | Measurement noise, not preference: scores rest on 2-3 samples, so smaller gaps are sampling variation | | `quality_tolerance` | 0.10 | Measurement noise, not preference: scores rest on 2-3 samples, so smaller gaps are sampling variation |
| `max_energy_per_request` | null | Per-request kWh ceiling. The plan is a 6.25 kWh quota — a wall, not a bill | | `assumed_cache_rate` | 0.917 | Share of prompt tokens served from the provider's prefix cache. Agent clients resend the conversation each turn, so most of it hits. Measured token-weighted over 40.7M tokens; measure your own against the provider's session view |
| `plan_kwh_per_period` | 6.25 | Reported in `/health` as burn against the allowance | | `assumed_completion_tokens` | 500 | Completion length assumed when pricing a candidate |
| `max_energy_per_request` | null | Per-request kWh ceiling — a wall, not a bill. Null disables it |
| `plan_kwh_per_period` | 6.25 | **Set this to your own plan's quota.** Reported in `/health` as burn against the allowance; it does not gate anything |
This replaced a weighted blend (cost 0.4 / eco 0.2 / proficiency 0.4). This replaced a weighted blend (cost 0.4 / eco 0.2 / proficiency 0.4).
Measurement retired it: turning the cost weight from 0.4 to **zero** changed Measurement retired it: turning the cost weight from 0.4 to **zero** changed
the winner in only 2 of 6 categories, so the blend was never steering on the winner in only 2 of 6 categories, so the blend was never steering on
quality — while 60% of every decision adjudicated fractions of a cent. quality — while 60% of every decision adjudicated fractions of a cent.
**Why cost ≠ list price:** Neuralwatt bills flat $8.00/kWh. The catalog's **Cost is priced per request, from catalog token prices scaled to the
`input_per_million` / `output_per_million` are **not** what gets charged. request's shape** — prompt size, `assumed_completion_tokens`,
Measured: `cost_usd / energy_kwh = 8.00` across every model. A model listing `assumed_cache_rate`. It is deliberately *not* a benchmark average.
at $4/1M can cost 10× more than one at $15/1M on the same prompt.
Neuralwatt bills flat per-kWh rather than per-token, so list price is not what
gets charged. Scoring used measured billed cost from a fixed 400-token
reference sweep for exactly that reason — and that was wrong for real traffic,
because the ranking depends on the workload's *shape*, not just the model. On
a 400-token prompt one model looked 3.2× cheaper than another; on a realistic
70k-token prompt the same pair inverted and the second was 5.0× cheaper. The
provider's attribution ratio moves with prompt size, so a fixed-shape
benchmark cannot rank models for a workload of a different shape.
List price is still not what is billed, but billing is capped at a multiple of
it, so it tracks the real ordering and bounds it — and it is free, needs no
sweep, and refreshes whenever the poller runs.
**Why cost ≠ eco:** Cost tracks energy (kWh), but carbon is energy × grid **Why cost ≠ eco:** Cost tracks energy (kWh), but carbon is energy × grid
intensity. Grid intensity spans ~49 gCO2/kWh (`FI`) to ~442 (`US-MIDA-PJM`) — roughly a 9× spread, intensity. Grid intensity spanned ~49 gCO2/kWh (`FI`) to ~442
and it moves: the provider's own 24h blended figure was 145.4. The models disagree: `glm-5.2-fast` is (`US-MIDA-PJM`) when measured — and it moves with time of day. The models
disagree: `glm-5.2-fast` is
2nd cheapest but 6th cleanest; `kimi-k3-flex` draws 3.7× less energy than 2nd cheapest but 6th cleanest; `kimi-k3-flex` draws 3.7× less energy than
`kimi-k2.7-code` while emitting 3.6× more carbon. Collapsing them picks a `kimi-k2.7-code` while emitting 3.6× more carbon. Collapsing them picks a
side. side.
### What routing actually does today — expect one model ### What routing actually returns, and why it moves
Run `/route` across all 9 categories and 3 tiers and every one of those 27 Sweeping `/route` across every category and tier is the fastest way to see
decisions currently returns **`qwen3.6-35b`**. That is not a bug and not a whether your data is doing anything. On the deployment this was written
misconfiguration, so it is worth stating plainly before you go looking for against, 9 categories × 3 tiers currently yields **5 distinct winners**
one. (`qwen3.6-35b`, `gemma-4-31b`, `deepseek-v4-flash`, `kimi-k3`,
`kimi-k3-fast`).
On this catalog `qwen3.6-35b` is **Pareto-dominant**: cheapest *and* cleanest That number is a diagnostic, not a target, and it is worth knowing what each
in the routable set, while scoring within 0.15 of the best model on outcome means:
proficiency. On `summarization` it costs 7× less and emits 65× less carbon
than `kimi-k3`, which beats it 1.00 to 0.85 on quality — no defensible
weighting picks `kimi-k3` there. One model winning is the correct answer to
the question the weights ask.
Proficiency still does work underneath: runner-up ordering reorders by - **One winner everywhere** is a legitimate answer, not a misconfiguration.
category (`kimi-k3-fast` climbs to 3rd on `docs_writing` and `summarization` It happened here: with cost, eco and proficiency all populated, one model
where it scores 1.00, and leaves the top four elsewhere). The axis is live; was Pareto-dominant — cheapest *and* cleanest in the routable set while
it simply cannot overturn a leader that wins on two axes at once. scoring within `quality_tolerance` of the best. No defensible weighting
picks anything else. If you see this, check whether the leader really is
dominant before reaching for the config.
- **Winners that change with context size** are the hard filters working.
A model is dropped once `required_context_tokens` exceeds its window, so a
long session can change model mid-conversation. Past the largest window,
`/v1/chat/completions` returns 422 naming the constraint rather than
silently truncating.
- **Winners that change by category** mean proficiency is live. That is the
only category-dependent term, so until the `proficiency` table has data,
`task_category` cannot change a decision at all — the classifier computes
it and the router pays for it for nothing.
The spread widens when the leader stops being eligible. Past ~94,196 tokens The spread here widened for two reasons worth copying: cost became a
of context `qwen3.6-35b` is filtered out and `glm-5.2-fast` takes over; past per-request estimate rather than a benchmark average, and tier stopped being
~790K nothing qualifies and `/v1/chat/completions` returns 422 naming the inferred from price alone. Both had been quietly excluding a cheap
constraint rather than truncating. So on a large codebase you will see the large-context model from every request above tier 1.
model change mid-session — that is the context filter working.
If a different balance is wanted, the lever is `weights` in `config.yaml`, If you want a different balance, the levers are `objective.quality_tolerance`
not more data. (how big a quality gap must be before it outranks cost) and
`objective.max_energy_per_request` (a hard ceiling). There is no weight to
tune — quality is the objective and cost is the tiebreak, which replaced an
earlier weighted blend.
## API Endpoints ## API Endpoints
@@ -455,6 +530,11 @@ structural verification verdict surfaces in the `X-Router-Verification` header
so a client can inspect it without parsing the response body. Valid values: so a client can inspect it without parsing the response body. Valid values:
`ok`, `truncated`, `malformed`, `unverifiable`, `none`. `ok`, `truncated`, `malformed`, `unverifiable`, `none`.
**Capability 422s**: When no model survives the hard filters, the 422 names the
active constraints. That now includes "vision-capable model" or
"json-mode-capable model" when the request carried images or a JSON-mode
`response_format`, alongside the existing context/tier/latency/tool reasons.
## Quick Usage ## Quick Usage
```bash ```bash
@@ -478,8 +558,65 @@ curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application
# Admit flex rows for overnight/async work # Admit flex rows for overnight/async work
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \ curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
-d '{"task":"nightly code review","latency_tolerance":"batch"}' -d '{"task":"nightly code review","latency_tolerance":"batch"}'
# Image URL request (routed only to vision-capable catalog rows)
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"auto","messages":[{"role":"user","content":[{"type":"text","text":"Describe this"},{"type":"image_url","image_url":{"url":"data:image/gif;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs="}}]}]}'
# JSON-mode request via response_format
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"auto","messages":[{"role":"user","content":"Return a JSON object with field answer"}],"response_format":{"type":"json_object"}}'
``` ```
## Logging and Traceability
One `route` line per request says what was decided; one `dispatch` line says
what it cost. Both carry a trace id, and `rid`/`sess` are columns in
`energy_observations`, so a journal line pivots to its database row and back.
```
route id=r9116d9 cat=coding_refactor tier=2 ctx=500 src=classifier tools=0
latency=interactive cand=8 pick=deepseek-v4-flash est_usd=0.00016296 ms=1868
dispatch id=r9116d9 model=deepseek-v4-flash rid=chatcmpl-... sess=c090d751
p_tok=59511 c_tok=415 kwh=4.8e-05 usd=0.000783 verdict=ok total_ms=5692
```
At `debug` it also says *why* — the classifier's answer and timing, every
candidate that was dropped and by which filter, and the ranking with scores:
```
classify id=r9116d9 cat=coding_refactor tier=2 confidence=0.95 ms=1863
filter id=r9116d9 model=gemma-4-31b reason=tier(1<2)
filter id=r9116d9 model=glm-5.2-short reason=access_level(preview)
filter id=r9116d9 model=kimi-k3-flex reason=latency_class(flex)
rank id=r9116d9 pos=0 model=deepseek-v4-flash prof=1 est_usd=0.00016296
```
**No conversation text is logged at any level**, prompt or answer — prompts
here run 60k–150k tokens and the journal is on disk. A test enforces it.
Set the level in `config.yaml` (`logging.level`), or override it without
touching a tracked file:
```bash
systemctl --user edit llm-router # Environment="LLM_ROUTER_LOG_LEVEL=debug"
systemctl --user restart llm-router
```
Reading it back:
```bash
journalctl --user -u 'llm-router*' -f # quote the glob in zsh
journalctl --user -u llm-router -f -o cat # message only
journalctl --user -u llm-router -p warning # severity actually filters
journalctl --user -u llm-router --grep 'chatcmpl-abc123' # one request's trail
journalctl --user -u llm-router --grep ' id=r9116d9' # every stage of it
```
`-p` works because the service emits journald priority prefixes when — and only
when — systemd owns its stderr. Run uvicorn in a terminal and the lines come
out clean.
## Scheduled Jobs (systemd) ## Scheduled Jobs (systemd)
Five user units. See `deploy/README.md` for full install/operate instructions. Five user units. See `deploy/README.md` for full install/operate instructions.
@@ -487,17 +624,23 @@ Five user units. See `deploy/README.md` for full install/operate instructions.
| Unit | Schedule | What it runs | | Unit | Schedule | What it runs |
|---|---|---| |---|---|---|
| `llm-router.service` | Continuous | FastAPI dispatcher | | `llm-router.service` | Continuous | FastAPI dispatcher |
| `llm-router-poller.timer` | 2 min after boot, then every 2 h | `poller.py` → `tier.py` | | `llm-router-poller.timer` | 2 min after boot, then every 2 h | triggers the poller unit |
| `llm-router-seed.timer` | Every 6 h | Small `seed_energy.py` sweep | | `llm-router-poller.service` | oneshot | `poller.py` → `tier.py` |
| `llm-router-seed.timer` | Every 6 h | triggers the seed unit |
| `llm-router-seed.service` | oneshot | small `seed_energy.py` sweep |
**The poller timer is load-bearing, not optional.** `freshness.stale_after_days` **The poller timer is load-bearing, not optional.** `freshness.stale_after_days`
is 3 with `exclude_stale: true` — an unpolled catalog marks every row stale is 3 with `exclude_stale: true` — an unpolled catalog marks every row stale
in 3 days and the router returns zero candidates for everything. in 3 days and the router returns zero candidates for everything.
**The seed timer spans time.** Energy attribution drifts with pool load **The seed timer spans time.** Energy attribution drifts with pool load
across hours (~50× for `deepseek-v4-flash` between sweeps), so a single across hours (~50× for one model between sweeps), so a single sweep measures
sweep measures one moment. The median has to span time — every 6 h sweep one moment. The median has to span time — every 6 h sweep accumulates into a
accumulates into a time-weighted median automatically. median-across-time automatically.
This now feeds `eco` only. Cost is priced per request from catalog prices, so
routing no longer depends on the sweep at all; disabling this timer costs you
carbon figures, not routing quality.
## Self-Eval Harness (`eval_proficiency.py`) ## Self-Eval Harness (`eval_proficiency.py`)
@@ -517,13 +660,40 @@ python eval_proficiency.py --dry-run # plan only
## Classifier Reliability Notes ## Classifier Reliability Notes
Several tuning decisions keep the classifier from cascading failures: The classifier is the one blocking LLM call on the request path, so it sets
the latency floor for every routed request. `classifier.base_url` /
`api_key_env` / `model` accept any OpenAI-compatible endpoint — a local
Ollama, an Ollama on another machine across a VPN, or a cloud model.
**Pick a non-reasoning model.** The classifier's entire output is ~45 tokens
of JSON, and a reasoning model spends its budget getting there. Replacing
`qwen3.5` with `mistral-nemo:12b` on identical prompts, cold load excluded:
| | `qwen3.5` | `mistral-nemo:12b` |
|---|---|---|
| category correct | 10/14 | 9/14 |
| tier correct | 1/5 (2 crashed) | 3/5 |
| **hard failures** | **4 of 19 calls** | **0** |
| latency mean / max | 6.6s / **15.6s** | 1.7s / **1.9s** |
Accuracy is a wash; the failure column and the tail are not. Every failure was
an unbounded thinking trace producing no JSON, which degrades to
`source: "fallback"` after ~15s. End-to-end `/route` went ~10s → ~1.7s.
Several settings keep it from cascading failures:
- **`max_input_chars: 8000`** clamps what the classifier is shown to head +
tail. It decides a category and a tier; it does not need the document, and
feeding it one is harmful rather than wasteful — on a ~20k-token prompt one
model returned empty after 28.7s and the other spent 41.8s echoing the input
into its JSON. Clamped, the same prompts classify in ~2.2s.
- **`max_retries: 0`** on the classifier client. The OpenAI SDK retries twice - **`max_retries: 0`** on the classifier client. The OpenAI SDK retries twice
by default, so `timeout_seconds` silently became a 3× wall-clock bound — a by default, so `timeout_seconds` silently became a 3× wall-clock bound — a
request could hang past 250s on a 120s setting with no log. request could hang past 250s on a 120s setting with no log.
- **`max_output_tokens: 1024`** bounds a reasoning model's chain of thought. - **`max_output_tokens: 1024`** bounds a reasoning model's chain of thought.
Without it, `qwen3.5` emits unbounded traces — Ollama keeps generating after Without it, a reasoning model emits unbounded traces — Ollama keeps generating after
the client gives up AND serializes per model, so one runaway request queues the client gives up AND serializes per model, so one runaway request queues
every later request behind it. 256 was tried first and was too tight; the every later request behind it. 256 was tried first and was too tight; the
trace would consume the whole budget and the model would be truncated before trace would consume the whole budget and the model would be truncated before
@@ -539,27 +709,39 @@ Several tuning decisions keep the classifier from cascading failures:
## Pointing a Coding Agent at It ## Pointing a Coding Agent at It
The `/v1` endpoints are OpenAI-compatible. Repo-local `opencode.json` is The `/v1` endpoints are OpenAI-compatible, so any normal client works —
already wired up (`cd ~/Sources/6krrt && opencode` routes by the router by opencode, an SDK, plain curl. A repo-local `opencode.json` is included, so
default). For global use, merge `provider.llm-router` into running `opencode` from a clone of this repo routes through the router by
default. For global use, merge its `provider.llm-router` block into
`~/.config/opencode/opencode.json`. `~/.config/opencode/opencode.json`.
Virtual model names: Virtual model names:
- `auto` → router picks, interactive mode (flex rows excluded) - `auto` → router picks, interactive mode (flex rows excluded)
- `auto:batch` → router picks, admits flex rows for async work - `auto:batch` → router picks, admits flex rows for async work
**opencode image support**: The repo's `opencode.json` declares
`"modalities": {"input": ["text", "image"]}` for every `llm-router` model.
This is required: opencode strips image parts client-side unless the provider
model declares image input. Without it, images never reach the router at all.
For global use, make sure the merged `~/.config/opencode/opencode.json` entry
carries the same modality block.
## Testing ## Testing
```bash ```bash
python -m pytest # run all 207 tests python -m pytest # 356 tests
python -m pytest --cov # with coverage python -m pytest --cov # with coverage
``` ```
No test calls a provider or a local model — the pure modules take rows and
config as arguments, so the suite runs offline on a clean checkout.
| Test file | What it covers | | Test file | What it covers |
|---|---| |---|---|
| `test_scoring.py` | `normalize_inverted`, `composite_score`, None-handling | | `test_scoring.py` | `normalize_inverted`, `composite_score`, None-handling |
| `test_routing.py` | Hard filters, candidate selection & ranking across all 9 categories × 3 tiers | | `test_routing.py` | Hard filters, candidate selection & ranking across all 9 categories × 3 tiers |
| `test_tiering.py` | Tier resolver precedence: override → reasoning → cost → mid | | `test_tiering.py` | Tier resolver precedence: override → reasoning → cost + context window → mid |
| `test_apply_tiering.py` | DB tiering pass, sanity guards | | `test_apply_tiering.py` | DB tiering pass, sanity guards |
| `test_poller_parsing.py` | Serving class, base model, access level parsing | | `test_poller_parsing.py` | Serving class, base model, access level parsing |
| `test_proficiency.py` | Blending rule, accumulation | | `test_proficiency.py` | Blending rule, accumulation |
@@ -568,9 +750,17 @@ python -m pytest --cov # with coverage
| `test_task_set.py` | Reference solutions validating every `code` task's checks & `exact` answers | | `test_task_set.py` | Reference solutions validating every `code` task's checks & `exact` answers |
| `test_verification.py` | Structural checks on every language, verifier edge cases, local-LLM gating | | `test_verification.py` | Structural checks on every language, verifier edge cases, local-LLM gating |
| `test_feedback.py` | Failure identification, idempotency, attribution filtering | | `test_feedback.py` | Failure identification, idempotency, attribution filtering |
| `test_iteration.py` | Retry budget per tier, and matching the retry to the failure kind |
| `test_session_identity.py` | Outcome attribution: session matching, ambiguity refusal |
| `test_config_endpoints.py` | Classifier and verifier are separately addressable; guards on the split |
## Setup ## Setup
**You need:** a Neuralwatt API key, Python 3.10+ (suite verified on 3.10 and
3.14), and an Ollama reachable from wherever this runs with a classifier model
pulled. Nothing else is assumed about the host — routing itself is SQLite and
arithmetic.
```bash ```bash
python -m venv .venv && source .venv/bin/activate python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt pip install -r requirements.txt
@@ -578,15 +768,50 @@ sqlite3 router.db < schema.sql
cp .env.example .env # fill in NEURALWATT_API_KEY cp .env.example .env # fill in NEURALWATT_API_KEY
python poller.py # populate the catalog python poller.py # populate the catalog
python tier.py # resolve tiers python tier.py # resolve tiers
python seed_energy.py # seed reference energy observations (~5 per model) python config.py # sanity-check config loads
python -m uvicorn dispatcher:app --reload python -m uvicorn dispatcher:app --reload
``` ```
Ollama must be running locally: Then edit `config.yaml` for your own setup — at minimum:
| Key | Why |
|---|---|
| `classifier.model` | must match a model `ollama list` reports |
| `classifier.base_url` | where that Ollama actually is |
| `objective.plan_kwh_per_period` | your plan's quota; `/health` reports burn against it |
| `objective.assumed_cache_rate` | 0.917 was measured from one client's traffic (40.7M tokens). Check yours against the provider's per-session cache-hit figures |
`python seed_energy.py` is optional. It sweeps a fixed reference workload to
populate `eco`, which is logged but is not an objective — routing works
without it. It costs real money and quota, so it is not in the path above.
### Where Ollama lives
```bash ```bash
ollama pull qwen3.5:latest ollama pull mistral-nemo:12b # or whatever you set as classifier.model
``` ```
It does not have to be on the machine running the router; the box with the
GPU usually isn't the laptop. To use one across a VPN, point **both**
endpoints at it:
```yaml
classifier:
base_url: "http://<vpn-ip>:11434/v1"
verification:
base_url: "http://<vpn-ip>:11434" # same host, so `model` can stay null
```
and apply `deploy/ollama-over-vpn.conf` on the serving host — Ollama binds
`127.0.0.1` by default and will otherwise refuse. Bind it to the VPN address
rather than `0.0.0.0`: Ollama has no authentication, so anything reaching the
port can run inference and enumerate your models.
Both endpoints move together because the verifier speaks Ollama's *native*
API and cannot follow the classifier to a cloud provider. Config load refuses
the case where they are on different hosts and `verification.model` is null,
because that combination fails silently.
## Known Limitations & Open Items ## Known Limitations & Open Items
- **Leaderboard priors unfilled** — `leaderboards.yaml` ships empty. `python leaderboard.py --check` - **Leaderboard priors unfilled** — `leaderboards.yaml` ships empty. `python leaderboard.py --check`
@@ -596,14 +821,22 @@ ollama pull qwen3.5:latest
- **Three models unsettled** — split-half stability at 7 samples: - **Three models unsettled** — split-half stability at 7 samples:
`kimi-k2.7-code-fast` (29×), `kimi-k3` (14×), `glm-5.2-flex` (2.2×). `kimi-k2.7-code-fast` (29×), `kimi-k3` (14×), `glm-5.2-flex` (2.2×).
More samples needed before their positioning is trustworthy. More samples needed before their positioning is trustworthy.
- **Verification observes but never intervenes** — failures are detected and - **Retry does not reach streaming** — a failed structural check now buys
fed back into `proficiency` automatically (`verification.py` → corrective attempts scaled by tier (`iteration.py`), but only on the
`feedback.py`), so the router does learn which models fail on real work. non-streaming path. Once bytes have gone to the client there is nothing to
What is missing is action: nothing retries or escalates on a failed check. take back, and buffering to allow correction would cost streaming itself.
That is deliberate — whether auto-escalation pays should be decided from `POST /outcome` is the answer for streamed traffic: the client reports
the observed failure rate, which is now being collected rather than afterwards, so it works identically either way.
guessed at. - **No auth** — the service holds a billable API key with no authentication
- **No auth** — the service holds a billable API key with no of its own. It binds loopback for that reason; widening the bind needs an
authentication of its own. Loopback binds only. auth layer first. The same applies to an Ollama shared over a VPN, which
also has no auth — bind it to the VPN address, never `0.0.0.0`.
- **Local energy is not accounted for** — the router measures what the cloud
provider bills and reports, but the electricity its own classifier and
verifier burn on your hardware is invisible to it. Since the whole premise
is spending cheap local compute to avoid wasting expensive cloud compute,
that half of the ledger is currently taken on faith. Closing it means
metering local draw and pricing it against a real tariff rather than
assuming local is free.
- **Context assembly (RAG)** is out of scope — the classifier sees the full - **Context assembly (RAG)** is out of scope — the classifier sees the full
conversation but does not perform document/code retrieval. conversation but does not perform document/code retrieval.

81
capabilities.py Normal file
View File

@@ -0,0 +1,81 @@
"""Detect request-side capability requirements from an OpenAI-format body.
Like ``routing.py``, ``scoring.py`` and ``tiering.py``, this module is free of
I/O: the request body comes in as a dict and capabilities come out as a value
object. ``dispatcher.py`` owns the parsing of the HTTP request into that dict.
Detection is read from the request body, not inferred by a classifier. This
mirrors the existing ``tools_present`` / ``min_tool_proficiency`` philosophy in
``routing.py``: a local classifier cannot reliably identify agentic or visual
work — asked to label unambiguous tool-use prompts it got 1-2 of 6, and vision
is the same class of problem. But the request states both exactly, in the
``tools`` array and in an ``image_url`` part, and reading them is free. When
the capability matters, it is a hard filter (a model that cannot be trusted
with tools that exist is not a candidate), never a weighted preference.
``has_reasoning_request`` is the one flag that is informational only: it is
recorded so dispatcher can observe it, but never fed to a routing filter.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
# OpenAI response_format.types that require the model to support JSON mode.
JSON_MODE_TYPES = frozenset({"json_object", "json_schema"})
@dataclass(frozen=True)
class RequestCapabilities:
has_images: bool = False
require_json_mode: bool = False
tools_present: bool = False
has_reasoning_request: bool = False
def detect_capabilities(body: dict[str, Any]) -> RequestCapabilities:
"""Read the capability requirements stated by an OpenAI-format request body.
``has_images`` scans ALL messages, not just the last: vision is a property
of the whole conversation, and an image in an earlier turn still has to be
understood by whoever answers the latest one. A message whose ``content`` is
a list contributes True if any part is a dict whose ``type`` is
``image_url``. String content and non-image parts never trigger it; a raw
base64 string sitting in a text part is not an image_url and must not be
matched.
``require_json_mode`` reads ``response_format.type`` against
``JSON_MODE_TYPES``. ``text`` and an absent ``response_format`` read False.
``tools_present`` is truthy exactly when the body carries a ``tools`` array.
``has_reasoning_request`` is True if the body carries either a
``reasoning_effort`` key or a ``reasoning`` key. This flag is recorded for
observation only and is never consumed by a routing filter.
"""
has_images = _any_message_has_images(body.get("messages") or [])
response_format = body.get("response_format")
require_json_mode = (
isinstance(response_format, dict)
and response_format.get("type") in JSON_MODE_TYPES
)
return RequestCapabilities(
has_images=has_images,
require_json_mode=require_json_mode,
tools_present=bool(body.get("tools")),
has_reasoning_request="reasoning_effort" in body or "reasoning" in body,
)
def _any_message_has_images(messages: list[Any]) -> bool:
for message in messages:
if not isinstance(message, dict):
continue
content = message.get("content")
if not isinstance(content, list):
continue
for part in content:
if isinstance(part, dict) and part.get("type") == "image_url":
return True
return False

243
config.py
View File

@@ -4,19 +4,37 @@ Loads and validates config.yaml for the local LLM router.
Usage: Usage:
from config import load_config from config import load_config
cfg = load_config("config.yaml") cfg = load_config("config.yaml")
cfg.weights.cost # etc. cfg.objective.quality_tolerance # etc.
""" """
from __future__ import annotations from __future__ import annotations
from pathlib import Path from pathlib import Path
from typing import Optional from typing import Optional
from urllib.parse import urlparse
import yaml import yaml
from pydantic import BaseModel, field_validator, model_validator from pydantic import BaseModel, ConfigDict, field_validator, model_validator
class Objective(BaseModel): class StrictModel(BaseModel):
"""Base for every config section: an unknown key is an error.
Pydantic ignores extra keys by default, which makes a typo or a
misplaced setting silently do nothing while the file still loads and
still looks configured. That is not hypothetical here — `max_input_chars`
was written into the `verification:` block instead of `classifier:`,
where it was accepted, ignored, and had no effect. It happened to carry
the same value as the code default, so nothing visibly broke; editing it
would simply have done nothing.
Anyone tuning this file needs a wrong key to say so.
"""
model_config = ConfigDict(extra="forbid")
class Objective(StrictModel):
"""What the router optimizes: quality, bounded by cost. """What the router optimizes: quality, bounded by cost.
Replaced a three-way weighted blend. See config.yaml for why — briefly, Replaced a three-way weighted blend. See config.yaml for why — briefly,
@@ -25,7 +43,7 @@ class Objective(BaseModel):
""" """
quality_tolerance: float = 0.10 quality_tolerance: float = 0.10
assumed_cache_rate: float = 0.84 assumed_cache_rate: float = 0.917
assumed_completion_tokens: int = 500 assumed_completion_tokens: int = 500
max_energy_per_request: Optional[float] = None max_energy_per_request: Optional[float] = None
plan_kwh_per_period: Optional[float] = None plan_kwh_per_period: Optional[float] = None
@@ -47,10 +65,44 @@ class Objective(BaseModel):
return v return v
class ContextConfig(BaseModel): class ContextOverride(StrictModel):
"""Per-model context handling, for a row whose real limits are known.
Typed rather than a bare ``dict`` so a typo INSIDE an override is an error
too. That is the whole point of StrictModel, and it was not true here: the
block validated, nothing read it, and config.yaml shipped a worked example
for it -- so anyone who followed that example got silence.
Both fields are optional; whichever is absent falls back to the global.
"""
safety_factor: Optional[float] = None
output_reserve_tokens: Optional[int] = None
@field_validator("safety_factor")
@classmethod
def factor_in_range(cls, v: Optional[float]) -> Optional[float]:
if v is not None and not (0.0 < v <= 1.0):
raise ValueError(
"context.per_model_overrides[...].safety_factor must be in (0, 1]"
)
return v
@field_validator("output_reserve_tokens")
@classmethod
def reserve_not_negative(cls, v: Optional[int]) -> Optional[int]:
if v is not None and v < 0:
raise ValueError(
"context.per_model_overrides[...].output_reserve_tokens must be >= 0"
)
return v
class ContextConfig(StrictModel):
safety_factor: float safety_factor: float
default_output_reserve_tokens: int default_output_reserve_tokens: int
per_model_overrides: dict = {} # Read by poller.ModelRow.effective_context_window.
per_model_overrides: dict[str, ContextOverride] = {}
@field_validator("safety_factor") @field_validator("safety_factor")
@classmethod @classmethod
@@ -60,7 +112,7 @@ class ContextConfig(BaseModel):
return v return v
class ProficiencyConfig(BaseModel): class ProficiencyConfig(StrictModel):
self_eval_min_samples: int self_eval_min_samples: int
leaderboard_weight: float leaderboard_weight: float
self_eval_weight: float self_eval_weight: float
@@ -77,7 +129,7 @@ class ProficiencyConfig(BaseModel):
return self return self
class TieringConfig(BaseModel): class TieringConfig(StrictModel):
cheap_completion_max: float cheap_completion_max: float
tier1_context_max: float = float("inf") tier1_context_max: float = float("inf")
model_tiers: dict[str, int] model_tiers: dict[str, int]
@@ -108,9 +160,20 @@ class TieringConfig(BaseModel):
return v return v
class RoutingConfig(BaseModel): class RoutingConfig(StrictModel):
allowed_access_levels: list[str] allowed_access_levels: list[str]
default_latency_tolerance: str default_latency_tolerance: str
# Applied only when the REQUEST carries tool definitions. None disables it.
min_tool_proficiency: Optional[float] = 0.5
tool_use_category: str = "tool_use_agentic"
# Whether a request carrying image parts is hard-restricted to vision-capable
# models. A wrong guess here is a guaranteed 400, so this gates by default
# and routing fails closed when the catalog flag is unknown.
require_vision: bool = True
# Whether a response_format requiring json_object/json_schema is hard-restricted
# to JSON-mode-capable models. Same guaranteed-failure argument.
require_json_mode: bool = True
@field_validator("allowed_access_levels") @field_validator("allowed_access_levels")
@classmethod @classmethod
@@ -136,15 +199,54 @@ class RoutingConfig(BaseModel):
return v return v
class VerificationConfig(BaseModel): class VerificationConfig(StrictModel):
local_llm_enabled: bool = True local_llm_enabled: bool = True
min_completion_tokens: int = 600 min_completion_tokens: int = 600
timeout_seconds: int = 60 timeout_seconds: int = 60
max_output_tokens: int = 1024 max_output_tokens: int = 1024
outcome_attribution_window_seconds: int = 120 outcome_attribution_window_seconds: int = 120
# The local checker's OWN endpoint, no longer derived from the
# classifier's. It speaks Ollama's NATIVE API (/api/chat, think=False),
# which no cloud provider offers, so it must keep pointing at an Ollama
# instance even when classification has been moved off this machine.
base_url: str = "http://localhost:11434"
# None means "whatever the classifier uses", which is correct only while
# both run on the same local Ollama. Set it explicitly once they diverge.
model: Optional[str] = None
class EscalationConfig(BaseModel): class LocalVisionConfig(StrictModel):
enabled: bool = False
base_url: str = "http://localhost:11434/v1" # OpenAI-compatible (classifier shape)
api_key_env: Optional[str] = None
model: str = "qwen3-vl:4b"
timeout_seconds: int = 60
max_images: int = 4
max_image_bytes: int = 9 * 1024 * 1024 # 9 MiB, Ollama default cap
@field_validator("timeout_seconds")
@classmethod
def timeout_positive(cls, v: int) -> int:
if v <= 0:
raise ValueError("local_vision.timeout_seconds must be > 0")
return v
@field_validator("max_images")
@classmethod
def images_positive(cls, v: int) -> int:
if v <= 0:
raise ValueError("local_vision.max_images must be > 0")
return v
@field_validator("max_image_bytes")
@classmethod
def image_bytes_positive(cls, v: int) -> int:
if v <= 0:
raise ValueError("local_vision.max_image_bytes must be > 0")
return v
class EscalationConfig(StrictModel):
enabled: bool enabled: bool
max_tier: int max_tier: int
min_confidence_before_bump: float min_confidence_before_bump: float
@@ -152,7 +254,7 @@ class EscalationConfig(BaseModel):
preemptive_on_low_confidence: bool = False preemptive_on_low_confidence: bool = False
class IterationConfig(BaseModel): class IterationConfig(StrictModel):
"""A tier's budget for corrective attempts after a verification failure.""" """A tier's budget for corrective attempts after a verification failure."""
enabled: bool = True enabled: bool = True
@@ -173,41 +275,58 @@ class IterationConfig(BaseModel):
return v return v
class FreshnessConfig(BaseModel): class FreshnessConfig(StrictModel):
stale_after_days: int stale_after_days: int
exclude_stale: bool exclude_stale: bool
exclude_deprecated: bool exclude_deprecated: bool
class DatabaseConfig(BaseModel): class DatabaseConfig(StrictModel):
path: str path: str
class ClassifierConfig(BaseModel): class ClassifierConfig(StrictModel):
provider: str provider: str
base_url: str base_url: str
# Env var holding the API key, for a classifier served by a provider that
# actually checks one. None means unauthenticated, which is the local
# Ollama case — it ignores the key entirely but the SDK requires one.
api_key_env: Optional[str] = None
model: str model: str
# Ceiling on the text handed to the classifier. 0 disables clamping.
max_input_chars: int = 8000
timeout_seconds: int timeout_seconds: int
temperature: float = 0.0 temperature: float = 0.0
max_output_tokens: int = 1024 max_output_tokens: int = 1024
outcome_attribution_window_seconds: int = 120
fallback_tier: int = 2 fallback_tier: int = 2
fallback_category: str = "general_chat" fallback_category: str = "general_chat"
response_format: str response_format: str
system_prompt: str system_prompt: str
class DispatchProvider(BaseModel): class DispatchProvider(StrictModel):
base_url: str base_url: str
api_key_env: str api_key_env: str
class LoggingConfig(BaseModel): class LoggingConfig(StrictModel):
# log_path is gone. Nothing ever wrote a file: the dispatcher logs to
# stderr and systemd captures that to the journal, so the setting named a
# destination that did not exist.
log_energy_observations: bool log_energy_observations: bool
log_path: str # LLM_ROUTER_LOG_LEVEL overrides this at runtime — see logs.resolve_level.
level: str = "info"
@field_validator("level")
@classmethod
def level_known(cls, v: str) -> str:
known = ("debug", "info", "warning", "error")
if v.strip().lower() not in known:
raise ValueError(f"logging.level must be one of {known}, got {v!r}")
return v.strip().lower()
class RouterConfig(BaseModel): class RouterConfig(StrictModel):
objective: Objective objective: Objective
context: ContextConfig context: ContextConfig
tiers: dict[int, str] tiers: dict[int, str]
@@ -215,6 +334,7 @@ class RouterConfig(BaseModel):
proficiency: ProficiencyConfig proficiency: ProficiencyConfig
routing: RoutingConfig routing: RoutingConfig
verification: VerificationConfig = VerificationConfig() verification: VerificationConfig = VerificationConfig()
local_vision: LocalVisionConfig = LocalVisionConfig()
escalation: EscalationConfig escalation: EscalationConfig
iteration: IterationConfig = IterationConfig() iteration: IterationConfig = IterationConfig()
freshness: FreshnessConfig freshness: FreshnessConfig
@@ -223,6 +343,57 @@ class RouterConfig(BaseModel):
dispatch_providers: dict[str, DispatchProvider] dispatch_providers: dict[str, DispatchProvider]
logging: LoggingConfig logging: LoggingConfig
@model_validator(mode="after")
def tool_use_category_is_a_real_category(self) -> "RouterConfig":
"""The tool filter joins on a category name; a typo would disable it.
A name that matches nothing produces NULL for every row, and NULL
means "unproven, do not disqualify" — so the filter would silently
pass everything. Failing at load beats a guard that quietly stops
guarding.
"""
if self.routing.min_tool_proficiency is None:
return self
if self.routing.tool_use_category not in self.proficiency.categories:
raise ValueError(
f"routing.tool_use_category "
f"({self.routing.tool_use_category!r}) is not in "
f"proficiency.categories — the tool-competence filter would "
f"join against nothing and silently pass every model."
)
return self
@model_validator(mode="after")
def verifier_model_is_stated_once_the_hosts_differ(self) -> "RouterConfig":
"""A remote classifier must not lend its model name to the verifier.
``verification.model`` falling back to ``classifier.model`` is correct
only while both point at the same Ollama. Once classification moves
off-host the fallback names a model the local Ollama has never heard
of, and the failure is SILENT: the verifier 404s, catches it, logs
"local verification unavailable" and records no sample. Verification
would appear to be on while producing nothing.
Caught by pointing the classifier at NeuralWatt and watching the
verifier POST ``deepseek-v4-flash`` to localhost:11434. Failing at
config load instead means the misconfiguration is impossible rather
than merely documented.
"""
if not self.verification.local_llm_enabled or self.verification.model:
return self
classifier_host = urlparse(self.classifier.base_url).hostname
verifier_host = urlparse(self.verification.base_url).hostname
if classifier_host != verifier_host:
raise ValueError(
"verification.model must be set explicitly when the classifier "
f"runs on a different host ({classifier_host} vs "
f"{verifier_host}). It would otherwise fall back to "
f"classifier.model ({self.classifier.model!r}), which the local "
"Ollama does not serve — and the verifier fails silently. "
"Set verification.model, or verification.local_llm_enabled: false."
)
return self
def load_config(path: str | Path = "config.yaml") -> RouterConfig: def load_config(path: str | Path = "config.yaml") -> RouterConfig:
path = Path(path) path = Path(path)
@@ -232,14 +403,32 @@ def load_config(path: str | Path = "config.yaml") -> RouterConfig:
return RouterConfig(**raw) return RouterConfig(**raw)
def summary_lines(cfg: RouterConfig) -> list[str]:
"""What ``python config.py`` prints.
A function rather than inline prints so a test can pin the attribute names.
The previous version read ``cfg.weights``, which had been replaced by
``cfg.objective`` -- so the setup step documented in both README.md and
CLAUDE.md said "Config loaded OK" and then died with AttributeError on a
config that had in fact loaded perfectly.
"""
return [
"Config loaded OK",
f" objective: quality_tolerance={cfg.objective.quality_tolerance}, "
f"max_energy_per_request={cfg.objective.max_energy_per_request}, "
f"plan_kwh_per_period={cfg.objective.plan_kwh_per_period}",
f" categories: {cfg.proficiency.categories}",
f" classifier: {cfg.classifier.model} @ {cfg.classifier.base_url}",
f" verifier: {cfg.verification.model or cfg.classifier.model} "
f"@ {cfg.verification.base_url}"
+ ("" if cfg.verification.local_llm_enabled else " (disabled)"),
f" tool filter: min_tool_proficiency={cfg.routing.min_tool_proficiency}",
f" dispatch providers: {list(cfg.dispatch_providers)}",
]
if __name__ == "__main__": if __name__ == "__main__":
import sys import sys
cfg_path = sys.argv[1] if len(sys.argv) > 1 else "config.yaml" cfg_path = sys.argv[1] if len(sys.argv) > 1 else "config.yaml"
cfg = load_config(cfg_path) print("\n".join(summary_lines(load_config(cfg_path))))
print("Config loaded OK")
print(f" weights: cost={cfg.weights.cost} eco={cfg.weights.eco} "
f"proficiency={cfg.weights.proficiency}")
print(f" categories: {cfg.proficiency.categories}")
print(f" classifier: {cfg.classifier.model} @ {cfg.classifier.base_url}")
print(f" dispatch providers: {list(cfg.dispatch_providers.keys())}")

194
config.yaml
View File

@@ -28,10 +28,18 @@ objective:
# to the actual request agree with the live measurement, cost nothing, and # to the actual request agree with the live measurement, cost nothing, and
# need no sweep. # need no sweep.
# Share of prompt tokens served from the provider's prefix cache. Measured # Share of prompt tokens served from the provider's prefix cache. Agent
# from real traffic: 1.9M of 2.2M prompt tokens, 84%. Agent clients resend # clients resend the whole conversation each turn, so most of it is a hit.
# the whole conversation each turn, so most of it is a cache hit. #
assumed_cache_rate: 0.84 # Measured token-weighted across 50 sessions and 40.7M tokens (2026-08-23):
# 91.7% overall, 92.6% on the sessions above 400k tokens, which are the ones
# that carry the cost. The previous 0.84 came from 2.2M tokens of much
# earlier traffic.
#
# Raising it changed NO winner in any of the nine categories at 60k context
# — checked before editing. It is here because the number should be true,
# not because the routing needed it.
assumed_cache_rate: 0.917
# Completion length assumed when pricing a request. Real sessions here median # Completion length assumed when pricing a request. Real sessions here median
# around 200-400 completion tokens against enormous prompts. # around 200-400 completion tokens against enormous prompts.
@@ -55,10 +63,22 @@ objective:
context: context:
safety_factor: 0.75 # fraction of advertised context treated as usable safety_factor: 0.75 # fraction of advertised context treated as usable
default_output_reserve_tokens: 4096 default_output_reserve_tokens: 4096
# If a model has a known-good override, list it here; otherwise the global # Per-model exceptions to the two settings above, for a row whose real
# safety_factor above applies. # limits you have measured. The global factor has to hold for the whole
# catalog, so it is deliberately pessimistic; a model you have actually
# pushed to its limit deserves its own number.
#
# Both keys are optional and each falls back to the global on its own.
# An override of 0 reserve tokens means zero, not "unset".
#
# per_model_overrides:
# qwen3.6-35b:
# safety_factor: 0.85
# output_reserve_tokens: 8192
#
# Takes effect on the next `python poller.py` -- effective_context_window is
# computed at poll time, not per request.
per_model_overrides: {} per_model_overrides: {}
# example: per_model_overrides: { "qwen3.6-35b": { safety_factor: 0.85 } }
tiers: tiers:
# Maps a tier number to a human label, purely for logging/dashboards. # Maps a tier number to a human label, purely for logging/dashboards.
@@ -150,15 +170,91 @@ routing:
# interactive, so a request has to opt in via latency_tolerance. # interactive, so a request has to opt in via latency_tolerance.
default_latency_tolerance: interactive # 'interactive' | 'batch' default_latency_tolerance: interactive # 'interactive' | 'batch'
# There is deliberately no flex discount knob. Cost scoring reads the mean # Minimum tool_use_agentic proficiency required of a model when the REQUEST
# cost actually billed for the reference workload (see seed_energy.py), and # carries tool definitions. A filter, not a weight, because it is a
# a flex row's measured cost already is its flex cost. # capability requirement rather than a preference.
#
# It does NOT ask whether the task is agentic — it asks whether the model
# can be trusted with tools that are on the table. The measured failure is
# the second one: deepseek-v4-flash scores 0.33 here, and the recorded case
# is a NON-agentic prompt ("it is 1:20pm, my meeting is at 3pm, how many
# minutes?") where it called two tools instead of subtracting. A model that
# over-reaches is a hazard wherever tools exist, not only where a classifier
# would say "agentic" — which it cannot do anyway: asked to identify six
# unambiguous tool-use prompts, the local models managed 2/6 and 1/6.
# Whether tools are present is stated in the request body. Read it.
#
# 0.5 sits in the empty band between the only two values the catalog
# currently holds (0.33 and 1.00), so it is not fitted to either. A model
# with NO measured tool score is unproven rather than proven bad, and is not
# dropped.
#
# CURRENTLY DISABLED (null) pending experimentation. The trade being
# measured: opencode sends `tools` on essentially every request, so with the
# filter on, deepseek-v4-flash is excluded from ordinary agent traffic and
# the ~7x cost advantage on coding routes goes unused. With it off, that
# advantage applies — and a model measured at 0.33 on tool use handles
# requests where tools are on the table.
#
# What would settle it is outcome data, not another benchmark: run with it
# off, let POST /outcome report real pass/fail, and compare
# tool_use_agentic proficiency for deepseek before and after. That is the
# one signal here that knows whether the work actually worked.
min_tool_proficiency: null
tool_use_category: tool_use_agentic
# Request-side capability gates. These read the request body (image parts,
# response_format) and hard-restrict to models whose catalog row declares the
# capability. Unlike min_tool_proficiency these are NOT proficiency gates —
# a wrong guess is a guaranteed provider 400, so they gate by default and
# fail closed when the catalog flag is unknown.
require_vision: true
require_json_mode: true
# There is deliberately no flex discount knob. Whether a flex row is usable
# at all is a hard filter above (latency_tolerance), not a price adjustment
# -- being held server-side during peak is a latency property, and the
# catalog advertises flex and standard at the same token price anyway.
local_vision:
# Local Ollama vision fallback. Used ONLY when a request carries image
# parts and routing finds NO cloud model that supports_vision — the cloud
# catalog currently excludes the cost leader (deepseek) on vision, so a
# fallback is what keeps image requests working instead of 422ing.
# It speaks the OpenAI-compatible /v1 surface, so this is an Ollama endpoint
# and the model must be pulled (`ollama pull qwen3-vl:4b`) on that host.
enabled: true
base_url: "http://localhost:11434/v1"
api_key_env: null
model: "qwen3-vl:4b"
timeout_seconds: 60
max_images: 4
max_image_bytes: 9437184
verification: verification:
# Structural checks (parse the code, never run it) are free and always on. # Structural checks (parse the code, never run it) are free, pure Python and
# always on — they need no model and run anywhere, down to an RPi.
# This section governs the LOCAL LLM check, which is not free. # This section governs the LOCAL LLM check, which is not free.
#
# Set local_llm_enabled: false on a host with no usable local inference.
# Structural checking and POST /outcome both keep working; only the
# refusal/incoherence class of failure stops being caught.
local_llm_enabled: true local_llm_enabled: true
# This check speaks Ollama's NATIVE API (/api/chat with think=False), which
# no cloud provider offers, so it is configured SEPARATELY from the
# classifier rather than derived from it. Deriving it meant that pointing
# classification anywhere else sent these requests to <that host>/api/chat.
#
# Point it at the SAME Ollama as the classifier when that is across a VPN;
# leave the model null in that case, since the hosts then match.
base_url: "http://localhost:11434"
# null means "whatever classifier.model is", correct only while both run on
# the same local Ollama — config load REFUSES the null once the hosts differ,
# because the fallback would name a model this Ollama has never heard of and
# the verifier would fail silently.
model: "mistral-nemo:12b"
# Only check answers this large. Measured on real traffic: a local check # Only check answers this large. Measured on real traffic: a local check
# costs ~15% of a median 193-token answer, so it would only pay if such # costs ~15% of a median 193-token answer, so it would only pay if such
# answers failed more than ~15% of the time. At 1,500 completion tokens the # answers failed more than ~15% of the time. At 1,500 completion tokens the
@@ -191,10 +287,28 @@ database:
path: "router.db" path: "router.db"
classifier: classifier:
# Local Ollama instance doing task classification / context sizing. # The local LLM that classifies each task before a cloud model answers it.
# "Local" means your hardware, not necessarily this machine — the box with
# the GPU is usually not the laptop you are typing on. It speaks plain
# OpenAI-compatible chat completions, so point it wherever Ollama lives:
#
# same machine base_url: http://localhost:11434/v1
# over a VPN base_url: http://<vpn-ip>:11434/v1
# (serving host needs deploy/ollama-over-vpn.conf; Ollama
# binds loopback-only by default and will refuse)
#
# Any OpenAI-compatible endpoint works, so a cloud model can classify too —
# set api_key_env for one that checks a key. Worth knowing before assuming
# local is the cheap option: on five prompts NeuralWatt's deepseek-v4-flash
# classified in 1.02s mean against qwen3.5's 11.58s on an RTX 6000, agreed
# with the label 5/5 against 2/4, and cost $0.093 per thousand calls. Local
# inference is not free, it is unbilled.
provider: "ollama" provider: "ollama"
base_url: "http://localhost:11434/v1" base_url: "http://localhost:11434/v1"
model: "qwen3.5:latest" # must match a model `ollama list` reports # Unset means unauthenticated, which is the Ollama case. Name the env var
# holding the key when the endpoint actually checks one.
api_key_env: null
model: "mistral-nemo:12b" # must match a model `ollama list` reports
# A cold Ollama took 43s to answer the first classification, which blew the # A cold Ollama took 43s to answer the first classification, which blew the
# old 30s ceiling and returned 503 to the client. Warm it is ~5s. The # old 30s ceiling and returned 503 to the client. Warm it is ~5s. The
# ceiling is for a cold model load, not the steady state. # ceiling is for a cold model load, not the steady state.
@@ -205,21 +319,36 @@ classifier:
# prompt is untraceable. # prompt is untraceable.
temperature: 0 temperature: 0
# Hard cap on the classifier's generation, to bound a failure mode that # Hard cap on the classifier's generation, to bound a failure mode that
# cascades: qwen3.5 is a reasoning model and will sometimes emit a long # cascades on REASONING models: they emit a long chain of thought, blow past
# chain of thought, blowing past timeout_seconds — and Ollama keeps # timeout_seconds, and Ollama keeps generating after the client gives up AND
# generating after the client gives up AND serializes per model, so one # serializes per model, so one runaway request queues every later one behind
# runaway request queues every later request behind it. # it. Measured on qwen3.5, which failed this way on 4 of 19 calls — each one
# a 15s wait ending in a silent fallback.
# #
# Must leave room for the thinking trace, not just the ~45-token answer. At # The current default (mistral-nemo) does not reason, so this does not bind
# 256 the trace consumed the whole budget on some prompts and the model was # for it. Kept anyway: it costs nothing when unused and is the only thing
# truncated before emitting any JSON at all, which read as an empty # standing between a swapped-in reasoning model and that cascade. If you do
# response. 1024 bounds the runaway while leaving the answer reachable. # swap one in, note 256 was too tight — the trace consumed the whole budget
# and the model was truncated before emitting any JSON.
max_output_tokens: 1024 max_output_tokens: 1024
# Where routing lands when the classifier times out, errors, or returns # Where routing lands when the classifier times out, errors, or returns
# something unparseable. A local model being slow should degrade routing, # something unparseable. A local model being slow should degrade routing,
# not refuse the request — the caller is a coding agent that would rather # not refuse the request — the caller is a coding agent that would rather
# have a mid-tier answer than a 502. # have a mid-tier answer than a 502.
# Ceiling on the text handed to the classifier (head + tail, middle elided).
# It decides a category and a tier; it does not need the document, and
# feeding it one is harmful rather than merely wasteful. Measured on a ~20k
# token prompt: qwen3.5 spent 28.7s and returned empty (budget consumed by
# its reasoning trace), mistral-nemo spent 41.8s echoing the input back
# inside its JSON. Both land on source: "fallback" — the same answer an
# instant failure gives, after 30-40s of local inference.
#
# Nothing is lost: chat_completions measures the real conversation with
# estimate_prompt_tokens and takes the larger value, so required_context
# never depends on what the classifier saw. 0 disables clamping.
max_input_chars: 8000
fallback_tier: 2 fallback_tier: 2
fallback_category: general_chat fallback_category: general_chat
response_format: "json" # ask Ollama to constrain output to valid JSON response_format: "json" # ask Ollama to constrain output to valid JSON
@@ -247,5 +376,26 @@ dispatch_providers:
api_key_env: "NEURALWATT_API_KEY" api_key_env: "NEURALWATT_API_KEY"
logging: logging:
# Whether to write a row to energy_observations for every completion. Off
# means no cost accounting, no reference sweep and no /outcome attribution,
# so leave it on unless you are debugging.
#
# There is no log_path. Nothing ever wrote a file -- the dispatcher prints to
# stderr and the systemd unit hands that to the journal (`journalctl --user
# -u llm-router -f`), so the setting named a destination that did not exist.
log_energy_observations: true log_energy_observations: true
log_path: "router.log"
# debug | info | warning | error.
#
# info gives one line per request: what it was classified as, which model won,
# what it cost, how long each stage took. debug adds why -- every candidate
# that was dropped and by which filter, the ranking with scores, the
# classifier's raw reply. It logs no conversation text at any level; prompts
# here run 60k-150k tokens and the journal is on disk.
#
# LLM_ROUTER_LOG_LEVEL overrides this, so a running service can be turned up
# without editing a tracked file:
#
# systemctl --user edit llm-router # Environment="LLM_ROUTER_LOG_LEVEL=debug"
# systemctl --user restart llm-router
level: info

99
deploy/README.md
View File

@@ -22,11 +22,18 @@ environment, so the API key has to come from a file.
```bash ```bash
# 1. The key. User units don't see your shell env, so .env is required. # 1. The key. User units don't see your shell env, so .env is required.
cd ~/Sources/6krrt cd /path/to/this/repo
echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env
# 2. Install and start # 2. Install, pointing the units at wherever you actually cloned this.
cp deploy/llm-router*.{service,timer} ~/.config/systemd/user/ # The shipped units say %h/llm-router; %h is systemd's expansion for your
# home directory, so only the part after it needs changing. Getting this
# wrong fails at start with status=200/CHDIR rather than anything obvious.
REPO=$(pwd)
mkdir -p ~/.config/systemd/user
for u in deploy/llm-router*.{service,timer}; do
sed "s|%h/llm-router|${REPO}|g" "$u" > ~/.config/systemd/user/"$(basename "$u")"
done
systemctl --user daemon-reload systemctl --user daemon-reload
systemctl --user enable --now llm-router.service llm-router-poller.timer llm-router-seed.timer systemctl --user enable --now llm-router.service llm-router-poller.timer llm-router-seed.timer
@@ -43,7 +50,9 @@ systemctl --user list-timers 'llm-router*'
```bash ```bash
systemctl --user status llm-router.service systemctl --user status llm-router.service
journalctl --user -u llm-router.service -f # request log journalctl --user -u 'llm-router*' -f # everything, live (quote the glob)
journalctl --user -u llm-router -f -o cat # the request log, message only
journalctl --user -u llm-router -p warning # fallbacks, retries, refusals
journalctl --user -u llm-router-poller.service # catalog refreshes journalctl --user -u llm-router-poller.service # catalog refreshes
systemctl --user restart llm-router.service # after editing config.yaml systemctl --user restart llm-router.service # after editing config.yaml
systemctl --user start llm-router-poller.service # force a refresh now systemctl --user start llm-router-poller.service # force a refresh now
@@ -53,6 +62,41 @@ systemctl --user start llm-router-poller.service # force a refresh now
restart. The catalog is read per-request, so a poller run takes effect restart. The catalog is read per-request, so a poller run takes effect
immediately. immediately.
### Turning up the logs
`logging.level` in config.yaml is the documented setting, but flipping it means
editing a tracked file. For a running service use a drop-in instead:
```bash
systemctl --user edit llm-router # Environment="LLM_ROUTER_LOG_LEVEL=debug"
systemctl --user restart llm-router
```
`info` gives one `route` and one `dispatch` line per request — category, tier,
model chosen, cost, latency. `debug` adds every candidate that was dropped and
by which filter, plus the ranking with scores. No conversation text is logged
at any level.
Every line carries a trace id, and the `dispatch` line carries the provider's
completion id and the session fingerprint, both of which are columns in
`energy_observations`:
```bash
journalctl --user -u llm-router --grep ' id=r9116d9' # one request, all stages
journalctl --user -u llm-router --grep 'chatcmpl-abc123' # from a DB row back to its decision
```
Severity filtering works because the service prefixes its lines with journald
priorities when systemd owns its stderr (`SyslogLevelPrefix` is on by default).
A foreground `uvicorn` prints them clean, so the same binary is readable either
way.
**The oneshot units buffer.** `poller.py` and `seed_energy.py` print progress
with plain `print()`, and Python block-buffers stdout when it is not a
terminal, so their output arrives in one dump at exit rather than
progressively. Add `Environment="PYTHONUNBUFFERED=1"` to those units if you
want to watch a sweep as it runs.
## A note on the bind address ## A note on the bind address
`--host 127.0.0.1` is deliberate. The service holds a billable API key and `--host 127.0.0.1` is deliberate. The service holds a billable API key and
@@ -61,10 +105,55 @@ your allowance. `ProtectHome=read-only` plus a `ReadWritePaths` exception for
the repo limits the blast radius on the filesystem, but nothing limits spend. the repo limits the blast radius on the filesystem, but nothing limits spend.
Putting this on a LAN address needs an auth layer first. Putting this on a LAN address needs an auth layer first.
## Using an Ollama on another machine
The local LLM does the classifying; it does not have to be on the machine you
are typing on, and usually the GPU isn't. Most developers already have
WireGuard or a VPN back to a home lab, so the normal shape is router and
editor on the laptop, Ollama on the workstation.
On the **serving** host (the one with the GPU):
```bash
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo cp deploy/ollama-over-vpn.conf \
/etc/systemd/system/ollama.service.d/override.conf
# set OLLAMA_HOST to that host's own VPN address — `ip -4 -o addr show`
sudo nano /etc/systemd/system/ollama.service.d/override.conf
sudo systemctl daemon-reload && sudo systemctl restart ollama
```
On the **client** host, in `config.yaml`:
```yaml
classifier:
base_url: "http://<vpn-ip>:11434/v1"
verification:
base_url: "http://<vpn-ip>:11434" # same host, so `model` can stay null
```
Both must move together. `verification` speaks Ollama's native `/api/chat`
and used to derive its URL from the classifier's; it no longer does, so
pointing only the classifier across the tunnel leaves the verifier talking to
a `localhost` Ollama that may not exist. Config load refuses the combination
where `verification.model` is null and the two hosts differ, because that
failure is otherwise silent — the verifier 404s, catches it, and records no
sample while appearing to be enabled.
Bind Ollama to the **VPN address, not `0.0.0.0`**. It has no authentication of
any kind: anything that reaches the port can run inference, enumerate your
models and pull new ones. Same reasoning as the dispatcher's loopback bind
above.
A cloud endpoint works too — set `classifier.api_key_env` to the env var
holding its key. On a five-prompt comparison NeuralWatt's `deepseek-v4-flash`
classified in 1.02s mean against `qwen3.5`'s 11.58s on an RTX 6000. Local
inference is not free, it is unbilled.
## Pointing opencode at it ## Pointing opencode at it
The repo-local `opencode.json` sets this up already, so running `opencode` The repo-local `opencode.json` sets this up already, so running `opencode`
from inside `~/Sources/6krrt` uses the router by default. To use it from from inside a clone of this repo uses the router by default. To use it from
anywhere, merge the `provider.llm-router` block into anywhere, merge the `provider.llm-router` block into
`~/.config/opencode/opencode.json` and set `"model": "llm-router/auto"`. `~/.config/opencode/opencode.json` and set `"model": "llm-router/auto"`.

12
deploy/llm-router-poller.service
View File

@@ -1,21 +1,21 @@
[Unit] [Unit]
Description=Refresh the NeuralWatt model catalog and re-resolve tiers Description=Refresh the NeuralWatt model catalog and re-resolve tiers
Documentation=file:%h/Sources/6krrt/CLAUDE.md Documentation=file:%h/llm-router/CLAUDE.md
After=network-online.target After=network-online.target
Wants=network-online.target Wants=network-online.target
[Service] [Service]
Type=oneshot Type=oneshot
WorkingDirectory=%h/Sources/6krrt WorkingDirectory=%h/llm-router
EnvironmentFile=%h/Sources/6krrt/.env EnvironmentFile=%h/llm-router/.env
# poller.py refreshes the catalog; tier.py re-resolves tiers from it. Tiers # poller.py refreshes the catalog; tier.py re-resolves tiers from it. Tiers
# are derived from cost and reasoning fields the poll may have changed, so # are derived from cost and reasoning fields the poll may have changed, so
# they always run as a pair. # they always run as a pair.
ExecStart=%h/Sources/6krrt/.venv/bin/python poller.py ExecStart=%h/llm-router/.venv/bin/python poller.py
ExecStart=%h/Sources/6krrt/.venv/bin/python tier.py ExecStart=%h/llm-router/.venv/bin/python tier.py
NoNewPrivileges=true NoNewPrivileges=true
PrivateTmp=true PrivateTmp=true
ProtectSystem=strict ProtectSystem=strict
ProtectHome=read-only ProtectHome=read-only
ReadWritePaths=%h/Sources/6krrt ReadWritePaths=%h/llm-router

10
deploy/llm-router-seed.service
View File

@@ -1,19 +1,19 @@
[Unit] [Unit]
Description=Sample the reference workload to keep cost/eco scoring current Description=Sample the reference workload to keep cost/eco scoring current
Documentation=file:%h/Sources/6krrt/CLAUDE.md Documentation=file:%h/llm-router/CLAUDE.md
After=network-online.target After=network-online.target
Wants=network-online.target Wants=network-online.target
[Service] [Service]
Type=oneshot Type=oneshot
WorkingDirectory=%h/Sources/6krrt WorkingDirectory=%h/llm-router
EnvironmentFile=%h/Sources/6krrt/.env EnvironmentFile=%h/llm-router/.env
# Fewer samples per run than a manual sweep, because the point is coverage # Fewer samples per run than a manual sweep, because the point is coverage
# across TIME rather than depth at one moment — see the timer. # across TIME rather than depth at one moment — see the timer.
ExecStart=%h/Sources/6krrt/.venv/bin/python seed_energy.py --samples 3 ExecStart=%h/llm-router/.venv/bin/python seed_energy.py --samples 3
NoNewPrivileges=true NoNewPrivileges=true
PrivateTmp=true PrivateTmp=true
ProtectSystem=strict ProtectSystem=strict
ProtectHome=read-only ProtectHome=read-only
ReadWritePaths=%h/Sources/6krrt ReadWritePaths=%h/llm-router

10
deploy/llm-router.service
View File

@@ -1,6 +1,6 @@
[Unit] [Unit]
Description=Local LLM model router (FastAPI dispatcher) Description=Local LLM model router (FastAPI dispatcher)
Documentation=file:%h/Sources/6krrt/CLAUDE.md Documentation=file:%h/llm-router/CLAUDE.md
# The classifier talks to Ollama on localhost and the dispatcher talks to # The classifier talks to Ollama on localhost and the dispatcher talks to
# NeuralWatt, so both loopback and real network need to be up. Ollama is a # NeuralWatt, so both loopback and real network need to be up. Ollama is a
# system service and cannot be ordered against from a user unit, so a failed # system service and cannot be ordered against from a user unit, so a failed
@@ -12,11 +12,11 @@ Wants=network-online.target
Type=exec Type=exec
# config.yaml, router.db and router.log are all referenced as relative paths, # config.yaml, router.db and router.log are all referenced as relative paths,
# so this has to be the repo root. # so this has to be the repo root.
WorkingDirectory=%h/Sources/6krrt WorkingDirectory=%h/llm-router
# Holds NEURALWATT_API_KEY. Create it with: # Holds NEURALWATT_API_KEY. Create it with:
# echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env # echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env
EnvironmentFile=%h/Sources/6krrt/.env EnvironmentFile=%h/llm-router/.env
ExecStart=%h/Sources/6krrt/.venv/bin/uvicorn dispatcher:app --host 127.0.0.1 --port 8080 ExecStart=%h/llm-router/.venv/bin/uvicorn dispatcher:app --host 127.0.0.1 --port 8080
Restart=on-failure Restart=on-failure
RestartSec=5s RestartSec=5s
@@ -28,7 +28,7 @@ PrivateTmp=true
ProtectSystem=strict ProtectSystem=strict
ProtectHome=read-only ProtectHome=read-only
# ...except the repo, which needs to be writable for router.db and router.log. # ...except the repo, which needs to be writable for router.db and router.log.
ReadWritePaths=%h/Sources/6krrt ReadWritePaths=%h/llm-router
[Install] [Install]
WantedBy=default.target WantedBy=default.target

41
deploy/ollama-over-vpn.conf Normal file
View File

@@ -0,0 +1,41 @@
# Serve this host's Ollama to the other machines you work from.
#
# The premise of this router is that a local LLM classifies your task before a
# cloud model answers it. "Local" means your hardware, not necessarily the
# machine you are typing on — the box with the GPU usually is not the laptop.
# Most developers already have WireGuard or a VPN back to a home lab, so the
# normal shape is: router and editor on the laptop, Ollama on the workstation,
# classifier.base_url pointing across the tunnel.
#
# Ollama listens on 127.0.0.1:11434 by default, so that fails with
# connection-refused until this drop-in is applied on the SERVING host.
#
# Install (needs root — ollama.service is a system unit):
# sudo mkdir -p /etc/systemd/system/ollama.service.d
# sudo cp deploy/ollama-over-vpn.conf \
# /etc/systemd/system/ollama.service.d/override.conf
# sudo nano /etc/systemd/system/ollama.service.d/override.conf # set the address
# sudo systemctl daemon-reload && sudo systemctl restart ollama
#
# Then on the client machine, in config.yaml:
# classifier.base_url: http://<vpn-ip-of-this-host>:11434/v1
# verification.base_url: http://<vpn-ip-of-this-host>:11434
#
# Bind to the VPN address, NOT 0.0.0.0. Ollama has no authentication of any
# kind: anything that can reach the port can run inference, enumerate your
# models and pull new ones. 0.0.0.0 also publishes it on whatever cafe or
# hotel wifi the laptop is sitting on. Same reasoning that keeps the
# dispatcher itself on loopback.
#
# Replace with this host's own VPN address — 10.x.x.x for WireGuard,
# 100.x.x.x for Tailscale. `ip -4 -o addr show` will tell you.
[Service]
Environment="OLLAMA_HOST=10.0.0.1:11434"
# Keep the model resident between requests. Without this, the first
# classification after an idle gap pays a cold load — measured at 43s, which
# exceeds even the 120s ceiling once the SDK's retries are counted. Serving a
# laptop makes this matter more, not less: requests arrive in bursts separated
# by long gaps, which is exactly the pattern that keeps evicting the model.
Environment="OLLAMA_KEEP_ALIVE=24h"

785
dispatcher.py
View File

File diff suppressed because it is too large Load Diff

81
eval_proficiency.py
View File

@@ -31,6 +31,7 @@ import argparse
import json import json
import os import os
import re import re
import secrets
import sqlite3 import sqlite3
import subprocess import subprocess
import sys import sys
@@ -96,14 +97,47 @@ def strip_fences(text: str) -> str:
return code.strip() return code.strip()
def parse_check_results(stdout: str, marker: str) -> dict[int, bool]:
"""Read the harness's own verdicts out of a script's stdout.
Keyed by check INDEX rather than counted, which bounds the total at the
number of checks even if a line somehow repeats, and drops anything that
is not exactly one of the harness's own three-token lines.
"""
results: dict[int, bool] = {}
for line in stdout.splitlines():
parts = line.split()
if len(parts) != 3 or parts[0] != marker or parts[2] not in ("PASS", "FAIL"):
continue
try:
index = int(parts[1])
except ValueError:
continue
results[index] = parts[2] == "PASS"
return results
def score_code(model_output: str, checks: list[str]) -> tuple[float, str]: def score_code(model_output: str, checks: list[str]) -> tuple[float, str]:
"""Execute the model's code and eval each check against it. """Execute the model's code and eval each check against it.
Returns (fraction of checks passing, detail). Partial credit is Returns (fraction of checks passing, detail). Partial credit is
deliberate: a function correct on three of four cases is genuinely better deliberate: a function correct on three of four cases is genuinely better
than one that fails everything, and a binary score throws that away. than one that fails everything, and a binary score throws that away.
The verdict lines carry a per-run nonce because this harness runs the
model's code as ``__main__``, and models routinely append a demo block. One
that prints the word PASS -- `print("self-test:", f(2) == 4 and "PASS")` --
was counted as a passing check by the old substring count, and a two-check
task scored 1.50. The model cannot predict the nonce, so it cannot vote on
its own work. That was the fourth harness bug here to score the rig rather
than the model; this one inflated instead of deflating, which is why it
went unnoticed.
""" """
if not checks:
return 0.0, "no checks"
code = strip_fences(model_output) code = strip_fences(model_output)
marker = f"CHECK-{secrets.token_hex(8)}"
harness = ( harness = (
HARNESS_PREAMBLE HARNESS_PREAMBLE
+ code + code
@@ -115,7 +149,7 @@ def score_code(model_output: str, checks: list[str]) -> tuple[float, str]:
+ " _ok = bool(eval(_c))\n" + " _ok = bool(eval(_c))\n"
+ " except Exception:\n" + " except Exception:\n"
+ " _ok = False\n" + " _ok = False\n"
+ " print('CHECK', _i, 'PASS' if _ok else 'FAIL')\n" + f" print({marker!r}, _i, 'PASS' if _ok else 'FAIL')\n"
) )
with tempfile.TemporaryDirectory() as tmp: with tempfile.TemporaryDirectory() as tmp:
script = Path(tmp) / "harness.py" script = Path(tmp) / "harness.py"
@@ -131,9 +165,8 @@ def score_code(model_output: str, checks: list[str]) -> tuple[float, str]:
except subprocess.TimeoutExpired: except subprocess.TimeoutExpired:
return 0.0, "timeout" return 0.0, "timeout"
passed = proc.stdout.count("PASS") results = parse_check_results(proc.stdout, marker)
if not checks: passed = sum(1 for i in range(len(checks)) if results.get(i))
return 0.0, "no checks"
if passed == 0 and proc.returncode != 0: if passed == 0 and proc.returncode != 0:
# Distinguish "wrote broken code" from "wrote code that fails cases" # Distinguish "wrote broken code" from "wrote code that fails cases"
first_line = (proc.stderr or "").strip().splitlines()[-1:] or [""] first_line = (proc.stderr or "").strip().splitlines()[-1:] or [""]
@@ -378,21 +411,32 @@ def eval_identities(conn: sqlite3.Connection, cfg: RouterConfig) -> list[dict]:
] ]
ALTERNATE_JUDGE = "qwen3.6-35b" # Tried in order when the configured judge belongs to the model under test.
# A LIST rather than one name, because a single alternate can itself be that
# model: with `--judge-model qwen3.6-35b`, the old guard fired on qwen3.6-35b
# and handed back qwen3.6-35b -- the exact conflict it exists to prevent, and
# silent, because self-judged scores look like any other judge output.
ALTERNATE_JUDGES = ("kimi-k3", "qwen3.6-35b", "gemma-4-31b")
def judge_for(model_id: str, default_judge: str) -> str: def judge_for(model_id: str, default_judge: str) -> Optional[str]:
"""Pick a judge that is not the model being judged. """Pick a judge that is not the model being judged, or None.
A model scoring its own prose is a known bias, and the default judge is A model scoring its own prose is a known bias, and the default judge is
itself in the evaluated set. Swapping in an alternate for its own family itself in the evaluated set. Matching is on the FAMILY, so a -fast row does
costs nothing and removes the obvious conflict. not get judged by its reasoning-on sibling either -- same weights.
Returns None when every candidate shares the model's family, so the caller
skips the task. No sample beats a self-graded one, which is the same rule
score_judge already follows for a judge that returned nothing usable.
""" """
from poller import parse_base_model_id from poller import parse_base_model_id
if parse_base_model_id(model_id) == parse_base_model_id(default_judge): family = parse_base_model_id(model_id)
return ALTERNATE_JUDGE for candidate in (default_judge, *ALTERNATE_JUDGES):
return default_judge if parse_base_model_id(candidate) != family:
return candidate
return None
def main() -> int: def main() -> int:
@@ -426,11 +470,11 @@ def main() -> int:
print("nothing to run", file=sys.stderr) print("nothing to run", file=sys.stderr)
return 1 return 1
judged = sum(1 for t in tasks if t["kind"] == "judge") judge_tasks = sum(1 for t in tasks if t["kind"] == "judge")
print( print(
f"{len(identities)} models x {len(tasks)} tasks = " f"{len(identities)} models x {len(tasks)} tasks = "
f"{len(identities) * len(tasks)} calls" f"{len(identities) * len(tasks)} calls"
+ (f" (+{len(identities) * judged} judge calls)" if judged else "") + (f" (+{len(identities) * judge_tasks} judge calls)" if judge_tasks else "")
) )
if args.dry_run: if args.dry_run:
for i in identities: for i in identities:
@@ -482,10 +526,17 @@ def main() -> int:
elif kind == "tool": elif kind == "tool":
score, detail = score_tool(tool_calls, task) score, detail = score_tool(tool_calls, task)
elif kind == "judge": elif kind == "judge":
judge_model = judge_for(model_id, args.judge_model)
if judge_model is None:
print(
f" {task['id']:30s} --- no judge outside "
f"{model_id}'s family, skipped"
)
continue
judged = score_judge( judged = score_judge(
settings.base_url, settings.base_url,
api_key, api_key,
judge_for(model_id, args.judge_model), judge_model,
task, task,
text, text,
) )

277
logs.py Normal file
View File

@@ -0,0 +1,277 @@
"""Structured logging for the dispatcher.
The service had no logger. It had four ``print(..., file=sys.stderr)`` calls,
all of them failure paths, so a request that WORKED said nothing at all — not
the category, the tier, the candidates, the model chosen, the cost, or the
latency. Watching a live agent session showed only uvicorn's access line, which
does not even name the model that served it.
Three things this module provides, in order of how much they matter:
**A trace id per request.** One request touches classification, filtering,
ranking, dispatch, verification and possibly a retry, and until now nothing
tied those together. The id is carried in a ContextVar so helpers read it
without every signature growing a parameter, and it is printed on every line.
**logfmt, not prose.** ``route id=r7f3a91 cat=coding_general pick=...`` reads
fine in ``journalctl`` and greps without a JSON parser. Prose reads better once
and aggregates never.
**Real journald priorities.** systemd strips a ``<N>`` prefix off a log line and
records the message at that priority (``SyslogLevelPrefix`` defaults to true),
which is what makes ``journalctl -p warning`` mean something. Without it every
line lands at PRIORITY 6 and severity cannot be filtered at all.
The prefix is emitted ONLY when ``$JOURNAL_STREAM`` is set, which systemd
exports when it owns our stderr and a terminal does not — so running uvicorn in
a foreground shell prints clean lines rather than a literal ``<6>`` on each one.
"""
from __future__ import annotations
import contextvars
import logging
import os
import secrets
import sys
from typing import Any, Optional, TextIO
LOGGER_NAME = "llm_router"
LEVEL_ENV = "LLM_ROUTER_LOG_LEVEL"
log = logging.getLogger(LOGGER_NAME)
LEVELS = {
"debug": logging.DEBUG,
"info": logging.INFO,
"warning": logging.WARNING,
"error": logging.ERROR,
}
# syslog priorities, which is what journald speaks. 5 (notice) and 1 (alert)
# are unused: nothing here is between info and warning, or above error.
PRIORITIES = {
logging.DEBUG: 7,
logging.INFO: 6,
logging.WARNING: 4,
logging.ERROR: 3,
logging.CRITICAL: 2,
}
NO_TRACE = "-"
_trace: contextvars.ContextVar[str] = contextvars.ContextVar("trace_id", default=NO_TRACE)
# --- trace ids ------------------------------------------------------------
def new_trace() -> str:
"""Start a new request trace and return its id."""
trace_id = "r" + secrets.token_hex(3)
_trace.set(trace_id)
return trace_id
def current_trace() -> str:
return _trace.get()
def set_trace(trace_id: str) -> None:
"""Adopt an existing trace id in this context.
Needed on the streaming path. A StreamingResponse's generator is iterated
from a different context than the endpoint that built it, so the ContextVar
set in the endpoint is NOT visible inside the generator — it must capture
the id as a local and re-set it here, or every streamed request logs no id
at all. That is the path all agent traffic takes.
"""
_trace.set(trace_id)
# --- rendering ------------------------------------------------------------
def render_value(value: Any) -> str:
"""One logfmt value: unambiguous, single-line, cheap to read."""
if value is None:
return "-"
if isinstance(value, bool):
return "1" if value else "0"
if isinstance(value, float):
# Six significant figures. Enough for microdollars and kWh in
# scientific notation, without 0.0007830000000000001.
return f"{value:.6g}"
text = str(value)
if text == "":
return '""'
if any(c in text for c in ' "\n\t='):
text = text.replace("\\", "\\\\").replace('"', '\\"')
text = text.replace("\n", " ").replace("\t", " ")
return f'"{text}"'
return text
def render(name: str, fields: dict[str, Any], trace_id: Optional[str] = None) -> str:
parts = [name, f"id={trace_id or current_trace()}"]
parts += [f"{key}={render_value(value)}" for key, value in fields.items()]
return " ".join(parts)
def _emit(
level: int, name: str, fields: dict[str, Any], trace_id: Optional[str] = None
) -> None:
"""Emit one logfmt line, if this level is enabled.
The level check comes first so DEBUG formatting costs nothing when DEBUG is
off — this runs on every request.
"""
if log.isEnabledFor(level):
log.log(level, "%s", render(name, fields, trace_id))
def event(level: int, name: str, **fields: Any) -> None:
_emit(level, name, fields)
def debug(name: str, **fields: Any) -> None:
_emit(logging.DEBUG, name, fields)
def info(name: str, **fields: Any) -> None:
_emit(logging.INFO, name, fields)
def warning(name: str, **fields: Any) -> None:
_emit(logging.WARNING, name, fields)
def error(name: str, **fields: Any) -> None:
_emit(logging.ERROR, name, fields)
class Bound:
"""An emitter pinned to one trace id, for code the ContextVar cannot reach.
A StreamingResponse's generator is resumed by starlette through a
threadpool, and every ``next()`` runs in a FRESH COPY of the caller's
context — so a trace id set on one resumption is gone by the next, and the
``finally`` block that logs the dispatch line sees nothing. Measured: the
streamed dispatch line logged ``id=-`` until this existed.
Carrying the id explicitly is the only thing that survives that, and
streaming is the path all agent traffic takes.
"""
__slots__ = ("trace_id",)
def __init__(self, trace_id: str):
self.trace_id = trace_id
def debug(self, name: str, **fields: Any) -> None:
_emit(logging.DEBUG, name, fields, self.trace_id)
def info(self, name: str, **fields: Any) -> None:
_emit(logging.INFO, name, fields, self.trace_id)
def warning(self, name: str, **fields: Any) -> None:
_emit(logging.WARNING, name, fields, self.trace_id)
def error(self, name: str, **fields: Any) -> None:
_emit(logging.ERROR, name, fields, self.trace_id)
def bind(trace_id: Optional[str] = None) -> Bound:
"""Pin an emitter to a trace id, defaulting to the current one."""
return Bound(trace_id or current_trace())
def enabled_for_debug() -> bool:
"""For callers that must do real work to produce debug fields."""
return log.isEnabledFor(logging.DEBUG)
# --- configuration --------------------------------------------------------
class PriorityFormatter(logging.Formatter):
"""Plain message, optionally prefixed with a journald priority."""
def __init__(self, prefix: bool):
super().__init__("%(message)s")
self.prefix = prefix
def format(self, record: logging.LogRecord) -> str:
message = super().format(record)
if not self.prefix:
return message
return f"<{PRIORITIES.get(record.levelno, 6)}>{message}"
def under_journald(stream: Optional[TextIO] = None) -> bool:
"""Whether this stream really is the journal.
systemd sets $JOURNAL_STREAM to the "device:inode" of the stream it
connected, and the variable is INHERITED by every child — including one
whose stderr has since been redirected to a file or a pipe. Presence alone
is therefore a false positive, and it fired immediately: a foreground
uvicorn started from a systemd-managed session, with its output redirected
to a file, wrote a literal <7> on every line.
Comparing the value against the actual fd is what systemd documents, and it
is the only check that answers the real question.
"""
expected = os.environ.get("JOURNAL_STREAM")
if not expected:
return False
device, _, inode = expected.partition(":")
try:
stat = os.fstat((stream if stream is not None else sys.stderr).fileno())
return stat.st_dev == int(device) and stat.st_ino == int(inode)
except (AttributeError, OSError, ValueError):
# No fileno at all (a StringIO under test), or an unparseable value.
return False
def resolve_level(configured: str) -> int:
"""The env var wins, so a live service can be debugged without a repo edit.
config.yaml is the documented home of the setting -- every knob belongs
there -- but flipping it means editing a tracked file and leaving a stray
diff. LLM_ROUTER_LOG_LEVEL goes in a systemd drop-in instead.
"""
override = os.environ.get(LEVEL_ENV)
if override:
level = LEVELS.get(override.strip().lower())
if level is not None:
return level
print(
f"{LEVEL_ENV}={override!r} is not one of {sorted(LEVELS)}; "
f"falling back to logging.level={configured!r}",
file=sys.stderr,
)
return LEVELS.get((configured or "info").strip().lower(), logging.INFO)
def configure(
level: str = "info",
*,
stream: Optional[TextIO] = None,
journald: Optional[bool] = None,
) -> logging.Logger:
"""Install the one handler this logger gets. Idempotent.
``propagate`` is off: uvicorn configures its own loggers and a root handler
installed by anything else would print every line a second time.
"""
resolved = resolve_level(level)
log.setLevel(resolved)
log.propagate = False
for existing in list(log.handlers):
log.removeHandler(existing)
target = stream if stream is not None else sys.stderr
handler = logging.StreamHandler(target)
handler.setLevel(resolved)
handler.setFormatter(
PriorityFormatter(under_journald(target) if journald is None else journald)
)
log.addHandler(handler)
return log

77
opencode.json
View File

@@ -15,6 +15,12 @@
"limit": { "limit": {
"context": 782324, "context": 782324,
"output": 16384 "output": 16384
},
"modalities": {
"input": [
"text",
"image"
]
} }
}, },
"auto:batch": { "auto:batch": {
@@ -22,6 +28,77 @@
"limit": { "limit": {
"context": 782324, "context": 782324,
"output": 16384 "output": 16384
},
"modalities": {
"input": [
"text",
"image"
]
}
},
"deepseek-v4-flash": {
"name": "Deepseek V4 Flash (pin)",
"limit": {
"context": 1048560,
"output": 16384
},
"modalities": {
"input": [
"text",
"image"
]
}
},
"gemma-4-31b": {
"name": "Gemma 4 31B (pin)",
"limit": {
"context": 262128,
"output": 16384
},
"modalities": {
"input": [
"text",
"image"
]
}
},
"kimi-k2.7-code": {
"name": "Kimi K2.7 Code (pin)",
"limit": {
"context": 262128,
"output": 16384
},
"modalities": {
"input": [
"text",
"image"
]
}
},
"kimi-k3": {
"name": "Kimi K3 (pin)",
"limit": {
"context": 1048560,
"output": 16384
},
"modalities": {
"input": [
"text",
"image"
]
}
},
"qwen3.6-35b": {
"name": "Qwen 3.6 35B (pin)",
"limit": {
"context": 131056,
"output": 16384
},
"modalities": {
"input": [
"text",
"image"
]
} }
} }
} }

24
poller.py
View File

@@ -127,12 +127,34 @@ class ModelRow:
deprecated: bool deprecated: bool
def effective_context_window(self, cfg: RouterConfig) -> Optional[int]: def effective_context_window(self, cfg: RouterConfig) -> Optional[int]:
"""Usable context, after the safety factor and an output reserve.
``context.per_model_overrides`` wins where it is set, which is what it
is for: the global factor is a guess that has to hold for the whole
catalog, while a row someone has actually measured deserves its own
number.
Compared with ``is not None`` rather than ``or``, so an override of 0
reserve tokens means zero rather than silently falling through to the
default. Present-but-falsy is a trap this project's own eval set tests
models on; the router should not walk into it.
"""
if not self.context_window: if not self.context_window:
return None return None
override = cfg.context.per_model_overrides.get(self.model_id)
factor = cfg.context.safety_factor
if override is not None and override.safety_factor is not None:
factor = override.safety_factor
# 11 of 19 catalog rows report no max_output_tokens, so the configured # 11 of 19 catalog rows report no max_output_tokens, so the configured
# reserve carries most of the catalog. # reserve carries most of the catalog.
if override is not None and override.output_reserve_tokens is not None:
reserve = override.output_reserve_tokens
else:
reserve = self.max_output_tokens or cfg.context.default_output_reserve_tokens reserve = self.max_output_tokens or cfg.context.default_output_reserve_tokens
usable = int(self.context_window * cfg.context.safety_factor) - reserve
usable = int(self.context_window * factor) - reserve
return max(usable, 0) return max(usable, 0)

97
proficiency_store.py
View File

@@ -23,13 +23,68 @@ def _now() -> str:
return datetime.now(timezone.utc).isoformat() return datetime.now(timezone.utc).isoformat()
def ensure_columns(conn: sqlite3.Connection) -> None:
"""Add a column an older router.db predates. Idempotent, and cheap.
schema.sql is CREATE TABLE IF NOT EXISTS, so it defines a NEW database and
silently does nothing to an existing one. Anything added later therefore
needs this, or the first write against a database created last week fails
with "no such column".
"""
columns = {row[1] for row in conn.execute("PRAGMA table_info(proficiency)")}
if "inherited_from" not in columns:
conn.execute("ALTER TABLE proficiency ADD COLUMN inherited_from TEXT")
conn.commit()
_backfill_inherited(conn)
def _backfill_inherited(conn: sqlite3.Connection) -> None:
"""One-time: mark rows this harness could not have measured directly.
ADD COLUMN gives every existing row NULL, which reads as "measured here"
and would leave exactly the rows this provenance was added for frozen
forever -- the migration would ship the fix and none of the repair.
Provenance that was never recorded cannot be recovered in general. It can
be for the rows that matter, and not by guessing: ``eval_identities``
selects ``latency_class='standard'`` rows, plus flex rows that have NO
standard equivalent. So a flex row WITH a standard equivalent was never a
candidate for direct evaluation, whatever its sample count says. That is
the harness's own selection rule read backwards.
Anything else keeps NULL, which is the safe direction: it means "do not
overwrite", so a real measurement is never lost to this.
"""
pairs = conn.execute(
"""
SELECT v.model_id, v.provider, s.model_id
FROM models v
JOIN models s
ON s.base_model_id = v.base_model_id
AND s.provider = v.provider
AND s.reasoning_mode = v.reasoning_mode
AND s.context_variant = v.context_variant
WHERE v.latency_class = 'flex' AND s.latency_class = 'standard'
"""
).fetchall()
for variant_id, provider, source_id in pairs:
conn.execute(
"""
UPDATE proficiency SET inherited_from = ?
WHERE model_id = ? AND provider = ? AND inherited_from IS NULL
""",
(source_id, variant_id, provider),
)
conn.commit()
def _read_row( def _read_row(
conn: sqlite3.Connection, model_id: str, provider: str, category: str conn: sqlite3.Connection, model_id: str, provider: str, category: str
) -> Optional[sqlite3.Row]: ) -> Optional[sqlite3.Row]:
conn.row_factory = sqlite3.Row conn.row_factory = sqlite3.Row
return conn.execute( return conn.execute(
""" """
SELECT leaderboard_score, self_eval_score, self_eval_samples SELECT leaderboard_score, self_eval_score, self_eval_samples, inherited_from
FROM proficiency FROM proficiency
WHERE model_id = ? AND provider = ? AND category = ? WHERE model_id = ? AND provider = ? AND category = ?
""", """,
@@ -46,7 +101,9 @@ def _write(
leaderboard_score: Optional[float], leaderboard_score: Optional[float],
self_eval_score: Optional[float], self_eval_score: Optional[float],
self_eval_samples: int, self_eval_samples: int,
inherited_from: Optional[str] = None,
) -> None: ) -> None:
ensure_columns(conn)
blended, source = blend( blended, source = blend(
leaderboard_score, leaderboard_score,
self_eval_score, self_eval_score,
@@ -59,14 +116,16 @@ def _write(
""" """
INSERT INTO proficiency ( INSERT INTO proficiency (
model_id, provider, category, leaderboard_score, model_id, provider, category, leaderboard_score,
self_eval_score, self_eval_samples, blended_score, source, last_updated self_eval_score, self_eval_samples, blended_score, source,
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) inherited_from, last_updated
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(model_id, provider, category) DO UPDATE SET ON CONFLICT(model_id, provider, category) DO UPDATE SET
leaderboard_score = excluded.leaderboard_score, leaderboard_score = excluded.leaderboard_score,
self_eval_score = excluded.self_eval_score, self_eval_score = excluded.self_eval_score,
self_eval_samples = excluded.self_eval_samples, self_eval_samples = excluded.self_eval_samples,
blended_score = excluded.blended_score, blended_score = excluded.blended_score,
source = excluded.source, source = excluded.source,
inherited_from = excluded.inherited_from,
last_updated = excluded.last_updated last_updated = excluded.last_updated
""", """,
( (
@@ -78,6 +137,7 @@ def _write(
self_eval_samples, self_eval_samples,
blended, blended,
source, source,
inherited_from,
_now(), _now(),
), ),
) )
@@ -102,6 +162,8 @@ def set_leaderboard(
score, score,
existing["self_eval_score"] if existing else None, existing["self_eval_score"] if existing else None,
existing["self_eval_samples"] if existing else 0, existing["self_eval_samples"] if existing else 0,
# A prior says nothing about where the self-eval half came from.
existing["inherited_from"] if existing else None,
) )
@@ -120,6 +182,14 @@ def add_self_eval(
``self_eval_samples`` keeps meaning "how much evidence stands behind ``self_eval_samples`` keeps meaning "how much evidence stands behind
this", which is what the blending threshold is gating on. this", which is what the blending threshold is gating on.
""" """
# Clamped at the one place every write passes through, which is what this
# module exists to be. score_judge already bounded its output; score_code
# did not, and a harness bug let a model's own demo output push a two-check
# task to 1.50. A blended_score above 1.0 raises `best` in
# rank_candidates and shifts every other candidate's quality band, so the
# damage is not confined to the row that carries the bad number.
scores = [min(1.0, max(0.0, s)) for s in scores]
existing = _read_row(conn, model_id, provider, category) existing = _read_row(conn, model_id, provider, category)
prev_score = existing["self_eval_score"] if existing else None prev_score = existing["self_eval_score"] if existing else None
prev_samples = existing["self_eval_samples"] if existing else 0 prev_samples = existing["self_eval_samples"] if existing else 0
@@ -150,8 +220,15 @@ def propagate_to_variants(
alone. Inheriting across those would attribute reasoning-on quality to a alone. Inheriting across those would attribute reasoning-on quality to a
reasoning-off row. reasoning-off row.
Anything already measured directly keeps its own scores; inheritance is Anything measured DIRECTLY keeps its own scores; inheritance is the
the fallback, never an overwrite. fallback, never an overwrite. A row that was inherited before is refreshed,
which needs ``inherited_from`` to tell the two apart -- both carry
self_eval_samples > 0, so the old "samples > 0 means leave it alone" test
could not distinguish them and froze every variant at its first
inheritance. Observed: kimi-k3 moved from 0.85 (n=2) to 0.957 (n=7) while
kimi-k3-flex sat at 0.85 with a week-old timestamp, and the run reported
"propagated 0 inherited rows". Flex rows serve `auto:batch`, so those
requests were ranking on scores their family had left behind.
Returns the number of rows written. Returns the number of rows written.
""" """
@@ -184,8 +261,13 @@ def propagate_to_variants(
for variant in variants: for variant in variants:
for row in source_rows: for row in source_rows:
existing = _read_row(conn, variant["model_id"], provider, row["category"]) existing = _read_row(conn, variant["model_id"], provider, row["category"])
if existing and (existing["self_eval_samples"] or 0) > 0: measured_here = (
continue # measured directly; do not overwrite with the family's existing is not None
and (existing["self_eval_samples"] or 0) > 0
and not existing["inherited_from"]
)
if measured_here:
continue # its own measurement outranks the family's
_write( _write(
conn, conn,
cfg, cfg,
@@ -195,6 +277,7 @@ def propagate_to_variants(
row["leaderboard_score"], row["leaderboard_score"],
row["self_eval_score"], row["self_eval_score"],
row["self_eval_samples"], row["self_eval_samples"],
source_model_id,
) )
written += 1 written += 1
return written return written

116
routing.py
View File

@@ -10,8 +10,10 @@ Two stages, in order:
that cannot hold the context, is below the required tier, is stale, or is that cannot hold the context, is below the required tier, is stale, or is
not actually reachable by this account is not a cost tradeoff to be not actually reachable by this account is not a cost tradeoff to be
weighed; it is not a candidate at all (design doc §3.2, §4). weighed; it is not a candidate at all (design doc §3.2, §4).
2. **Weighted scoring** (``rank_candidates``) — order what survives by 2. **Ranking** (``rank_candidates``) — order what survives by quality, with
``w_cost*cost + w_eco*eco + w_prof*proficiency``. cost breaking ties inside ``quality_tolerance``. NOT a weighted blend: that
was measured to be nearly inert, since moving the cost weight from 0.4 to
zero changed the winner in only 2 of 6 categories.
The flex filter is the one that is easy to get wrong. NeuralWatt's ``-flex`` The flex filter is the one that is easy to get wrong. NeuralWatt's ``-flex``
rows are the same weights at the same advertised price, so on every scored rows are the same weights at the same advertised price, so on every scored
@@ -31,7 +33,7 @@ INTERACTIVE = "interactive"
BATCH = "batch" BATCH = "batch"
def is_eligible( def rejection_reason(
row: dict, row: dict,
*, *,
required_context_tokens: int, required_context_tokens: int,
@@ -40,35 +42,103 @@ def is_eligible(
allowed_access_levels: Sequence[str], allowed_access_levels: Sequence[str],
exclude_stale: bool, exclude_stale: bool,
exclude_deprecated: bool, exclude_deprecated: bool,
) -> bool: min_tool_proficiency: float | None = None,
"""Whether a model row survives every hard filter. require_vision: bool = False,
require_json_mode: bool = False,
) -> str | None:
"""Why this row is not a candidate, or None if it is one.
Phrased as the reason rather than a bool so the debug log can say WHICH
filter dropped a model and with what numbers, without a second
implementation of these rules drifting away from this one.
Reasons are single tokens so a log line needs no quoting.
A NULL ``effective_context_window`` fails the context filter: an unknown A NULL ``effective_context_window`` fails the context filter: an unknown
window cannot be shown to be large enough, and silently truncating window cannot be shown to be large enough, and silently truncating
mid-task is worse than routing elsewhere. mid-task is worse than routing elsewhere.
``min_tool_proficiency`` is set when the REQUEST carries tool definitions,
and is a capability requirement rather than a preference — which is why it
is a filter and not a term in the ranking.
The distinction that matters: this does not ask whether the task is
"agentic". It asks whether the model can be trusted with tools that are
on the table. Those are different questions, and the measured failure is
the second one — `deepseek-v4-flash` scores 0.33 here, and the recorded
case is a *non*-agentic prompt ("it is 1:20pm, my meeting is at 3pm, how
many minutes?") where it called two tools instead of subtracting. A model
that over-reaches for tools is a hazard on every request where tools
exist, not only on the ones a classifier would label agentic.
That also sidesteps the classifier entirely, which is the point: asked to
identify six unambiguous tool-use prompts, the local models managed 2/6
and 1/6. Whether tools are present is stated in the request body. Reading
it is exact and free; inferring it is neither.
""" """
eff_ctx = row.get("effective_context_window") eff_ctx = row.get("effective_context_window")
if eff_ctx is None or eff_ctx < required_context_tokens: if eff_ctx is None:
return False return "context(unknown)"
if eff_ctx < required_context_tokens:
return f"context({eff_ctx}<{required_context_tokens})"
tier = row.get("tier") tier = row.get("tier")
if tier is None or tier < required_tier: if tier is None:
return False return "tier(unknown)"
if tier < required_tier:
return f"tier({tier}<{required_tier})"
availability = row.get("availability") availability = row.get("availability")
if exclude_stale and availability == "stale": if exclude_stale and availability == "stale":
return False return "stale"
if exclude_deprecated and (availability == "deprecated" or row.get("deprecated")): if exclude_deprecated and (availability == "deprecated" or row.get("deprecated")):
return False return "deprecated"
if row.get("access_level", "public") not in allowed_access_levels: access_level = row.get("access_level", "public")
return False if access_level not in allowed_access_levels:
return f"access_level({access_level})"
# Flex rows may be queued behind a capacity gap; only batch work opts in. # Flex rows may be queued behind a capacity gap; only batch work opts in.
if latency_tolerance == INTERACTIVE and row.get("latency_class") == "flex": if latency_tolerance == INTERACTIVE and row.get("latency_class") == "flex":
return False return "latency_class(flex)"
return True # Capability flags fail CLOSED on unknown, deliberately unlike the tool
# gate just below. A proficiency MEASUREMENT with no evidence is "unproven,
# not bad" -- admitting it loses nothing. But a capability FLAG that is
# absent means "cannot confirm the capability", and routing a request that
# needs vision or JSON mode to a model that might lack it is a guaranteed
# provider 400. Same precedent as the context(unknown) gate above.
if require_vision:
vision = row.get("supports_vision")
if vision is None:
return "vision(unknown)"
if not vision:
return "vision(unsupported)"
if require_json_mode:
jm = row.get("supports_json_mode")
if jm is None:
return "json_mode(unknown)"
if not jm:
return "json_mode(unsupported)"
if min_tool_proficiency is not None:
tool_score = row.get("tool_proficiency")
# Absent evidence does not disqualify, consistently with the tier-1
# context gate: a model nobody has evaluated for tool use yet is
# unproven, not proven bad. Only a measured score below the bar drops
# a candidate.
if tool_score is not None and tool_score < min_tool_proficiency:
return f"tool_proficiency({tool_score:g}<{min_tool_proficiency:g})"
return None
def is_eligible(row: dict, **filters) -> bool:
"""Whether a model row survives every hard filter.
A thin wrapper so there is exactly one copy of the rules; see
``rejection_reason`` for what they are and why.
"""
return rejection_reason(row, **filters) is None
def select_candidates( def select_candidates(
@@ -80,6 +150,9 @@ def select_candidates(
allowed_access_levels: Sequence[str], allowed_access_levels: Sequence[str],
exclude_stale: bool, exclude_stale: bool,
exclude_deprecated: bool, exclude_deprecated: bool,
min_tool_proficiency: float | None = None,
require_vision: bool = False,
require_json_mode: bool = False,
) -> list[dict]: ) -> list[dict]:
"""Apply every hard filter, preserving input order.""" """Apply every hard filter, preserving input order."""
return [ return [
@@ -93,6 +166,9 @@ def select_candidates(
allowed_access_levels=allowed_access_levels, allowed_access_levels=allowed_access_levels,
exclude_stale=exclude_stale, exclude_stale=exclude_stale,
exclude_deprecated=exclude_deprecated, exclude_deprecated=exclude_deprecated,
min_tool_proficiency=min_tool_proficiency,
require_vision=require_vision,
require_json_mode=require_json_mode,
) )
] ]
@@ -222,8 +298,14 @@ def rank_candidates(
# while a real gap (tool_use_agentic spans 0.67) still decides outright. # while a real gap (tool_use_agentic spans 0.67) still decides outright.
best = max(r["proficiency_score"] for r in ranked) best = max(r["proficiency_score"] for r in ranked)
def band(r: dict) -> int: def band(r: dict) -> float:
return int((best - r["proficiency_score"]) / quality_tolerance) # 0 means "no band": rank strictly by quality, with cost breaking only
# exact ties. A legitimate setting -- "never trade quality for cost" --
# and one the config validator has always accepted, while this line
# divided by it.
if quality_tolerance <= 0:
return -r["proficiency_score"]
return float(int((best - r["proficiency_score"]) / quality_tolerance))
ranked.sort( ranked.sort(
key=lambda r: ( key=lambda r: (

38
schema.sql
View File

@@ -66,6 +66,13 @@ CREATE TABLE IF NOT EXISTS proficiency (
self_eval_samples INTEGER DEFAULT 0, self_eval_samples INTEGER DEFAULT 0,
blended_score REAL, -- computed: see blending rule in design doc blended_score REAL, -- computed: see blending rule in design doc
source TEXT, -- 'leaderboard' | 'self_eval' | 'blended' source TEXT, -- 'leaderboard' | 'self_eval' | 'blended'
-- Which model this row's scores were COPIED from, or NULL if they were
-- measured on this row directly. Provenance, not decoration: without it
-- an inherited row is indistinguishable from a measured one (both carry
-- self_eval_samples > 0), so propagate_to_variants could not tell which
-- rows it was allowed to refresh and froze every variant permanently at
-- its first inheritance.
inherited_from TEXT,
last_updated TEXT NOT NULL, last_updated TEXT NOT NULL,
PRIMARY KEY (model_id, provider, category), PRIMARY KEY (model_id, provider, category),
FOREIGN KEY (model_id, provider) REFERENCES models (model_id, provider) FOREIGN KEY (model_id, provider) REFERENCES models (model_id, provider)
@@ -95,23 +102,32 @@ CREATE TABLE IF NOT EXISTS energy_observations (
task_category TEXT, task_category TEXT,
prompt_tokens INTEGER, prompt_tokens INTEGER,
completion_tokens INTEGER, completion_tokens INTEGER,
-- energy_kwh is what NeuralWatt BILLS, but it is not a measure of model -- energy_kwh is what NeuralWatt BILLS. It equals avg_power_watts *
-- efficiency: it equals avg_power_watts * duration_seconds * -- duration_seconds * attribution_ratio, where attribution_ratio is this
-- attribution_ratio, where attribution_ratio is this request's share of a -- request's share of a shared multi-tenant GPU pool. Up close that term
-- shared multi-tenant GPU pool. Eight identical calls to one model inside -- looks like noise: eight identical calls to one model inside one minute
-- one minute varied 20x in energy_kwh and 22x in attribution_ratio, -- varied 20x, correlating +0.997 with the ratio while power and duration
-- correlation +0.997, while duration and power held steady. Ranking models -- held steady.
-- on this ranks how busy the provider was, not how efficient the model is. --
energy_kwh REAL, -- response.energy.energy_kwh (attributed, noisy) -- It is not noise, and this is the ATTRIBUTED figure scoring reads. Across
-- the reference sweep the median ratio spans 750x BETWEEN models while the
-- typical spread WITHIN one is 1.8x, and the values are quantized (0.001,
-- 0.25, 0.5, 0.75) -- that is serving concurrency, a stable per-model
-- property. A model whose GPUs carry more concurrent requests genuinely
-- costs less per request. The median over repeated sweeps absorbs what
-- noise remains.
energy_kwh REAL, -- response.energy.energy_kwh (attributed)
energy_btu REAL, -- energy_kwh * 3412.14, purely for comedic dashboard value energy_btu REAL, -- energy_kwh * 3412.14, purely for comedic dashboard value
-- The pre-attribution terms. avg_power_watts * duration_seconds is the -- The pre-attribution terms. avg_power_watts * duration_seconds is the
-- pool's energy over the request, independent of how many other tenants -- pool's energy over the request, independent of how many other tenants
-- shared it, and it is ~10x more stable (1.85x spread vs 20x on the same -- shared it. Scoring on that product was TRIED and is wrong: it discards
-- eight calls). That product is what scoring ranks on. -- the 750x between-model signal above to suppress a 1.8x within-model one.
-- Kept as a diagnostic (dispatcher.gross_energy_kwh) so the decomposition
-- stays inspectable, not as a scoring input.
avg_power_watts REAL, -- response.energy.avg_power_watts avg_power_watts REAL, -- response.energy.avg_power_watts
duration_seconds REAL, -- response.energy.duration_seconds duration_seconds REAL, -- response.energy.duration_seconds
attribution_ratio REAL, -- kept so the noise term stays inspectable attribution_ratio REAL, -- the share term, kept so the identity checks out
-- Carbon is what design doc §4 actually scores eco on, and NeuralWatt -- Carbon is what design doc §4 actually scores eco on, and NeuralWatt
-- reports it per-request rather than making us derive it. Grid intensity -- reports it per-request rather than making us derive it. Grid intensity

14
scoring.py
View File

@@ -4,8 +4,18 @@ These functions are deliberately free of I/O: no DB reads, no file reads,
no config import at runtime. Thresholds and weights are passed as no config import at runtime. Thresholds and weights are passed as
arguments so the module stays testable and reusable. arguments so the module stays testable and reusable.
Scoring model (design doc §4): What is still live here: ``proficiency_score``, which is the objective
composite = w_cost*cost_score + w_eco*eco_score + w_prof*proficiency_score ``routing.rank_candidates`` orders on, and ``cost_score``, which it reports for
visibility without ranking on it.
The weighted composite below is NOT the scoring model any more. It was
``w_cost*cost + w_eco*eco + w_prof*proficiency`` until measurement retired it:
moving the cost weight from 0.4 to zero changed the winner in only 2 of 6
categories, so 40% of every decision was steering nothing, while eco was
optimizing a goal nobody had stated. Ranking is now quality first, cost as the
tiebreak inside ``quality_tolerance``, and a hard energy ceiling. ``eco_score``
and ``composite_score`` survive as arithmetic with tests; nothing in the
router calls them.
None-handling rule (consistent across cost/eco): candidates without data None-handling rule (consistent across cost/eco): candidates without data
are excluded from the min/max range computation and receive a neutral 0.5. are excluded from the min/max range computation and receive a neutral 0.5.

57
seed_energy.py
View File

@@ -19,10 +19,24 @@ calls to one model inside one minute spanned 20x, correlating +0.997 with
the attribution ratio while power and duration held steady. Two earlier the attribution ratio while power and duration held steady. Two earlier
sweeps of these same 13 models disagreed by up to 36x on that basis. sweeps of these same 13 models disagreed by up to 36x on that basis.
So routing scores on `power * duration` (see `dispatcher.gross_energy_kwh`), Scoring reads the ATTRIBUTED figures, not `power * duration`. Ranking on the
which held to 1.85x across those same eight calls. The summary prints both pre-attribution product was tried and is wrong: the median attribution ratio
spreads side by side; if the billed column is wildly wider than the gross spans 750x between models against a 1.8x spread within one, so stripping it
column, that is the artifact, not the models. discards the larger real signal to suppress the smaller noisy one. The summary
still prints both spreads side by side, because the gap between them is what
that reasoning rests on.
What this sweep feeds today is `eco` and the per-request energy ceiling
(objective.max_energy_per_request). It no longer feeds `cost`: cost is priced
per request from catalog prices scaled to the request's shape, because a fixed
400-token workload ranks models backwards for real traffic.
Coverage across TIME is the point of running it repeatedly. Attribution tracks
pool load and pool load tracks time of day -- between two sweeps hours apart,
deepseek-v4-flash moved ~50x and qwen3.6-35b ~7x the other way, enough to
invert their ranking. More samples inside one sweep measures one moment more
precisely; `load_candidates` takes the median over ALL seed_reference rows, so
repeated sweeps accumulate into a median across time for free.
What lands in the table are real observations of real calls, identical in What lands in the table are real observations of real calls, identical in
kind to what the dispatcher logs; they are simply generated deliberately kind to what the dispatcher logs; they are simply generated deliberately
@@ -77,14 +91,28 @@ def routable_models(conn: sqlite3.Connection, allowed_levels: list[str]) -> list
return [dict(r) for r in rows] return [dict(r) for r in rows]
def sample_once(base_url: str, api_key: str, model_id: str, timeout: int = 300) -> dict: def sample_once(
base_url: str,
api_key: str,
model_id: str,
max_tokens: int = REFERENCE_MAX_TOKENS,
timeout: int = 300,
) -> dict:
"""One reference call. ``max_tokens`` is a parameter because --max-tokens is.
It used to be hardcoded while the banner printed whatever --max-tokens had
been passed, so the flag moved the report and not the request -- and the
resulting rows landed in the same seed_reference median as the 400-token
ones, quietly mixing two workload shapes in the one axis that exists to
hold the workload constant across models.
"""
resp = requests.post( resp = requests.post(
f"{base_url}/chat/completions", f"{base_url}/chat/completions",
headers={"authorization": f"Bearer {api_key}"}, headers={"authorization": f"Bearer {api_key}"},
json={ json={
"model": model_id, "model": model_id,
"messages": [{"role": "user", "content": REFERENCE_PROMPT}], "messages": [{"role": "user", "content": REFERENCE_PROMPT}],
"max_tokens": REFERENCE_MAX_TOKENS, "max_tokens": max_tokens,
"temperature": 0, "temperature": 0,
}, },
timeout=timeout, timeout=timeout,
@@ -140,20 +168,29 @@ def main() -> int:
results[model_id] = [] results[model_id] = []
for i in range(args.samples): for i in range(args.samples):
try: try:
payload = sample_once(settings.base_url, api_key, model_id) payload = sample_once(
settings.base_url, api_key, model_id, args.max_tokens
)
except requests.RequestException as e: except requests.RequestException as e:
print(f" {model_id:26s} sample {i + 1}: FAILED {type(e).__name__}: {e}") print(f" {model_id:26s} sample {i + 1}: FAILED {type(e).__name__}: {e}")
continue continue
usage = payload.get("usage") or {} usage = payload.get("usage") or {}
telemetry = extract_telemetry(payload) telemetry = extract_telemetry(payload)
# Keyword arguments deliberately. log_observation grew request_id,
# session_key and session_dir in the middle of its signature and
# made the trailing three keyword-only; this call still passed six
# positionals, so every sweep died on TypeError after its first
# BILLED call -- and TypeError is not a RequestException, so the
# `except` below never caught it.
log_observation( log_observation(
model_id, model_id,
m["provider"], m["provider"],
SEED_CATEGORY, SEED_CATEGORY,
usage.get("prompt_tokens"), payload.get("id"),
usage.get("completion_tokens"), prompt_tokens=usage.get("prompt_tokens"),
telemetry, completion_tokens=usage.get("completion_tokens"),
telemetry=telemetry,
) )
if telemetry.allowance_remaining_usd is not None: if telemetry.allowance_remaining_usd is not None:
if allowance_start is None: if allowance_start is None:

84
tests/test_capabilities.py Normal file
View File

@@ -0,0 +1,84 @@
"""Tests for capabilities.py — request-side capability detection.
Each case proves one detection fires and that the others stay False, mirroring
the "rejects and does not over-reject" shape of test_routing.py.
"""
from capabilities import detect_capabilities
def test_no_parts_yields_no_capabilities():
caps = detect_capabilities({"messages": [{"content": "plain text"}]})
assert caps == detect_capabilities({})
assert not caps.has_images
assert not caps.require_json_mode
assert not caps.tools_present
assert not caps.has_reasoning_request
def test_image_url_part_sets_has_images():
body = {
"messages": [
{"content": [{"type": "image_url", "url": "https://x/img.png"}]}
]
}
assert detect_capabilities(body).has_images is True
def test_image_in_an_earlier_message_counts():
body = {
"messages": [
{"content": [{"type": "image_url", "url": "https://x/1.png"}]},
{"content": "now what do you make of it?"},
]
}
assert detect_capabilities(body).has_images is True
def test_non_image_parts_do_not_trigger():
body = {
"messages": [
{
"content": [
{"type": "text", "text": "hello"},
{"type": "tool_use", "id": "t1"},
]
}
]
}
assert detect_capabilities(body).has_images is False
def test_string_content_does_not_trigger_images():
# A raw base64 string in a text part is not an image_url part.
body = {"messages": [{"content": "data:image/png;base64,AA=="}]}
assert detect_capabilities(body).has_images is False
def test_json_object_response_format_sets_require_json_mode():
body = {"response_format": {"type": "json_object"}}
assert detect_capabilities(body).require_json_mode is True
def test_json_schema_response_format_sets_require_json_mode():
body = {"response_format": {"type": "json_schema", "json_schema": {}}}
assert detect_capabilities(body).require_json_mode is True
def test_text_or_missing_response_format_does_not_require_json_mode():
assert detect_capabilities({}).require_json_mode is False
assert detect_capabilities({"response_format": {"type": "text"}}).require_json_mode is False
def test_tools_array_sets_tools_present():
assert detect_capabilities({"tools": [{"type": "function"}]}).tools_present is True
# An empty array is no tools at all.
assert detect_capabilities({"tools": []}).tools_present is False
def test_reasoning_effort_sets_has_reasoning_request():
assert detect_capabilities({"reasoning_effort": "high"}).has_reasoning_request is True
def test_reasoning_param_sets_has_reasoning_request():
assert detect_capabilities({"reasoning": {"enabled": True}}).has_reasoning_request is True

807
tests/test_chat_completions.py Normal file
View File

@@ -0,0 +1,807 @@
"""Tests for the OpenAI-compatible surface — the endpoint every client uses.
This file exists because it did not. `/v1/chat/completions` is how opencode,
an SDK and plain curl all reach the router, and it had zero tests, which is
how a NameError survived on the pass-through path: `alternatives` read
`decision.runners_up`, but `decision` is only bound inside `if wants_routing`,
so every non-streaming request naming a real model id returned 500 before the
provider was ever called.
Nothing here touches the network. The classifier, the provider and the local
verifier are all stubbed, so the tests run in milliseconds and pin behaviour
rather than reachability.
"""
import io
import json
import sqlite3
from pathlib import Path
import pytest
from starlette.testclient import TestClient
import dispatcher
from dispatcher import Classification, app
ROOT = Path(__file__).resolve().parent.parent
SCHEMA_SQL = (ROOT / "schema.sql").read_text()
CHEAP = "cheap-model"
DEAR = "dear-model"
class FakeResponse:
"""Just enough of requests.Response for both dispatcher paths."""
def __init__(self, payload=None, *, status_code=200, lines=None):
self.status_code = status_code
self._payload = payload or {}
self._lines = lines or []
self.text = json.dumps(self._payload)
self.closed = False
def json(self):
return self._payload
def iter_lines(self, decode_unicode=False):
yield from self._lines
def close(self):
self.closed = True
def completion(model, content="hello there", *, completion_tokens=12):
"""A provider response shaped like NeuralWatt's, telemetry blocks included."""
return {
"id": "chatcmpl-test-1",
"model": model,
"choices": [
{"message": {"role": "assistant", "content": content},
"finish_reason": "stop"}
],
"usage": {"prompt_tokens": 31, "completion_tokens": completion_tokens},
"energy": {"energy_kwh": 5.0e-05, "avg_power_watts": 400.0,
"duration_seconds": 1.4, "attribution_ratio": 0.25,
"carbon_g_co2eq": 2.4e-03, "carbon_source": "agent_cache",
"grid_id": "FI"},
"cost": {"request_cost_usd": 4.0e-04},
}
@pytest.fixture
def router(tmp_path, monkeypatch):
"""The dispatcher pointed at a throwaway catalog, with nothing dialled out."""
db_path = tmp_path / "test.db"
conn = sqlite3.connect(db_path)
conn.executescript(SCHEMA_SQL)
for model_id, completion_price, vision in (
(CHEAP, 0.30, 1),
(DEAR, 9.00, 0),
):
conn.execute(
"""
INSERT INTO models (
model_id, provider, base_model_id, tier, context_window,
effective_context_window, max_output_tokens,
cost_per_1m_prompt, cost_per_1m_completion,
supports_vision, supports_json_mode,
latency_class, reasoning_mode, context_variant,
access_level, availability, last_updated
) VALUES (?, 'neuralwatt', ?, 2, 262128, 192500, 16384, ?, ?,
?, 1, 'standard', 'default', 'full', 'public', 'active',
'2026-08-22T00:00:00+00:00')
""",
(model_id, model_id, completion_price / 3, completion_price, vision),
)
conn.commit()
conn.close()
monkeypatch.setattr(dispatcher.cfg.database, "path", str(db_path))
# No Ollama in the test environment, and the local check would otherwise
# fire as a background task and try to reach localhost:11434.
monkeypatch.setattr(dispatcher.cfg.verification, "local_llm_enabled", False)
# The local vision fallback is off unless a test opts in; without this it
# would fire for every image request and try to reach localhost.
monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", False)
monkeypatch.setenv("NEURALWATT_API_KEY", "test-key")
calls = []
def fake_post(url, headers=None, json=None, stream=False, timeout=None):
calls.append({"url": url, "body": json, "stream": stream})
if stream:
return FakeResponse(lines=STREAM_LINES)
return FakeResponse(completion(json["model"]))
monkeypatch.setattr(dispatcher.requests, "post", fake_post)
monkeypatch.setattr(
dispatcher, "classify",
lambda task, context: Classification(
task_category="coding_general", task_tier=2,
required_context_tokens=100, confidence=0.9,
),
)
yield TestClient(app), calls, db_path
STREAM_LINES = [
'data: {"id":"chatcmpl-stream-1","choices":[{"delta":{"content":"hel"}}]}',
"",
'data: {"id":"chatcmpl-stream-1","choices":[{"delta":{"content":"lo"},'
'"finish_reason":"stop"}],"usage":{"prompt_tokens":31,'
'"completion_tokens":9}}',
"",
': energy {"energy_kwh": 5e-05, "carbon_g_co2eq": 0.0024, '
'"carbon_source": "agent_cache"}',
': cost {"request_cost_usd": 0.0004}',
"data: [DONE]",
"",
]
def _messages(text="write me a function"):
return [{"role": "user", "content": text}]
# --- the regression -------------------------------------------------------
def test_a_real_model_id_is_dispatched_rather_than_routed(router):
"""The documented pass-through: 'any real model id — dispatched as asked'.
This raised NameError on `decision` before the provider was ever called,
so the whole path 500'd while streaming clients never noticed.
"""
client, calls, _ = router
resp = client.post(
"/v1/chat/completions",
json={"model": DEAR, "messages": _messages()},
)
assert resp.status_code == 200
assert calls[0]["body"]["model"] == DEAR, "the caller's choice must survive"
assert resp.headers["X-Router-Model"] == DEAR
def test_a_real_model_id_does_not_pay_for_classification(router, monkeypatch):
"""A named model has nothing to classify; the ~2s round-trip is skipped."""
client, _, _ = router
def explode(task, context):
raise AssertionError("classifier consulted for an explicit model id")
monkeypatch.setattr(dispatcher, "classify", explode)
assert client.post(
"/v1/chat/completions", json={"model": CHEAP, "messages": _messages()}
).status_code == 200
# --- routing --------------------------------------------------------------
def test_auto_routes_and_reports_the_model_it_actually_used(router):
"""Quality ties, so cost breaks it — and the client is told what ran."""
client, calls, _ = router
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _messages()},
)
assert resp.status_code == 200
assert calls[0]["body"]["model"] == CHEAP
assert resp.headers["X-Router-Model"] == CHEAP
# The body must stay a valid OpenAI response naming the real model, not 'auto'.
assert resp.json()["model"] == CHEAP
def test_a_provider_prefixed_router_name_still_routes(router):
"""opencode sends `llm-router/auto`; only the virtual names are stripped."""
client, calls, _ = router
resp = client.post(
"/v1/chat/completions",
json={"model": "llm-router/auto", "messages": _messages()},
)
assert resp.status_code == 200
assert calls[0]["body"]["model"] == CHEAP
def test_no_eligible_model_is_a_422_naming_the_filters(router, monkeypatch):
"""A dead end must say which constraint killed it, not just 'no model'."""
client, calls, _ = router
monkeypatch.setattr(
dispatcher, "classify",
lambda task, context: Classification(
task_category="coding_general", task_tier=3,
required_context_tokens=100, confidence=0.9,
),
)
resp = client.post(
"/v1/chat/completions", json={"model": "auto", "messages": _messages()}
)
assert resp.status_code == 422
assert "tier >= 3" in resp.json()["detail"]
assert not calls, "nothing should be dispatched when nothing qualifies"
# --- streaming ------------------------------------------------------------
def test_a_stream_is_proxied_verbatim_including_the_telemetry_comments(router):
"""NeuralWatt's energy/cost blocks are SSE comments; clients ignore them.
They must still reach the client untouched — the router reads them on the
way past rather than buffering the stream to strip them.
"""
client, calls, _ = router
resp = client.post(
"/v1/chat/completions",
json={"model": DEAR, "messages": _messages(), "stream": True},
)
assert resp.status_code == 200
assert calls[0]["stream"] is True
body = resp.text
assert "hel" in body and "lo" in body
assert ': energy {"energy_kwh": 5e-05' in body
assert "data: [DONE]" in body
def test_a_streamed_call_still_logs_its_energy(router):
"""Streaming is how every agent client talks; unlogged, it is most of the traffic."""
client, _, db_path = router
client.post(
"/v1/chat/completions",
json={"model": DEAR, "messages": _messages(), "stream": True},
)
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
row = conn.execute(
"SELECT model_id, request_id, energy_kwh, cost_usd, completion_tokens "
"FROM energy_observations ORDER BY id DESC LIMIT 1"
).fetchone()
conn.close()
assert row["model_id"] == DEAR
assert row["request_id"] == "chatcmpl-stream-1"
assert row["energy_kwh"] == pytest.approx(5e-05)
assert row["cost_usd"] == pytest.approx(4e-04)
assert row["completion_tokens"] == 9
def test_a_dropped_upstream_connection_ends_the_stream_cleanly(router, monkeypatch):
"""The upstream connection can die mid-stream (NeuralWatt closing early, a
network blip). Left uncaught, requests.exceptions.ChunkedEncodingError
propagated straight out of the generator, which Starlette surfaced as an
unhandled ASGI exception -- a full traceback in the log, and the client's
connection cut dead with no error payload or [DONE].
"""
import requests as requests_module
client, _, _ = router
def broken_lines():
yield 'data: {"id":"chatcmpl-stream-1","choices":[{"delta":{"content":"hel"}}]}'
raise requests_module.exceptions.ChunkedEncodingError("Response ended prematurely")
def fake_post(url, headers=None, json=None, stream=False, timeout=None):
return FakeResponse(lines=broken_lines())
monkeypatch.setattr(dispatcher.requests, "post", fake_post)
resp = client.post(
"/v1/chat/completions",
json={"model": DEAR, "messages": _messages(), "stream": True},
)
assert resp.status_code == 200
body = resp.text
assert "hel" in body
assert "upstream stream interrupted" in body
assert "data: [DONE]" in body
# --- what a request leaves in the journal ---------------------------------
#
# The reason this exists: watching a live session showed only uvicorn's access
# line, which does not even name the model that served it.
@pytest.fixture
def logbuf():
"""Capture what would reach the journal, through the real formatter."""
import logs
buf = io.StringIO()
logs.configure("debug", stream=buf, journald=False)
yield buf
for handler in list(logs.log.handlers):
logs.log.removeHandler(handler)
def _lines(logbuf, event):
return [ln for ln in logbuf.getvalue().splitlines() if ln.startswith(f"{event} ")]
def _fields(line):
out = {}
for token in line.split(" ")[1:]:
if "=" in token:
key, _, value = token.partition("=")
out[key] = value
return out
def test_a_routed_request_logs_exactly_one_decision_line(router, logbuf):
client, _, _ = router
client.post("/v1/chat/completions",
json={"model": "auto", "messages": _messages()})
routes = _lines(logbuf, "route")
assert len(routes) == 1, "routing twice for context must not log twice"
fields = _fields(routes[0])
assert fields["pick"] == CHEAP
assert fields["cat"] == "coding_general"
assert fields["tier"] == "2"
assert int(fields["ms"]) >= 0
def test_the_dispatch_line_carries_the_join_key(router, logbuf):
"""rid is what pivots a journal line to its energy_observations row."""
client, _, _ = router
client.post("/v1/chat/completions",
json={"model": "auto", "messages": _messages()})
fields = _fields(_lines(logbuf, "dispatch")[0])
assert fields["rid"] == "chatcmpl-test-1"
assert fields["model"] == CHEAP
assert fields["c_tok"] == "12"
def test_every_line_of_one_request_shares_a_trace_id(router, logbuf):
client, _, _ = router
client.post("/v1/chat/completions",
json={"model": "auto", "messages": _messages()})
ids = {_fields(ln)["id"] for ln in logbuf.getvalue().splitlines()}
assert len(ids) == 1
assert ids != {"-"}
def test_a_streamed_request_still_has_a_trace_id(router, logbuf):
"""The trap: a StreamingResponse generator runs in a different context.
Read from the ContextVar inside the generator it comes back empty, and
every streamed request -- which is all agent traffic -- logs a blank id.
"""
client, _, _ = router
client.post("/v1/chat/completions",
json={"model": DEAR, "messages": _messages(), "stream": True})
dispatched = _fields(_lines(logbuf, "dispatch")[0])
assert dispatched["id"] != "-"
assert dispatched["stream"] == "1"
assert dispatched["rid"] == "chatcmpl-stream-1"
def test_debug_says_which_filter_dropped_a_model(router, logbuf):
"""'No model satisfies the hard filters' is otherwise a dead end."""
client, _, db_path = router
conn = sqlite3.connect(db_path)
conn.execute(
"""
INSERT INTO models (
model_id, provider, tier, context_window, effective_context_window,
latency_class, access_level, availability, last_updated
) VALUES ('held-model', 'neuralwatt', 2, 262128, 192500, 'flex',
'public', 'active', '2026-08-22T00:00:00+00:00')
"""
)
conn.commit()
conn.close()
client.post("/v1/chat/completions",
json={"model": "auto", "messages": _messages()})
dropped = {_fields(ln)["model"]: _fields(ln)["reason"]
for ln in _lines(logbuf, "filter")}
assert dropped["held-model"] == "latency_class(flex)"
def test_a_pass_through_is_labelled_as_one(router, logbuf):
client, _, _ = router
client.post("/v1/chat/completions",
json={"model": DEAR, "messages": _messages()})
assert not _lines(logbuf, "route"), "nothing was routed"
assert _fields(_lines(logbuf, "passthrough")[0])["model"] == DEAR
def test_no_conversation_text_ever_reaches_the_log(router, logbuf):
"""Prompts here run 60k-150k tokens and the journal is on disk.
Asserted at DEBUG, the most verbose level, on both the request and the
answer.
"""
secret = "SQUAMOUS-EPHEMERAL-9137"
client, _, _ = router
client.post("/v1/chat/completions",
json={"model": "auto", "messages": _messages(f"refactor {secret}")})
assert secret not in logbuf.getvalue()
# The stubbed provider answers "hello there"; that must not appear either.
assert "hello there" not in logbuf.getvalue()
def test_the_decision_line_reports_the_real_provenance(router, logbuf):
"""The re-route for measured context must not read as a client override.
chat_completions routes a second time when the conversation measures larger
than the classifier guessed, passing category and tier back in — which
makes the resulting Classification say source='override'. In a log line
that means "the client chose this", and the client chose nothing: the
classifier ran and only the context figure was replaced.
"""
client, _, _ = router
# ~1200 chars / 3 = 400 tokens measured, above the stubbed estimate of 100.
client.post("/v1/chat/completions",
json={"model": "auto", "messages": _messages("refactor " + "x " * 600)})
fields = _fields(_lines(logbuf, "route")[0])
assert fields["src"] == "classifier", "the classifier did decide this"
assert fields["ctx_src"] == "measured", "but the context came from measurement"
assert int(fields["ctx"]) > 100
def test_a_caller_supplied_context_says_so(router, logbuf):
client, _, _ = router
client.post("/route", json={"task": "refactor this",
"task_category": "coding_general",
"task_tier": 2,
"required_context_tokens": 5000})
fields = _fields(_lines(logbuf, "route")[0])
assert fields["src"] == "override"
assert fields["ctx_src"] == "caller"
# --- capability gates: vision and JSON mode -------------------------------
def _image_messages(text="what is in this image?"):
"""A user turn carrying an inline image_url part, OpenAI multimodal shape."""
return [
{
"role": "user",
"content": [
{"type": "text", "text": text},
{
"type": "image_url",
"image_url": {"url": "data:image/png;base64,AAAA"},
},
],
}
]
def _drop_cheap_by_tier(client_type, db_path):
"""Make CHEAP ineligible so DEAR is the only remaining candidate."""
conn = sqlite3.connect(db_path)
conn.execute("UPDATE models SET tier = 1 WHERE model_id = ?", (CHEAP,))
conn.commit()
conn.close()
def _local_vision_fake(monkeypatch, router_calls, content="local caption", status=200):
"""Point the local vision fallback at a fake that returns `content`."""
def fake_post(url, headers=None, json=None, stream=False, timeout=None):
if "/chat/completions" in url:
router_calls.append({"url": url, "body": json, "stream": stream, "local": True})
return FakeResponse(
{
"choices": [
{"message": {"role": "assistant", "content": content},
"finish_reason": "stop"}
]
},
status_code=status,
)
router_calls.append({"url": url, "body": json, "stream": stream})
return FakeResponse(completion(json["model"]))
monkeypatch.setattr(dispatcher.requests, "post", fake_post)
def test_an_image_request_is_routed_to_a_vision_model(router):
"""Image parts hard-restrict to vision-capable rows; CHEAP has vision."""
client, calls, _ = router
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _image_messages()},
)
assert resp.status_code == 200
assert calls[0]["body"]["model"] == CHEAP
assert resp.headers["X-Router-Model"] == CHEAP
def test_an_image_request_excludes_a_non_vision_candidate(router):
"""DEAR is eligible-tiered but lacks vision, so no candidate meets the ask."""
client, calls, db_path = router
_drop_cheap_by_tier(dispatcher, db_path)
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _image_messages()},
)
assert resp.status_code == 422
assert "vision" in resp.json()["detail"]
assert not calls, "a non-vision model must never be dispatched for an image"
def test_no_vision_model_uses_local_fallback_when_enabled(router, monkeypatch):
"""No cloud vision candidate, local vision on: the local model answers."""
client, calls, db_path = router
_drop_cheap_by_tier(dispatcher, db_path)
monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True)
_local_vision_fake(monkeypatch, calls, content="local caption")
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _image_messages()},
)
assert resp.status_code == 200
assert resp.json()["choices"][0]["message"]["content"] == "local caption"
local = [c for c in calls if c["url"].endswith("/chat/completions")]
assert local, "the local fallback should POST to its own /chat/completions"
parts = local[0]["body"]["messages"][0]["content"]
assert any(
isinstance(p, dict) and p.get("type") == "image_url" for p in parts
), "the image_url part must survive into the local call"
def test_local_fallback_respects_streaming(router, monkeypatch):
client, calls, db_path = router
_drop_cheap_by_tier(dispatcher, db_path)
monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True)
_local_vision_fake(monkeypatch, calls, content="streamed caption")
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _image_messages(), "stream": True},
)
assert resp.status_code == 200
assert "streamed caption" in resp.text
assert "data: [DONE]" in resp.text
def test_local_fallback_failure_still_422s(router, monkeypatch):
"""A local vision failure must be visible, not swallowed into a 200."""
client, calls, db_path = router
_drop_cheap_by_tier(dispatcher, db_path)
monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True)
_local_vision_fake(monkeypatch, calls, status=500)
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _image_messages()},
)
assert resp.status_code == 422
assert "vision" in resp.json()["detail"]
def test_a_json_object_response_format_routes_to_a_json_capable_model(router):
"""response_format json_object hard-restricts to JSON-mode-capable rows."""
client, calls, _ = router
resp = client.post(
"/v1/chat/completions",
json={
"model": "auto",
"messages": _messages(),
"response_format": {"type": "json_object"},
},
)
assert resp.status_code == 200
assert calls[0]["body"]["model"] == CHEAP
def test_a_json_object_request_without_a_json_capable_model_422s(router):
client, calls, db_path = router
conn = sqlite3.connect(db_path)
conn.execute("UPDATE models SET supports_json_mode = 0")
conn.commit()
conn.close()
resp = client.post(
"/v1/chat/completions",
json={
"model": "auto",
"messages": _messages(),
"response_format": {"type": "json_object"},
},
)
assert resp.status_code == 422
assert "json" in resp.json()["detail"]
assert not calls
def test_a_plain_text_request_is_unaffected_by_the_gates(router):
"""No images, no response_format: routing is exactly as before."""
client, calls, _ = router
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _messages()},
)
assert resp.status_code == 200
assert calls[0]["body"]["model"] == CHEAP
assert resp.headers["X-Router-Model"] == CHEAP
def test_a_pinned_non_vision_model_with_an_image_is_a_clear_422(router):
"""A pin that cannot satisfy the request fails early with a named reason."""
client, calls, _ = router
resp = client.post(
"/v1/chat/completions",
json={"model": DEAR, "messages": _image_messages()},
)
assert resp.status_code == 422
assert "vision" in resp.json()["detail"]
assert not calls, "no provider call should happen for an impossible pin"
def test_the_two_pass_reroute_passes_image_capability(router):
"""The measured-context reroute must carry the image gate, not lose it."""
client, calls, _ = router
# ~1200 chars / 3 = 400 measured tokens > the stubbed estimate of 100, so
# chat_completions reroutes — and the reroute must still require vision.
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _image_messages("refactor " + "x " * 600)},
)
assert resp.status_code == 200
assert resp.headers["X-Router-Model"] == CHEAP
assert calls[0]["body"]["model"] == CHEAP
def test_local_fallback_refuses_a_remote_image_url(router, monkeypatch):
"""A remote http(s) URL is an SSRF vector by indirection and must not be
forwarded to the local model, even when the fallback is enabled."""
client, calls, db_path = router
_drop_cheap_by_tier(dispatcher, db_path)
monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True)
_local_vision_fake(monkeypatch, calls, content="should not be used")
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "describe this"},
{
"type": "image_url",
"image_url": {"url": "http://internal/private.png"},
},
],
}
]
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": messages},
)
assert resp.status_code == 422
assert "vision" in resp.json()["detail"]
local = [c for c in calls if c.get("local")]
assert not local, "a remote image URL must never reach the local vision endpoint"
def test_local_fallback_wont_masquerade_an_empty_answer_as_200(router, monkeypatch):
"""A 200 with empty content is a failed answer, and must fall through to 422
rather than return an empty 200 — a hidden failure must not look like a win."""
client, calls, db_path = router
_drop_cheap_by_tier(dispatcher, db_path)
monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True)
_local_vision_fake(monkeypatch, calls, content=" ")
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _image_messages()},
)
assert resp.status_code == 422
assert "vision" in resp.json()["detail"]
def test_local_fallback_refuses_too_many_images(router, monkeypatch):
"""An image count above the budget refuses the local call before spending it."""
client, calls, db_path = router
_drop_cheap_by_tier(dispatcher, db_path)
monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True)
_local_vision_fake(monkeypatch, calls, content="caption")
many = _image_messages()
for _ in range(dispatcher.cfg.local_vision.max_images + 1):
many[0]["content"].append({"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}})
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": many},
)
assert resp.status_code == 422
local = [c for c in calls if c.get("local")]
assert not local, "over-budget image count must skip the local call"
def test_local_fallback_refuses_when_api_key_env_is_missing(router, monkeypatch):
"""A configured local vision api_key_env with no env var declines the call."""
client, calls, db_path = router
_drop_cheap_by_tier(dispatcher, db_path)
monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True)
monkeypatch.setattr(dispatcher.cfg.local_vision, "api_key_env", "LOCAL_VISION_KEY")
monkeypatch.delenv("LOCAL_VISION_KEY", raising=False)
_local_vision_fake(monkeypatch, calls, content="caption")
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _image_messages()},
)
assert resp.status_code == 422
local = [c for c in calls if c.get("local")]
assert not local, "a missing vision key must skip the local call"
def test_local_fallback_refuses_an_unparseable_response(router, monkeypatch):
"""An unparseable local response body must not yield a 200."""
client, calls, db_path = router
_drop_cheap_by_tier(dispatcher, db_path)
monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True)
def fake_post(url, headers=None, json=None, stream=False, timeout=None):
calls.append({"url": url, "body": json, "stream": stream, "local": True})
return FakeResponse({}, status_code=200)
monkeypatch.setattr(dispatcher.requests, "post", fake_post)
resp = client.post(
"/v1/chat/completions",
json={"model": "auto", "messages": _image_messages()},
)
assert resp.status_code == 422
assert "vision" in resp.json()["detail"]
def test_a_pinned_non_json_model_with_json_response_format_422s(router):
"""The JSON-mode arm of the pass-through check mirrors the vision one: a pin
whose model lacks JSON mode gets a clear 422, not a provider 400."""
client, calls, db_path = router
conn = sqlite3.connect(db_path)
conn.execute("UPDATE models SET supports_json_mode = 0 WHERE model_id = ?", (DEAR,))
conn.commit()
conn.close()
resp = client.post(
"/v1/chat/completions",
json={
"model": DEAR,
"messages": _messages(),
"response_format": {"type": "json_object"},
},
)
assert resp.status_code == 422
assert "json" in resp.json()["detail"]
assert not calls, "no provider call should happen for an impossible json pin"

63
tests/test_classifier_input.py Normal file
View File

@@ -0,0 +1,63 @@
"""The classifier gets the instruction, not the document.
It decides a category and a tier. Handing it a pasted document is harmful
rather than merely wasteful — measured on a ~20k-token prompt, both local
models failed and neither failed cleanly:
qwen3.5 28.7s, finish_reason=length, empty content
mistral-nemo 41.8s, echoed the input back inside its JSON
Both surface as unparseable output, i.e. 30-40s of local inference spent to
reach the same `source: "fallback"` an instant failure would have produced.
With the clamp, the same prompts classify correctly in ~2.2s.
"""
from __future__ import annotations
from dispatcher import clamp_for_classifier
def test_short_input_is_untouched():
assert clamp_for_classifier("classify me", 8000) == "classify me"
def test_input_at_the_limit_is_untouched():
text = "x" * 8000
assert clamp_for_classifier(text, 8000) == text
def test_the_instruction_survives_at_either_end():
# Which end carries the instruction depends on how the caller phrased it:
# "Translate this: <doc>" puts it first, "<doc> — translate this" last.
# The middle of a pasted document is the one part that never carries it.
text = "HEAD-INSTRUCTION " + ("filler " * 20000) + " TAIL-INSTRUCTION"
out = clamp_for_classifier(text, 8000)
assert "HEAD-INSTRUCTION" in out
assert "TAIL-INSTRUCTION" in out
def test_clamped_output_is_bounded():
text = "y" * 500_000
out = clamp_for_classifier(text, 8000)
# Head + tail + the elision marker, not the original.
assert len(out) < 8200
assert len(out) < len(text)
def test_the_elision_is_visible_to_the_model():
# The model should be able to tell it is seeing an excerpt, rather than
# silently reasoning about a document that appears to jump mid-sentence.
out = clamp_for_classifier("z" * 50_000, 8000)
assert "elided" in out
def test_zero_disables_clamping():
# Escape hatch: a deployment with a large-context classifier and a reason
# to use it should not have to patch code.
text = "w" * 100_000
assert clamp_for_classifier(text, 0) == text
def test_negative_is_treated_as_disabled_not_as_a_crash():
text = "w" * 100_000
assert clamp_for_classifier(text, -1) == text

232
tests/test_config_endpoints.py Normal file
View File

@@ -0,0 +1,232 @@
"""The classifier and the local verifier are separate endpoints.
They used to be one: the verifier derived its URL by stripping ``/v1`` off
``classifier.base_url``. That silently coupled two unrelated decisions, and
the coupling only becomes visible when the classifier is moved off-host —
pointing classification at a cloud provider would have sent every local
verification to ``<provider>/api/chat``, which does not exist.
The verifier speaks Ollama's NATIVE API (``/api/chat`` with ``think=False``)
because that is the only way to disable the reasoning trace, so it cannot
follow the classifier anywhere the classifier can go.
"""
from __future__ import annotations
import copy
import pytest
import yaml
from config import RouterConfig
@pytest.fixture
def raw() -> dict:
with open("config.yaml") as fh:
return yaml.safe_load(fh)
def test_moving_the_classifier_to_a_cloud_provider_leaves_the_verifier_local(raw):
cfg = copy.deepcopy(raw)
cfg["classifier"]["base_url"] = "https://api.neuralwatt.com/v1"
cfg["classifier"]["api_key_env"] = "NEURALWATT_API_KEY"
cfg["classifier"]["model"] = "deepseek-v4-flash"
loaded = RouterConfig(**cfg)
assert loaded.classifier.base_url == "https://api.neuralwatt.com/v1"
# The whole point: this did NOT follow the line above.
assert "localhost" in loaded.verification.base_url
def test_the_verifier_defaults_to_a_local_ollama(raw):
cfg = copy.deepcopy(raw)
cfg["verification"].pop("base_url", None)
assert RouterConfig(**cfg).verification.base_url == "http://localhost:11434"
def test_an_absent_api_key_env_means_unauthenticated(raw):
# The local Ollama case. It ignores the key entirely, but the SDK requires
# one to be set, so the dispatcher substitutes a placeholder. Asserted on
# an explicit config rather than on whatever config.yaml currently says,
# which is a deployment choice and not a property of the code.
cfg = copy.deepcopy(raw)
cfg["classifier"]["base_url"] = "http://localhost:11434/v1"
cfg["classifier"].pop("api_key_env", None)
assert RouterConfig(**cfg).classifier.api_key_env is None
cfg["classifier"]["api_key_env"] = "NEURALWATT_API_KEY"
assert RouterConfig(**cfg).classifier.api_key_env == "NEURALWATT_API_KEY"
def test_verification_model_may_be_unset_while_both_run_on_one_host(raw):
cfg = copy.deepcopy(raw)
cfg["classifier"]["base_url"] = "http://localhost:11434/v1"
cfg["verification"]["model"] = None
loaded = RouterConfig(**cfg)
# Resolved by the dispatcher as `verification.model or classifier.model`,
# which is right only because the hosts match.
assert loaded.verification.model is None
def test_an_unset_verifier_model_is_refused_once_the_hosts_differ(raw):
# The failure this prevents is SILENT, which is why it is a load-time
# error rather than a documented caveat. Observed directly: with the
# classifier on NeuralWatt and this left null, the verifier POSTed
# `deepseek-v4-flash` to localhost:11434, 404d, caught it, logged "local
# verification unavailable" and recorded no sample. Verification looked
# enabled while producing nothing.
cfg = copy.deepcopy(raw)
cfg["classifier"]["base_url"] = "https://api.neuralwatt.com/v1"
cfg["classifier"]["model"] = "deepseek-v4-flash"
cfg["verification"]["model"] = None
cfg["verification"]["local_llm_enabled"] = True
with pytest.raises(ValueError, match="verification.model must be set"):
RouterConfig(**cfg)
def test_a_split_host_setup_loads_once_the_verifier_model_is_stated(raw):
cfg = copy.deepcopy(raw)
cfg["classifier"]["base_url"] = "https://api.neuralwatt.com/v1"
cfg["classifier"]["api_key_env"] = "NEURALWATT_API_KEY"
cfg["classifier"]["model"] = "deepseek-v4-flash"
cfg["verification"]["model"] = "qwen3.5:latest"
loaded = RouterConfig(**cfg)
assert loaded.classifier.model == "deepseek-v4-flash"
assert loaded.verification.model == "qwen3.5:latest"
def test_disabling_the_local_check_lifts_the_requirement(raw):
# A host with no local inference at all: nothing to name, nothing to guard.
cfg = copy.deepcopy(raw)
cfg["classifier"]["base_url"] = "https://api.neuralwatt.com/v1"
cfg["verification"]["model"] = None
cfg["verification"]["local_llm_enabled"] = False
loaded = RouterConfig(**cfg)
assert loaded.verification.local_llm_enabled is False
def test_a_host_with_no_local_inference_can_turn_the_local_check_off(raw):
# An RPi has no usable local model. Structural verification is pure Python
# and keeps running; only the LLM check goes away.
cfg = copy.deepcopy(raw)
cfg["verification"]["local_llm_enabled"] = False
loaded = RouterConfig(**cfg)
assert loaded.verification.local_llm_enabled is False
# --- an unknown key is an error, not a no-op ------------------------------
def test_a_key_in_the_wrong_section_is_rejected(raw):
# Exactly the mistake that shipped: max_input_chars belongs to the
# classifier and was written into verification, where pydantic's default
# extra="ignore" accepted it, dropped it, and left the code default in
# force. It carried the same value, so nothing looked wrong -- but editing
# it would have done nothing at all.
cfg = copy.deepcopy(raw)
cfg["verification"]["max_input_chars"] = 4000
with pytest.raises(ValueError, match="max_input_chars"):
RouterConfig(**cfg)
def test_a_misspelled_key_is_rejected(raw):
cfg = copy.deepcopy(raw)
cfg["routing"]["min_tool_proficency"] = 0.5 # sic
with pytest.raises(ValueError, match="min_tool_proficency"):
RouterConfig(**cfg)
def test_the_shipped_config_has_no_unknown_keys(raw):
# Guards the whole file, not just the sections a test happens to name.
RouterConfig(**raw)
def test_the_tool_filter_can_be_turned_off_from_config(raw):
# It is a knob to experiment with, so null must be a legal value rather
# than something requiring a code change.
cfg = copy.deepcopy(raw)
cfg["routing"]["min_tool_proficiency"] = None
assert RouterConfig(**cfg).routing.min_tool_proficiency is None
def test_disabling_the_filter_lifts_the_category_name_check(raw):
# With no filter there is nothing to join, so an unused category name
# must not block startup.
cfg = copy.deepcopy(raw)
cfg["routing"]["min_tool_proficiency"] = None
cfg["routing"]["tool_use_category"] = "not_a_real_category"
assert RouterConfig(**cfg).routing.min_tool_proficiency is None
# --- the CLI sanity check -------------------------------------------------
def test_the_config_summary_names_fields_that_exist(raw):
"""`python config.py` is the documented setup step, and it crashed.
It printed cfg.weights, which had been replaced by cfg.objective, so the
one command whose job is to prove the config is fine reported "Config
loaded OK" and then died with AttributeError. A function plus this test
means the names cannot rot silently again.
"""
from config import summary_lines
lines = summary_lines(RouterConfig(**raw))
assert lines[0] == "Config loaded OK"
body = "\n".join(lines[1:])
assert "quality_tolerance=" in body
assert "classifier:" in body
assert "dispatch providers:" in body
def test_a_removed_key_is_rejected_rather_than_ignored(raw):
"""log_path named a file nothing ever wrote; leaving it valid would lie."""
import copy
import pytest
cfg = copy.deepcopy(raw)
cfg["logging"]["log_path"] = "router.log"
with pytest.raises(Exception, match="log_path|Extra inputs"):
RouterConfig(**cfg)
# --- capability gates and the local vision fallback ------------------------
def test_the_shipped_config_loads_with_the_new_keys(raw):
loaded = RouterConfig(**raw)
assert loaded.routing.require_vision is True
assert loaded.routing.require_json_mode is True
assert loaded.local_vision.model == "qwen3-vl:4b"
assert loaded.local_vision.enabled is False
def test_a_typo_in_require_vision_is_rejected(raw):
cfg = copy.deepcopy(raw)
cfg["routing"]["require_visoin"] = True # sic
with pytest.raises(ValueError, match="require_visoin"):
RouterConfig(**cfg)
def test_local_vision_can_be_disabled_from_config(raw):
cfg = copy.deepcopy(raw)
cfg["local_vision"]["enabled"] = False
assert RouterConfig(**cfg).local_vision.enabled is False
def test_local_vision_defaults_when_absent(raw):
cfg = copy.deepcopy(raw)
cfg.pop("local_vision")
assert RouterConfig(**cfg).local_vision.enabled is False
def test_nonpositive_local_timeout_is_rejected(raw):
cfg = copy.deepcopy(raw)
cfg["local_vision"]["timeout_seconds"] = 0
with pytest.raises(ValueError, match="timeout_seconds"):
RouterConfig(**cfg)

91
tests/test_eval_scoring.py
View File

@@ -13,6 +13,8 @@ point of testing it.
import pytest import pytest
from eval_proficiency import ( from eval_proficiency import (
judge_for,
parse_check_results,
extract_judge_json, extract_judge_json,
judge_for, judge_for,
normalize_answer, normalize_answer,
@@ -207,3 +209,92 @@ def test_a_single_leading_space_does_not_destroy_a_correct_answer():
def test_indented_fenced_block_is_still_recovered(): def test_indented_fenced_block_is_still_recovered():
code = ' ```python\ndef f(x):\n return x * 2\n```' code = ' ```python\ndef f(x):\n return x * 2\n```'
assert score_code(code, ["f(2)==4"])[0] == 1.0 assert score_code(code, ["f(2)==4"])[0] == 1.0
# --- harness integrity ----------------------------------------------------
#
# The harness runs the model's code as `__main__`, so anything the model prints
# lands in the same stdout the verdicts are read from. Counting the substring
# "PASS" therefore let a model vote on its own work.
def test_a_models_own_demo_output_cannot_award_it_marks():
"""Measured at 1.50 on a two-check task before the verdicts carried a nonce."""
code = (
"def add(a, b):\n"
" return a + b\n"
"\n"
'if __name__ == "__main__":\n'
' print("self-test:", add(1, 2) == 3 and "PASS" or "FAIL")\n'
)
score, detail = score_code(code, ["add(1,2)==3", "add(-1,1)==0"])
assert score == 1.0
assert detail == "2/2 checks"
def test_printing_PASS_cannot_rescue_a_wrong_answer():
"""The failure that matters: noise must not manufacture credit."""
code = (
"def add(a, b):\n"
" return 0\n"
"\n"
'if __name__ == "__main__":\n'
' print("PASS PASS PASS")\n'
)
# Both checks genuinely fail for a function that always returns 0.
assert score_code(code, ["add(1,2)==3", "add(2,2)==4"])[0] == 0.0
def test_a_score_can_never_exceed_one():
"""Whatever a model prints, the fraction stays a fraction."""
code = "def f(x):\n return x\n" + 'print("CHECK 0 PASS\\n" * 50)\n'
assert score_code(code, ["f(1)==1"])[0] <= 1.0
def test_check_results_are_read_by_index_not_counted():
marker = "CHECK-deadbeef"
stdout = (
"PASS\n" # the model's noise
f"{marker} 0 PASS\n"
f"{marker} 1 FAIL\n"
"CHECK 2 PASS\n" # right shape, wrong marker
f"{marker} 0 PASS\n" # a repeat cannot double-count
)
assert parse_check_results(stdout, marker) == {0: True, 1: False}
# --- judge selection ------------------------------------------------------
def test_the_configured_judge_is_used_when_it_is_a_different_model():
assert judge_for("gemma-4-31b", "kimi-k3") == "kimi-k3"
def test_a_model_is_never_judged_by_its_own_family():
"""Including the case that broke it: the judge and the alternate agreeing."""
assert judge_for("kimi-k3", "kimi-k3") != "kimi-k3"
# `--judge-model qwen3.6-35b` used to hand qwen3.6-35b back to itself.
assert judge_for("qwen3.6-35b", "qwen3.6-35b") != "qwen3.6-35b"
def test_a_fast_row_is_not_judged_by_its_reasoning_on_sibling():
"""Same weights, so the conflict is the same one."""
assert judge_for("qwen3.6-35b-fast", "qwen3.6-35b") not in (
"qwen3.6-35b", "qwen3.6-35b-fast"
)
def test_no_judge_outside_the_family_means_no_sample():
"""Skipping beats self-grading, the same rule an unusable judge reply follows."""
assert judge_for("kimi-k3", "kimi-k3", ) is not None # alternates exist
import eval_proficiency
original = eval_proficiency.ALTERNATE_JUDGES
try:
eval_proficiency.ALTERNATE_JUDGES = ("kimi-k3-fast",)
assert eval_proficiency.judge_for("kimi-k3", "kimi-k3") is None
finally:
eval_proficiency.ALTERNATE_JUDGES = original

19
tests/test_load_candidates.py
View File

@@ -170,3 +170,22 @@ def test_context_estimate_reads_multimodal_text_parts():
{"type": "image_url", "image_url": {"url": "data:..."}}, {"type": "image_url", "image_url": {"url": "data:..."}},
]}] ]}]
assert estimate_prompt_tokens(msgs) == 100 assert estimate_prompt_tokens(msgs) == 100
def test_context_estimate_counts_tool_definitions():
# upstream_body = {**body, "model": target} forwards `tools` to the
# provider verbatim on every call, so it counts against the same context
# window and has to count here too. Omitting it undercounts every
# tool-carrying request -- nearly all agent traffic -- which is exactly
# how qwen3.6-35b got a live 400 for "too long ... even after
# compaction" while the router's own estimate said it fit.
from dispatcher import estimate_prompt_tokens
msgs = [{"role": "user", "content": "a" * 300}]
tools = [{"type": "function", "function": {
"name": "read_file",
"parameters": {"type": "object", "properties": {"path": {"type": "string"}}},
}}]
without_tools = estimate_prompt_tokens(msgs)
with_tools = estimate_prompt_tokens(msgs, tools=tools)
assert with_tools > without_tools

238
tests/test_logs.py Normal file
View File

@@ -0,0 +1,238 @@
"""Tests for the router's structured logging.
Written against the real handler and formatter rather than pytest's caplog,
because the thing worth pinning is what actually lands in the journal --
including the priority prefix, which caplog would never see.
"""
import io
import logging
import os
import pytest
import logs
@pytest.fixture(autouse=True)
def isolated_logger(monkeypatch):
"""Every test gets a fresh logger and a clean environment."""
monkeypatch.delenv(logs.LEVEL_ENV, raising=False)
monkeypatch.delenv("JOURNAL_STREAM", raising=False)
logs.set_trace(logs.NO_TRACE)
yield
for handler in list(logs.log.handlers):
logs.log.removeHandler(handler)
def _capture(level="debug", **kwargs):
buf = io.StringIO()
logs.configure(level, stream=buf, **kwargs)
return buf
# --- rendering ------------------------------------------------------------
def test_a_line_is_logfmt_with_the_trace_id_first():
buf = _capture()
logs.set_trace("r123456")
logs.info("route", cat="coding_general", tier=2)
assert buf.getvalue() == "route id=r123456 cat=coding_general tier=2\n"
def test_values_with_spaces_are_quoted():
buf = _capture()
logs.info("filter", detail="held during peak")
assert 'detail="held during peak"' in buf.getvalue()
def test_a_value_never_breaks_the_line():
"""One event, one line -- a newline in a detail string would forge a record."""
buf = _capture()
logs.info("verify", detail="line one\nline two")
assert buf.getvalue().count("\n") == 1
def test_none_is_a_dash_and_bools_are_binary():
buf = _capture()
logs.info("route", pick=None, tools=True, stream=False)
assert "pick=- tools=1 stream=0" in buf.getvalue()
def test_floats_are_readable_not_repr():
"""Microdollars and kWh, without 0.0007830000000000001."""
buf = _capture()
logs.info("dispatch", usd=0.000783123456, kwh=4.75e-05)
assert "usd=0.000783123" in buf.getvalue()
assert "kwh=4.75e-05" in buf.getvalue()
def test_an_empty_string_is_visible():
buf = _capture()
logs.info("classify", cat="")
assert 'cat=""' in buf.getvalue()
# --- levels ---------------------------------------------------------------
def test_debug_is_silent_at_info_level():
buf = _capture("info")
logs.debug("filter", model="m")
logs.info("route", pick="m")
assert "filter" not in buf.getvalue()
assert "route" in buf.getvalue()
def test_enabled_for_debug_reports_the_level():
_capture("info")
assert logs.enabled_for_debug() is False
_capture("debug")
assert logs.enabled_for_debug() is True
def test_the_env_var_overrides_the_configured_level(monkeypatch):
"""So a live service can be turned up without editing a tracked file."""
monkeypatch.setenv(logs.LEVEL_ENV, "debug")
buf = _capture("warning")
logs.debug("filter", model="m")
assert "filter" in buf.getvalue()
def test_a_nonsense_env_level_falls_back_rather_than_crashing(monkeypatch):
monkeypatch.setenv(logs.LEVEL_ENV, "loud")
assert logs.resolve_level("warning") == logging.WARNING
# --- journald -------------------------------------------------------------
def test_no_priority_prefix_in_a_terminal():
"""A foreground uvicorn should not print a literal <6> on every line."""
buf = _capture(journald=False)
logs.info("route", pick="m")
assert buf.getvalue().startswith("route ")
def test_priority_prefix_under_journald():
"""This is what makes `journalctl -p warning` mean anything."""
buf = _capture(journald=True)
logs.warning("retry", verdict="malformed")
logs.error("upstream", status=500)
logs.info("route", pick="m")
lines = buf.getvalue().splitlines()
assert lines[0].startswith("<4>") # warning
assert lines[1].startswith("<3>") # error
assert lines[2].startswith("<6>") # info
def test_journald_is_detected_by_matching_the_actual_stream(tmp_path, monkeypatch):
"""The env var alone is not enough -- it is inherited by every child.
A foreground uvicorn started from a systemd-managed session, with output
redirected to a file, inherited JOURNAL_STREAM and put a literal <7> on
every line. systemd documents comparing the value against the fd.
"""
path = tmp_path / "out.log"
with open(path, "w") as handle:
stat = os.fstat(handle.fileno())
monkeypatch.setenv("JOURNAL_STREAM", f"{stat.st_dev}:{stat.st_ino}")
assert logs.under_journald(handle) is True
# Same variable, a different stream: this is the inherited case.
monkeypatch.setenv("JOURNAL_STREAM", f"{stat.st_dev}:{stat.st_ino + 1}")
assert logs.under_journald(handle) is False
def test_no_journald_variable_means_no_prefix():
assert logs.under_journald() is False
def test_a_stream_with_no_fileno_is_not_journald():
"""StringIO under test, and anything else without an fd."""
assert logs.under_journald(io.StringIO()) is False
def test_a_malformed_journald_variable_is_ignored(monkeypatch):
monkeypatch.setenv("JOURNAL_STREAM", "not-a-device")
assert logs.under_journald() is False
# --- trace ids ------------------------------------------------------------
def test_a_trace_id_is_stable_until_a_new_one_is_started():
first = logs.new_trace()
assert logs.current_trace() == first
assert logs.new_trace() != first
def test_trace_ids_do_not_collide():
assert len({logs.new_trace() for _ in range(200)}) == 200
def test_configure_is_idempotent():
"""Re-importing or re-configuring must not double every line."""
buf = io.StringIO()
logs.configure("info", stream=buf, journald=False)
logs.configure("info", stream=buf, journald=False)
logs.info("route", pick="m")
assert buf.getvalue().count("route") == 1
def test_the_logger_does_not_propagate():
"""uvicorn configures its own logging; a root handler would double-print."""
_capture()
assert logs.log.propagate is False
# --- bound emitters -------------------------------------------------------
def test_a_bound_emitter_ignores_the_ambient_trace():
"""What the streaming path needs: an id that survives a context copy."""
buf = _capture()
bound = logs.bind("rfixed1")
logs.set_trace("rother2")
bound.info("dispatch", model="m")
assert "id=rfixed1" in buf.getvalue()
def test_bind_defaults_to_the_current_trace():
buf = _capture()
logs.set_trace("rcurrent")
logs.bind().warning("retry", verdict="malformed")
assert "id=rcurrent" in buf.getvalue()
def test_a_bound_emitter_respects_the_level():
buf = _capture("info")
logs.bind("r1").debug("filter", model="m")
assert buf.getvalue() == ""

156
tests/test_outcome_attribution.py Normal file
View File

@@ -0,0 +1,156 @@
"""Tests for which completion an outcome report attaches to.
`POST /outcome` is the only ground truth the router gets, so attributing one
to the wrong conversation is worse than losing it: a model gets penalized for
work it never did. The guard is a short window plus a refusal -- if more than
one conversation was served inside it, the report is refused rather than
guessed at.
The window was not a window. `observed_at` is written by
`datetime.now(timezone.utc).isoformat()`, which separates date from time with
'T', while `datetime('now', ...)` returns a space. Compared as strings, 'T'
sorts after ' ', so the time of day never participated once the dates matched
and a 120-second window admitted everything served that day. Measured on the
live DB: 14 rows across 2 sessions where the correct comparison matched 0.
These tests write timestamps through the same call the dispatcher uses, so
they stay honest if that format ever changes.
"""
import sqlite3
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pytest
import dispatcher
from dispatcher import AMBIGUOUS, SEED_CATEGORY, _most_recent_if_unambiguous
ROOT = Path(__file__).resolve().parent.parent
SCHEMA_SQL = (ROOT / "schema.sql").read_text()
# Short, so "earlier today" is unambiguously outside it. The old row below sits
# at the first instant of the current UTC day -- the earliest time that still
# shares today's date, which is what the string comparison needed to go wrong.
WINDOW_SECONDS = 5
def _now() -> datetime:
return datetime.now(timezone.utc)
def _start_of_today() -> datetime:
return _now().replace(hour=0, minute=0, second=0, microsecond=1)
@pytest.fixture(autouse=True)
def short_window(monkeypatch):
monkeypatch.setattr(
dispatcher.cfg.verification,
"outcome_attribution_window_seconds",
WINDOW_SECONDS,
)
@pytest.fixture
def db(tmp_path):
conn = sqlite3.connect(tmp_path / "test.db")
conn.row_factory = sqlite3.Row
conn.executescript(SCHEMA_SQL)
yield conn
conn.close()
def _observe(conn, *, when: datetime, session_key: str, request_id: str,
category: str = "coding_general"):
conn.execute(
"""
INSERT INTO energy_observations (
model_id, provider, request_id, session_key, task_category, observed_at
) VALUES ('m', 'neuralwatt', ?, ?, ?, ?)
""",
# Written exactly the way log_observation writes it.
(request_id, session_key, category, when.isoformat()),
)
conn.commit()
def test_a_session_from_earlier_today_is_outside_the_window(db):
"""The regression: same calendar date is not the same as inside 5 seconds."""
_observe(db, when=_start_of_today(), session_key="morning", request_id="old")
_observe(db, when=_now(), session_key="now", request_id="new")
row = _most_recent_if_unambiguous(db)
assert row is not AMBIGUOUS, "an 8-hours-stale session must not create ambiguity"
assert row["request_id"] == "new"
def test_an_old_session_alone_is_not_recent_enough_to_attribute(db):
"""With nothing inside the window there is nothing to attach to -- 404, not a guess."""
_observe(db, when=_start_of_today(), session_key="morning", request_id="old")
assert _most_recent_if_unambiguous(db) is None
def test_two_live_conversations_are_refused(db):
"""The guard the window exists to serve, still working."""
_observe(db, when=_now(), session_key="alice", request_id="a")
_observe(db, when=_now(), session_key="bob", request_id="b")
assert _most_recent_if_unambiguous(db) is AMBIGUOUS
def test_one_conversation_across_several_turns_is_not_ambiguous(db):
"""Several completions, one session -- the ordinary case must still attribute."""
for i in range(3):
_observe(db, when=_now(), session_key="alice", request_id=f"a{i}")
row = _most_recent_if_unambiguous(db)
assert row is not AMBIGUOUS
assert row["request_id"] == "a2"
def test_the_reference_sweep_is_not_a_conversation(db):
"""seed_energy's traffic must never make a real report look ambiguous."""
_observe(db, when=_now(), session_key=None, request_id="seed",
category=SEED_CATEGORY)
_observe(db, when=_now(), session_key="alice", request_id="a")
row = _most_recent_if_unambiguous(db)
assert row is not AMBIGUOUS
assert row["request_id"] == "a"
def test_quota_burn_counts_only_the_last_thirty_days(db, tmp_path, monkeypatch):
"""Same comparison, same fix -- an old row must not inflate the burn figure.
The row sits at the START of the day 30 days ago: the cutoff falls on that
same date but later in it, which is exactly the case a string comparison
gets wrong. A row from 45 days ago would be excluded either way and would
pin nothing.
"""
monkeypatch.setattr(dispatcher.cfg.database, "path", str(tmp_path / "test.db"))
monkeypatch.setattr(dispatcher.cfg.objective, "plan_kwh_per_period", 6.25)
just_outside = (_now() - timedelta(days=30)).replace(
hour=0, minute=0, second=0, microsecond=1
)
db.execute(
"""
INSERT INTO energy_observations (model_id, provider, energy_kwh, observed_at)
VALUES ('m', 'neuralwatt', 1.0, ?)
""",
(just_outside.isoformat(),),
)
db.execute(
"""
INSERT INTO energy_observations (model_id, provider, energy_kwh, observed_at)
VALUES ('m', 'neuralwatt', 0.25, ?)
""",
(_now().isoformat(),),
)
db.commit()
assert dispatcher.quota_burn()["metered_kwh_30d"] == pytest.approx(0.25)

84
tests/test_poller_parsing.py
View File

@@ -105,3 +105,87 @@ def test_family_never_collapses_to_empty():
# A degenerate id made only of suffix tokens must keep a segment, or every # A degenerate id made only of suffix tokens must keep a segment, or every
# such row would share one meaningless family # such row would share one meaningless family
assert parse_base_model_id("flex") == "flex" assert parse_base_model_id("flex") == "flex"
# --- per-model context overrides ------------------------------------------
#
# The knob validated and was never read, while config.yaml shipped a worked
# example for it. Anyone who followed that example got silence.
def _row(**overrides):
from poller import ModelRow
fields = dict(
model_id="qwen3.6-35b", provider="neuralwatt", base_model_id="qwen3.6-35b",
display_name=None, cost_per_1m_prompt=None, cost_per_1m_completion=None,
cost_per_1m_prompt_cached=None, context_window=200_000,
max_output_tokens=None, supports_tools=True, supports_json_mode=True,
supports_vision=False, supports_reasoning=True,
reasoning_default_enabled=True, latency_class="standard",
reasoning_mode="default", context_variant="full", access_level="public",
pricing_tbd=False, deprecated=False,
)
fields.update(overrides)
return ModelRow(**fields)
def _cfg(overrides=None):
import copy
import yaml
from config import RouterConfig
with open("config.yaml") as fh:
raw = copy.deepcopy(yaml.safe_load(fh))
raw["context"]["per_model_overrides"] = overrides or {}
return RouterConfig(**raw)
def test_the_global_safety_factor_applies_without_an_override():
# 200000 * 0.75 - 4096
assert _row().effective_context_window(_cfg()) == 145_904
def test_a_per_model_safety_factor_is_actually_applied():
cfg = _cfg({"qwen3.6-35b": {"safety_factor": 0.85}})
# 200000 * 0.85 - 4096
assert _row().effective_context_window(cfg) == 165_904
def test_an_override_for_another_model_does_not_leak():
cfg = _cfg({"kimi-k3": {"safety_factor": 0.85}})
assert _row().effective_context_window(cfg) == 145_904
def test_a_zero_output_reserve_means_zero_not_unset():
"""Present-but-falsy: `or` would silently fall through to 4096."""
cfg = _cfg({"qwen3.6-35b": {"output_reserve_tokens": 0}})
assert _row().effective_context_window(cfg) == 150_000
def test_an_override_reserve_beats_the_models_advertised_ceiling():
cfg = _cfg({"qwen3.6-35b": {"output_reserve_tokens": 8192}})
assert _row(max_output_tokens=16384).effective_context_window(cfg) == 141_808
def test_each_key_falls_back_independently():
cfg = _cfg({"qwen3.6-35b": {"safety_factor": 0.85}})
# factor overridden, reserve still the global default
assert _row().effective_context_window(cfg) == 165_904
def test_a_typo_inside_an_override_is_an_error():
"""The point of typing it: StrictModel has to reach inside the block."""
with pytest.raises(Exception, match="safety_factr|Extra inputs"):
_cfg({"qwen3.6-35b": {"safety_factr": 0.85}})
def test_an_out_of_range_override_is_an_error():
with pytest.raises(Exception, match="safety_factor"):
_cfg({"qwen3.6-35b": {"safety_factor": 1.5}})

172
tests/test_proficiency.py
View File

@@ -177,3 +177,175 @@ def test_a_directly_measured_variant_is_never_overwritten(tmp_path):
"SELECT self_eval_score FROM proficiency WHERE model_id='kimi-k3-flex'" "SELECT self_eval_score FROM proficiency WHERE model_id='kimi-k3-flex'"
).fetchone() ).fetchone()
assert got["self_eval_score"] == 0.2 assert got["self_eval_score"] == 0.2
def test_a_score_above_one_never_reaches_the_table(tmp_path):
"""The single write path is where a bad scorer gets stopped.
A blended_score above 1.0 does not just misreport one row: it raises
`best` in rank_candidates and shifts every other candidate's quality band.
"""
from config import load_config
from proficiency_store import add_self_eval
cfg = load_config("config.yaml")
conn = _models_db(tmp_path, [("kimi-k3", "kimi-k3", "standard", "default", "full")])
add_self_eval(conn, cfg, "kimi-k3", "nw", "coding_general", [1.5, 1.0])
row = conn.execute(
"SELECT self_eval_score, blended_score FROM proficiency WHERE model_id = 'kimi-k3'"
).fetchone()
assert row["self_eval_score"] == 1.0
assert row["blended_score"] <= 1.0
def test_a_negative_score_is_floored_at_zero(tmp_path):
from config import load_config
from proficiency_store import add_self_eval
cfg = load_config("config.yaml")
conn = _models_db(tmp_path, [("kimi-k3", "kimi-k3", "standard", "default", "full")])
add_self_eval(conn, cfg, "kimi-k3", "nw", "coding_general", [-0.5, 0.5])
assert conn.execute(
"SELECT self_eval_score FROM proficiency WHERE model_id = 'kimi-k3'"
).fetchone()["self_eval_score"] == 0.25
def test_an_inherited_variant_refreshes_when_its_family_is_remeasured(tmp_path):
"""Inheritance was a one-shot: a variant froze at its first copy, forever.
The old guard skipped any row with self_eval_samples > 0, and an inherited
row has samples > 0 because inheritance copies them -- so it could never
tell "measured here" from "copied here" and never refreshed. Observed live:
kimi-k3 reached 0.957 while kimi-k3-flex sat at 0.85 with a week-old
timestamp, and the eval reported "propagated 0 inherited rows". Flex rows
serve auto:batch, so those requests ranked on stale scores.
"""
from config import load_config
from proficiency_store import add_self_eval, propagate_to_variants
cfg = load_config("config.yaml")
conn = _models_db(tmp_path, [
("kimi-k3", "kimi-k3", "standard", "default", "full"),
("kimi-k3-flex", "kimi-k3", "flex", "default", "full"),
])
add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [0.85, 0.85])
assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 1
inherited = conn.execute(
"SELECT blended_score, inherited_from FROM proficiency WHERE model_id='kimi-k3-flex'"
).fetchone()
assert inherited["blended_score"] == pytest.approx(0.85)
assert inherited["inherited_from"] == "kimi-k3"
# The family learns more; the variant must follow it.
add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [1.0, 1.0, 1.0, 1.0])
assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 1
refreshed = conn.execute(
"SELECT blended_score, self_eval_samples FROM proficiency WHERE model_id='kimi-k3-flex'"
).fetchone()
assert refreshed["blended_score"] == pytest.approx(0.95)
assert refreshed["self_eval_samples"] == 6
def test_a_measured_variant_still_outranks_its_family(tmp_path):
"""The protection the old guard was reaching for, kept intact."""
from config import load_config
from proficiency_store import add_self_eval, propagate_to_variants
cfg = load_config("config.yaml")
conn = _models_db(tmp_path, [
("kimi-k3", "kimi-k3", "standard", "default", "full"),
("kimi-k3-flex", "kimi-k3", "flex", "default", "full"),
])
add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [1.0])
add_self_eval(conn, cfg, "kimi-k3-flex", "nw", "docs_writing", [0.2])
assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 0
row = conn.execute(
"SELECT blended_score, inherited_from FROM proficiency WHERE model_id='kimi-k3-flex'"
).fetchone()
assert row["blended_score"] == pytest.approx(0.2)
assert row["inherited_from"] is None
def test_a_database_without_the_column_is_migrated(tmp_path):
"""schema.sql only defines a NEW database; existing ones need the ALTER."""
import sqlite3
from config import load_config
from proficiency_store import add_self_eval, ensure_columns
cfg = load_config("config.yaml")
conn = _models_db(tmp_path, [("kimi-k3", "kimi-k3", "standard", "default", "full")])
conn.execute("ALTER TABLE proficiency DROP COLUMN inherited_from")
assert "inherited_from" not in {
r[1] for r in conn.execute("PRAGMA table_info(proficiency)")
}
ensure_columns(conn)
ensure_columns(conn) # idempotent
add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [1.0])
assert conn.execute(
"SELECT inherited_from FROM proficiency WHERE model_id='kimi-k3'"
).fetchone()["inherited_from"] is None
def test_migration_unfreezes_variants_that_predate_the_column(tmp_path):
"""The migration must ship the repair, not just the fix.
ADD COLUMN gives every existing row NULL, and NULL means "measured here" --
so without the backfill, the rows this whole change exists for would stay
frozen and the live flex rows would never catch up.
"""
from config import load_config
from proficiency_store import add_self_eval, ensure_columns, propagate_to_variants
cfg = load_config("config.yaml")
conn = _models_db(tmp_path, [
("kimi-k3", "kimi-k3", "standard", "default", "full"),
("kimi-k3-flex", "kimi-k3", "flex", "default", "full"),
])
# An old database: both rows carry samples, neither records provenance.
add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [0.957])
add_self_eval(conn, cfg, "kimi-k3-flex", "nw", "docs_writing", [0.85])
conn.execute("ALTER TABLE proficiency DROP COLUMN inherited_from")
ensure_columns(conn)
assert conn.execute(
"SELECT inherited_from FROM proficiency WHERE model_id='kimi-k3-flex'"
).fetchone()["inherited_from"] == "kimi-k3"
# ...so the next run actually moves it.
assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 1
assert conn.execute(
"SELECT blended_score FROM proficiency WHERE model_id='kimi-k3-flex'"
).fetchone()["blended_score"] == pytest.approx(0.957)
def test_the_backfill_never_touches_a_standard_row(tmp_path):
"""Only flex rows with a standard equivalent are provably inherited."""
from config import load_config
from proficiency_store import add_self_eval, ensure_columns
cfg = load_config("config.yaml")
conn = _models_db(tmp_path, [
("kimi-k3", "kimi-k3", "standard", "default", "full"),
("kimi-k3-fast", "kimi-k3", "standard", "reduced", "full"),
])
add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [1.0])
add_self_eval(conn, cfg, "kimi-k3-fast", "nw", "docs_writing", [0.888])
conn.execute("ALTER TABLE proficiency DROP COLUMN inherited_from")
ensure_columns(conn)
for model_id in ("kimi-k3", "kimi-k3-fast"):
assert conn.execute(
"SELECT inherited_from FROM proficiency WHERE model_id=?", (model_id,)
).fetchone()["inherited_from"] is None

217
tests/test_routing.py
View File

@@ -6,7 +6,12 @@ proving it rejects and a case proving it does not over-reject.
import pytest import pytest
from routing import is_eligible, rank_candidates, select_candidates from routing import (
is_eligible,
rank_candidates,
rejection_reason,
select_candidates,
)
@@ -25,6 +30,8 @@ def _row(**overrides) -> dict:
"latency_class": "standard", "latency_class": "standard",
"reasoning_mode": "default", "reasoning_mode": "default",
"context_variant": "full", "context_variant": "full",
"supports_vision": 1,
"supports_json_mode": 1,
} }
row.update(overrides) row.update(overrides)
return row return row
@@ -233,3 +240,211 @@ def test_unmeasured_cost_is_admitted_under_a_ceiling():
def test_ceiling_at_exactly_the_cost_admits(): def test_ceiling_at_exactly_the_cost_admits():
rows = [_row(model_id="borderline", energy=1e-6, proficiency=0.5)] rows = [_row(model_id="borderline", energy=1e-6, proficiency=0.5)]
assert len(rank_candidates(rows, max_energy_per_request=1e-6)) == 1 assert len(rank_candidates(rows, max_energy_per_request=1e-6)) == 1
# --- tool competence is read from the request, not guessed at -------------
def test_tools_in_the_request_exclude_a_model_that_overreaches_for_them():
# deepseek-v4-flash measures 0.33 here. The recorded failure is a
# NON-agentic prompt ("it is 1:20pm, my meeting is at 3pm, how many
# minutes?") where it called two tools instead of subtracting -- so the
# hazard is tools being available, not the task being agentic.
weak = _row(model_id="overreacher", proficiency=1.0, tool_proficiency=0.33)
strong = _row(model_id="reliable", proficiency=1.0, tool_proficiency=1.0)
assert _eligible(weak, min_tool_proficiency=0.5) is False
assert _eligible(strong, min_tool_proficiency=0.5) is True
def test_without_tools_the_same_model_is_fine():
# The filter applies only when the request carries tool definitions;
# otherwise a cheap tool-clumsy model is a perfectly good choice.
weak = _row(model_id="overreacher", tool_proficiency=0.33)
assert _eligible(weak, min_tool_proficiency=None) is True
def test_an_unmeasured_model_is_unproven_not_disqualified():
# Same principle as the tier-1 context gate: absent evidence must not
# decide anything. A model nobody has evaluated for tool use yet is
# unproven, not proven bad.
unknown = _row(model_id="unevaluated", tool_proficiency=None)
assert _eligible(unknown, min_tool_proficiency=0.5) is True
def test_the_threshold_is_exclusive_at_the_boundary():
assert _eligible(_row(tool_proficiency=0.5), min_tool_proficiency=0.5) is True
assert _eligible(_row(tool_proficiency=0.49), min_tool_proficiency=0.5) is False
def test_the_tool_filter_can_change_the_winner():
# The whole point: it is a hard filter, so it removes a candidate that
# would otherwise win on cost rather than merely penalizing it.
rows = [
_row(model_id="cheap-clumsy", proficiency=1.0, tool_proficiency=0.33,
cost_per_1m_prompt=0.1, cost_per_1m_completion=0.2),
_row(model_id="dearer-reliable", proficiency=1.0, tool_proficiency=1.0,
cost_per_1m_prompt=1.0, cost_per_1m_completion=2.0),
]
no_tools = select_candidates(
rows, required_context_tokens=1000, required_tier=1,
latency_tolerance="interactive", allowed_access_levels=["public"],
exclude_stale=True, exclude_deprecated=True, min_tool_proficiency=None)
with_tools = select_candidates(
rows, required_context_tokens=1000, required_tier=1,
latency_tolerance="interactive", allowed_access_levels=["public"],
exclude_stale=True, exclude_deprecated=True, min_tool_proficiency=0.5)
assert rank_candidates(no_tools, prompt_tokens=1000)[0]["model_id"] == "cheap-clumsy"
assert rank_candidates(with_tools, prompt_tokens=1000)[0]["model_id"] == "dearer-reliable"
# --- capability filters ----------------------------------------------------
def test_vision_required_excludes_a_model_without_vision():
blind = _row(model_id="blind", supports_vision=0)
sighted = _row(model_id="sighted", supports_vision=1)
assert _eligible(blind, require_vision=True) is False
assert _eligible(sighted, require_vision=True) is True
def test_vision_not_required_admits_everyone():
# The gate is off by default; not asking for vision must not exclude
blind = _row(model_id="blind", supports_vision=0)
assert _eligible(blind, require_vision=False) is True
def test_unknown_vision_capability_fails_closed():
# A capability FLAG that is absent means "cannot confirm". Unlike a
# proficiency MEASUREMENT, admit-on-None would route a vision request to a
# model that might lack it -- a guaranteed provider 400.
unknown = _row(model_id="unknown", supports_vision=None)
assert _eligible(unknown, require_vision=True) is False
def test_json_mode_required_excludes_without_support():
plain = _row(model_id="plain", supports_json_mode=0)
json = _row(model_id="json", supports_json_mode=1)
assert _eligible(plain, require_json_mode=True) is False
assert _eligible(json, require_json_mode=True) is True
def test_unknown_json_mode_fails_closed():
unknown = _row(model_id="unknown", supports_json_mode=None)
assert _eligible(unknown, require_json_mode=True) is False
def test_json_mode_gate_admits_supporting_model():
assert _eligible(_row(supports_json_mode=1), require_json_mode=True) is True
def test_the_vision_filter_can_change_the_winner():
# Same shape as the tool-filter winner test: the gate is a hard filter, so
# it removes a model that would otherwise win on cost rather than merely
# penalizing it.
rows = [
_row(model_id="cheap-blind", proficiency=1.0, supports_vision=0,
cost_per_1m_prompt=0.1, cost_per_1m_completion=0.2),
_row(model_id="dearer-sighted", proficiency=1.0, supports_vision=1,
cost_per_1m_prompt=1.0, cost_per_1m_completion=2.0),
]
no_images = select_candidates(
rows, required_context_tokens=1000, required_tier=1,
latency_tolerance="interactive", allowed_access_levels=["public"],
exclude_stale=True, exclude_deprecated=True, require_vision=False)
with_images = select_candidates(
rows, required_context_tokens=1000, required_tier=1,
latency_tolerance="interactive", allowed_access_levels=["public"],
exclude_stale=True, exclude_deprecated=True, require_vision=True)
assert rank_candidates(no_images, prompt_tokens=1000)[0]["model_id"] == "cheap-blind"
assert rank_candidates(with_images, prompt_tokens=1000)[0]["model_id"] == "dearer-sighted"
def test_rejection_reason_names_the_missing_capability():
assert _reason(_row(supports_vision=0), require_vision=True) == "vision(unsupported)"
assert _reason(_row(supports_vision=None), require_vision=True) == "vision(unknown)"
assert _reason(_row(supports_json_mode=0), require_json_mode=True) == "json_mode(unsupported)"
assert _reason(_row(supports_json_mode=None), require_json_mode=True) == "json_mode(unknown)"
def test_zero_tolerance_ranks_strictly_on_quality():
""""Never trade quality for cost" is a legitimate setting.
The validator has always accepted 0; `band()` divided by it, so every
request raised ZeroDivisionError on a config that had loaded cleanly.
"""
rows = [_row(model_id="cheap", cost=1e-9, proficiency=0.98),
_row(model_id="dear", cost=1e-3, proficiency=1.00)]
assert rank_candidates(rows, quality_tolerance=0)[0]["model_id"] == "dear"
def test_zero_tolerance_still_breaks_exact_ties_on_cost():
rows = [_row(model_id="dear", cost=1e-3, proficiency=1.00),
_row(model_id="cheap", cost=1e-9, proficiency=1.00)]
assert rank_candidates(rows, quality_tolerance=0)[0]["model_id"] == "cheap"
# --- rejection reasons ----------------------------------------------------
#
# is_eligible returns a bool, so a dropped model used to vanish without
# explanation and "no model satisfies the hard filters" was a dead end. The
# reason string is what the debug log prints, so it has to be exact rather than
# re-derived somewhere else.
def _reason(row, **overrides):
kwargs = {
"required_context_tokens": 10_000,
"required_tier": 2,
"latency_tolerance": "interactive",
"allowed_access_levels": ["public"],
"exclude_stale": True,
"exclude_deprecated": True,
}
kwargs.update(overrides)
return rejection_reason(row, **kwargs)
def test_an_eligible_row_has_no_reason():
assert _reason(_row()) is None
def test_the_reason_names_the_filter_and_its_numbers():
assert _reason(_row(effective_context_window=5_000)) == "context(5000<10000)"
assert _reason(_row(tier=1)) == "tier(1<2)"
assert _reason(_row(availability="stale")) == "stale"
assert _reason(_row(availability="deprecated")) == "deprecated"
assert _reason(_row(access_level="canary")) == "access_level(canary)"
assert _reason(_row(latency_class="flex")) == "latency_class(flex)"
def test_unknown_values_say_unknown_rather_than_comparing():
assert _reason(_row(effective_context_window=None)) == "context(unknown)"
assert _reason(_row(tier=None)) == "tier(unknown)"
def test_the_tool_filter_reports_the_measured_score():
reason = _reason(_row(tool_proficiency=0.33), min_tool_proficiency=0.5)
assert reason == "tool_proficiency(0.33<0.5)"
def test_reasons_are_single_tokens():
"""They go straight into a logfmt value; a space would force quoting."""
for row in (_row(tier=1), _row(availability="stale"),
_row(access_level="canary"), _row(latency_class="flex"),
_row(supports_vision=0), _row(supports_vision=None),
_row(supports_json_mode=0), _row(supports_json_mode=None)):
reason = _reason(row, require_vision=True, require_json_mode=True)
assert reason is not None
assert " " not in reason
def test_is_eligible_still_agrees_with_the_reason():
"""One copy of the rules, two views of it."""
for row in (_row(), _row(tier=1), _row(latency_class="flex")):
assert is_eligible(row, **{
"required_context_tokens": 10_000, "required_tier": 2,
"latency_tolerance": "interactive", "allowed_access_levels": ["public"],
"exclude_stale": True, "exclude_deprecated": True,
}) == (_reason(row) is None)

142
tests/test_seed_sweep.py Normal file
View File

@@ -0,0 +1,142 @@
"""End-to-end test of the reference sweep, with the provider stubbed.
The sweep is what makes `eco` and the per-request energy ceiling real numbers
instead of the neutral 0.5, and `llm-router-seed.timer` runs it every six
hours. It had been dead for weeks: `log_observation` grew `request_id`,
`session_key` and `session_dir` in the middle of its signature and made the
trailing three keyword-only, while this call still passed six positionals. The
result was a TypeError -- not a RequestException, so the per-sample `except`
never caught it -- raised AFTER the first billed completion. Every scheduled
run spent money and wrote nothing.
So this is a smoke test of `main()` rather than an assertion about a
signature: it catches this class of drift wherever it next appears, and it
costs nothing to run because `sample_once` never leaves the process.
"""
import sqlite3
import sys
from pathlib import Path
import pytest
import dispatcher
import seed_energy
ROOT = Path(__file__).resolve().parent.parent
SCHEMA_SQL = (ROOT / "schema.sql").read_text()
MODEL = "gemma-4-31b"
def _payload(model_id):
return {
"id": "chatcmpl-seed-1",
"model": model_id,
"choices": [{"message": {"content": "A B-tree is..."},
"finish_reason": "length"}],
"usage": {"prompt_tokens": 31, "completion_tokens": 400},
"energy": {"energy_kwh": 4.75e-05, "avg_power_watts": 420.0,
"duration_seconds": 1.6, "attribution_ratio": 0.25,
"carbon_g_co2eq": 2.32e-04, "carbon_source": "agent_cache",
"grid_id": "FI"},
"cost": {"request_cost_usd": 3.8e-04, "allowance_remaining_usd": 49.5},
}
@pytest.fixture
def sweep(tmp_path, monkeypatch):
db_path = tmp_path / "test.db"
conn = sqlite3.connect(db_path)
conn.executescript(SCHEMA_SQL)
conn.execute(
"""
INSERT INTO models (
model_id, provider, tier, latency_class, access_level,
availability, cost_per_1m_completion, last_updated
) VALUES (?, 'neuralwatt', 1, 'standard', 'public', 'active', 0.42,
'2026-08-22T00:00:00+00:00')
""",
(MODEL,),
)
conn.commit()
conn.close()
# main() loads its own config; log_observation reads dispatcher's. They must
# agree on the database or the sweep writes somewhere the test cannot see.
monkeypatch.setattr(dispatcher.cfg.database, "path", str(db_path))
monkeypatch.setattr(seed_energy, "load_config", lambda path: dispatcher.cfg)
monkeypatch.setenv("NEURALWATT_API_KEY", "test-key")
monkeypatch.setattr(seed_energy.time, "sleep", lambda _: None)
calls = []
def fake_sample(base_url, api_key, model_id, max_tokens=400, timeout=300):
calls.append({"model_id": model_id, "max_tokens": max_tokens})
return _payload(model_id)
monkeypatch.setattr(seed_energy, "sample_once", fake_sample)
yield calls, db_path
def _run(monkeypatch, *argv):
monkeypatch.setattr(sys, "argv", ["seed_energy.py", *argv])
return seed_energy.main()
def test_a_sweep_writes_the_observation_it_paid_for(sweep, monkeypatch):
"""The regression: this raised TypeError after the first billed call."""
calls, db_path = sweep
assert _run(monkeypatch, "--samples", "2") == 0
assert len(calls) == 2
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
rows = conn.execute(
"SELECT * FROM energy_observations WHERE task_category = ?",
(seed_energy.SEED_CATEGORY,),
).fetchall()
conn.close()
assert len(rows) == 2
row = rows[0]
assert row["model_id"] == MODEL
assert row["prompt_tokens"] == 31
assert row["completion_tokens"] == 400
assert row["energy_kwh"] == pytest.approx(4.75e-05)
assert row["carbon_g_co2eq"] == pytest.approx(2.32e-04)
assert row["cost_usd"] == pytest.approx(3.8e-04)
def test_the_sweep_records_the_completion_id(sweep, monkeypatch):
"""Positional drift silently dropped request_id; it is the join key for /outcome."""
_, db_path = sweep
_run(monkeypatch, "--samples", "1")
conn = sqlite3.connect(db_path)
request_id = conn.execute(
"SELECT request_id FROM energy_observations"
).fetchone()[0]
conn.close()
assert request_id == "chatcmpl-seed-1"
def test_max_tokens_reaches_the_request(sweep, monkeypatch):
"""The flag was parsed, printed in the banner, and never sent."""
calls, _ = sweep
_run(monkeypatch, "--samples", "1", "--max-tokens", "800")
assert calls[0]["max_tokens"] == 800
def test_a_dry_run_spends_nothing(sweep, monkeypatch):
calls, db_path = sweep
assert _run(monkeypatch, "--dry-run") == 0
assert not calls
conn = sqlite3.connect(db_path)
assert conn.execute("SELECT COUNT(*) FROM energy_observations").fetchone()[0] == 0
conn.close()