diff --git a/CLAUDE.md b/CLAUDE.md index 97303cf..b9d8e26 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,12 +7,17 @@ next steps, and is the one to trust on what is currently true. ## What this is -A router that uses a local model (served via Ollama on an RTX 6000, 24GB) to -classify incoming coding/documentation tasks — category, tier, required -context size — and dispatch each task to the cheapest/best-fit open-weight -model on **Neuralwatt Cloud**, weighted by cost, ecological impact -(Neuralwatt exposes real energy-per-request data), and per-category -proficiency. +A router that uses a local model (served via Ollama) to classify incoming +coding/documentation tasks — category, tier, required context size — and +dispatch each task to the cheapest/best-fit open-weight model on **Neuralwatt +Cloud**, weighted by cost, per-category proficiency, and a per-request energy +ceiling. + +Every measurement in this file was taken on one deployment against one +provider account. They are recorded because the reasoning is worth more than +the conclusion, but treat them as observations with a date on them, not as +constants — the catalog, prices, grid intensity and pool load all move. When +a number here decides something, re-run the measurement before trusting it. Neuralwatt is the only provider. OpenRouter was removed — the `provider` column and the `(model_id, provider)` primary key stay so a second provider @@ -24,8 +29,9 @@ can be added later without a migration. CPU-bound; iteration speed on the scoring/weighting logic matters more than raw execution speed at this scale) - SQLite for the decision table -- Ollama (OpenAI-compatible endpoint at `localhost:11434/v1`) for local - classification +- Ollama for local classification, via any OpenAI-compatible endpoint — + `localhost:11434/v1`, or an Ollama on another machine across a VPN + (`classifier.base_url`) - FastAPI for the dispatcher service ## Billing is per-kWh, not per-token — and neither is what scoring uses @@ -85,8 +91,9 @@ is load-bearing, and `tests/test_routing.py` pins it. for real traffic, because the reference workload is the wrong shape. The sweep sends a 400-token prompt with a 400-token completion. Real agent -traffic is a 150,000-token prompt with a ~400-token completion and 84% cache -hits. The attribution ratio moves with prompt size, so the ranking inverts: +traffic is a 150,000-token prompt with a ~400-token completion and ~92% cache +hits (token-weighted over 50 sessions and 40.7M tokens on 2026-08-23; the +figure was 84% when measured on 2.2M tokens of earlier traffic). The attribution ratio moves with prompt size, so the ranking inverts: | workload | winner | |---|---| @@ -176,8 +183,16 @@ median over ALL `seed_reference` rows, so repeated sweeps accumulate into a median-across-time for free — hence `llm-router-seed.timer`, which runs a small sweep every 6 hours. -Until several sweeps have accumulated, treat the cost and eco ordering as -provisional. A single sweep's ranking is one sample of a moving quantity. +Until several sweeps have accumulated, treat the eco ordering as provisional. +A single sweep's ranking is one sample of a moving quantity. + +**And none have accumulated since 6e729ad.** That commit moved +`log_observation`'s trailing arguments to keyword-only without updating +`seed_energy.py`, so every timer run since spent one billed completion and then +died on `TypeError` — which is not a `RequestException`, so the per-sample +`except` did not catch it. Fixed, and the sweep now has an offline end-to-end +test, but the accumulation this section describes starts from the next run +rather than from months of history. ## What's built and working @@ -215,7 +230,77 @@ provisional. A single sweep's ranking is one sample of a moving quantity. `code` executes the model's Python against checks, `exact` compares a normalized answer, `tool` inspects the tool call structurally, and only the four prose categories fall back to a `judge`. -- `tests/` — 207 tests across 11 files, all passing. +- `capabilities.py` — request-side capability detection. `detect_capabilities` + reads an OpenAI-format body and returns `RequestCapabilities` (`has_images`, + `require_json_mode`, `tools_present`, `has_reasoning_request`). It scans all + messages for `image_url` parts and reads `response_format.type` against + `{json_object, json_schema}`. Detection is read from the request body, not + inferred by a classifier. +- `logs.py` — the service's structured logging: a per-request trace id in a + ContextVar, logfmt rendering, and journald priority prefixes emitted only + when systemd actually owns stderr. `logs.bind()` exists because a + StreamingResponse's generator is resumed in a fresh copy of the caller's + context, so the ContextVar cannot reach it — the path all agent traffic + takes. Level from `logging.level`, overridden by `LLM_ROUTER_LOG_LEVEL`. +- `tests/` — 356 tests across 19 files, all passing, all offline. Verified on + Python 3.10 and 3.14; nothing declares `requires-python`, so 3.10 is the + tested floor rather than a promised one. + +### Request-side capability gates + +The router treats some capabilities as hard filters, read directly from the +request body. Two new gates live in `routing.rejection_reason` and +`routing.select_candidates`: + +- `require_vision` — active when the request carries `image_url` parts and + `routing.require_vision` is true. A model passes only if its catalog row says + `supports_vision = 1`. `supports_vision = NULL` fails closed: an unknown flag + means the capability cannot be confirmed, and routing an image request to a + model that might lack vision is a guaranteed provider 400. +- `require_json_mode` — active when `response_format.type` is `json_object` or + `json_schema` and `routing.require_json_mode` is true. A model passes only if + `supports_json_mode = 1`; `NULL` also fails closed, for the same reason. + +Both gates default to **on** in `config.yaml`, because a wrong guess produces a +400. This is a deliberate asymmetry against the tool-proficiency gate below: +capability **flags** fail closed on unknown, while quality **measurements** +(tool proficiency, energy) admit on absent evidence ("unproven, not bad"). + +`tools_present` remains a measurement gate, not a flag gate. Every routable +catalog row has `supports_tools = 1`, so that flag would be inert. What matters +is the measured `tool_use_agentic` proficiency. A model with no measured tool +score is admitted; a measured score below `routing.min_tool_proficiency` is +dropped — but only when the request carries a `tools` array. This gated filter +ships disabled (`null`). + +`has_reasoning_request` is detected in `capabilities.py` but is informational +only. Reasoning stays on tiering: `reasoning_default_enabled` decides the tier, +not `supports_reasoning` (which only means "the endpoint accepts a reasoning +param" and is true for 17 of 19 rows). There is no reasoning capability gate. + +### Pass-through capability check + +When a client pins a real model id in `/v1/chat/completions`, the router +dispatches as asked. Before spending a provider call, `_check_pinned_capabilities` +reads the model's `supports_vision` and `supports_json_mode` flags and returns a +clear 422 if the pin cannot satisfy the request. The pin could never have worked, +so failing early is better than an opaque provider 400. + +### Local Ollama vision fallback + +`local_vision:` in `config.yaml` configures a fallback path for image requests +that find no cloud vision candidate. It ships **disabled** (`enabled: false`). +When enabled and routing returns no selected model, `_run_local_vision` sends +the original messages, with `image_url` parts intact, to a local Ollama model +(`qwen3-vl:4b` by default). The local answer then **replaces** the completion: +`_local_vision_response` returns a normal OpenAI-shaped response, including a +stream-wrapped version for `stream: true`. It does not inject a caption into a +cloud call, because the streaming proxy cannot rewrite bytes mid-stream. + +Budget guards refuse images too numerous or too large before the local call is +made. If the local call fails for any reason, it falls through to the ordinary +`422 No model satisfies the hard filters` so the failure is visible rather than a +silent empty response. ### Serving class: one base model, many rows @@ -279,47 +364,145 @@ and is still correctly absent from `tool_use_agentic`, where its measured 0.33 drops it out of the quality band. That is the eval data earning it the slot rather than a thumb on the scale — no `model_tiers` override was needed. +### Why tools and reasoning stay on their existing signals + +Tools stay on the measured `tool_use_agentic` proficiency gate, not a +`supports_tools` flag gate. Every routable catalog row already has +`supports_tools = 1`, so a flag gate would be inert. The real signal is the +measured proficiency, because the observed failure is a model over-reaching for +tools on a non-agentic prompt. `routing.min_tool_proficiency` captures that +measurement and only applies when the request carries a `tools` array. + +Reasoning stays on tiering (`reasoning_default_enabled`), not on a new flag gate. +`supports_reasoning` only means the endpoint accepts a reasoning parameter, and +that is true for 17 of 19 rows — nearly the whole catalog. `has_reasoning_request` +is detected purely for observation. Making it a gate would add no useful +filtering, because the decision of whether a request needs reasoning is already +encoded in the requested tier. + +The fail-closed asymmetry, stated plainly: **capability flags fail closed on +unknown; quality measurements admit on absent evidence.** A missing +`supports_vision` or `supports_json_mode` flag means "cannot confirm", so the +model is dropped. A missing `tool_use_agentic` score or energy measurement means +"unproven, not bad", so the model is admitted. The first wrong guess is a +guaranteed 400; the second is just an empty data point that the neutral default +handles. + ## Proficiency: category now changes routing -`proficiency_score` is the ONLY category-dependent term in the composite, so +`proficiency_score` is the ONLY category-dependent term in the ranking, so until this table had data, `task_category` could not change a decision at all — the classifier computed it, the router paid ~10s for it, and then it -made no difference. It does now: +made no difference. It does now: sweeping 9 categories x 3 tiers currently +returns **5 distinct winners** at both 50k and 120k of context. -| scoring | distinct models across 27 decisions (9 categories x 3 tiers) | +| context | winners over 27 decisions | |---|---| -| catalog list price | 2 | -| measured cost + eco | 3 | -| proficiency only (cost/eco data missing) | 7 | -| **all three axes live** | **1** | +| 50k | `qwen3.6-35b` (10), `gemma-4-31b` (7), `deepseek-v4-flash` (5), `kimi-k3` (3), `kimi-k3-fast` (2) | +| 120k | `kimi-k2.7-code` (10), `gemma-4-31b` (7), `deepseek-v4-flash` (5), `kimi-k3` (3), `kimi-k3-fast` (2) | -That last row is not a regression, and it is worth understanding before -tuning anything. With cost, eco and proficiency all populated, -`qwen3.6-35b` is **Pareto-dominant**: cheapest AND cleanest in the routable -set, while scoring within 0.15 of the best model on proficiency. On -`summarization` it costs 7x less and emits 65x less carbon than `kimi-k3`, -which beats it 1.00 to 0.85 on quality — no defensible weighting picks -`kimi-k3` from that. One model winning is the correct answer to the question -the weights ask. +**This spread is recent, and how it got here is the useful part.** For a long +time all 27 decisions returned ONE model, and that was the correct answer at +the time rather than a bug: with cost and eco both populated, `qwen3.6-35b` +was Pareto-dominant — cheapest AND cleanest in the routable set, while +scoring within `quality_tolerance` of the best. No defensible weighting picks +anything else out of that. -Proficiency is still doing work underneath. The runner-up ordering reorders -by category — `kimi-k3-fast` climbs to 3rd on `docs_writing` and -`summarization` (proficiency 1.00) and drops out of the top four elsewhere. -So the axis is live; it just cannot overturn a leader that wins on two axes -at once. +Two corrections widened it, and neither was a tuning change: -The 7-model spread appeared only while cost and eco were missing and scoring -neutral 0.5. If a genuinely different balance is wanted, the lever is -`weights` in config.yaml, not more data. +- **Cost stopped being a benchmark average.** It is now priced per request + from catalog prices scaled to the request's shape, so the ranking depends + on the workload instead of on a 400-token reference sweep that no real + traffic resembles. +- **Tier stopped being inferred from price.** `deepseek-v4-flash` was pinned + to tier 1 for being cheap, which excluded it from every tier-2 request + regardless of what any score said. + +Note what changes between the two rows above: only the leader, and only +because of the hard context filter. That is the filter working, not the +scoring disagreeing with itself. + +**If you see one model win everything again, check for dominance before +reaching for config.** One winner is a legitimate outcome. The lever, if a +genuinely different balance is wanted, is `objective.quality_tolerance` (how +large a quality gap must be before it outranks a cost saving) or +`objective.max_energy_per_request` (a hard ceiling). There is no weight to +tune — quality is the objective and cost is the tiebreak. ### What the task set actually found -**Coding does not discriminate these models.** All 13 rows score exactly -1.00 on `coding_general`, `coding_refactor` and `debugging` — and that is -after the tasks were deliberately hardened with touching intervals, full -semver, present-but-falsy defaults, late-binding closures and a binary search -that infinite-loops. Every model in this catalog is simply good at that class -of problem, so cost and eco decide coding routes, which is the right outcome. +**The benchmark could not discriminate these models on coding.** Every row +scored exactly 1.00 on `coding_general`, `coding_refactor` and `debugging` — +and that is after the tasks were deliberately hardened with touching +intervals, full semver, present-but-falsy defaults, late-binding closures and +a binary search that infinite-loops. Every model in this catalog is simply +good at that class of problem, so cost decides coding routes, which is the +right outcome. + +**Real traffic broke one of those ties, which the benchmark never could.** +`coding_general` now spans 0.86-1.00: `glm-5.2-fast` fell to 0.862 over 29 +samples folded in by `feedback.py` from an actual agent session, and crossed +`self_eval_min_samples` on the way, so it reads `self_eval` rather than +`self_eval_thin`. That is the intended shape of this system — the 23-task +benchmark establishes a floor, and your own traffic is what refines it. +`coding_refactor` and `debugging` are still flat at 1.00, awaiting the same +treatment. + +**A 1.00 can also be a sampling artifact, and `docs_writing` was one.** At 2 +samples per model the category read 0.70-1.00 with a model at the ceiling, and +the router paid for that ceiling: `kimi-k3-fast` won every docs route. Six more +benchmark passes moved every score and left NOTHING at 1.00: + +| model | n=2 | n=11-14 | +|---|---|---| +| `kimi-k3` | 0.85 | **0.973** | +| `kimi-k2.7-code` | 0.85 | 0.886 | +| `deepseek-v4-flash` | 0.80 | 0.864 | +| `kimi-k3-fast` | **1.00** | 0.864 | +| `qwen3.6-35b` | 0.85 | 0.800 | +| `gemma-4-31b` | 0.85 | 0.786 | + +The winner moved to `kimi-k2.7-code`, **3.2x cheaper** at 50k of context +($0.0441 -> $0.0136), with no config change — `kimi-k3` scores higher but sits +inside `quality_tolerance`, so cost breaks the tie. `deepseek-v4-flash` +($0.0024) misses the band by 0.009, which is the kind of margin the tolerance +exists to describe rather than a verdict. + +**The whole spread rests on one rubric line, though.** `docs_function` is +effectively saturated — 1.00 on nine of every ten samples — and nearly every +`docs_gotcha` deduction is the same omission: the model documents that order is +preserved, that the first occurrence is kept, and what `key` does, then never +says elements must be hashable. That is real discrimination, since it is a real +property of the function, but one sentence is deciding a category. Treat this +ordering as thinner than n=14 makes it look. + +**The self-judging guard costs sample density, and it shows up here.** Most +models reached n=14; `kimi-k3` and `kimi-k3-fast` reached only 11, because +those two are the ones diverted to the alternate judge `qwen3.6-35b`, which +returns unparseable JSON more often than `kimi-k3` does. The guard is still +right — a model grading its own family is worse than a thinner sample — but +the alternate judges should be picked for parseability, not just for being +someone else. + +The reflex when a category looks flat is to reach for `quality_tolerance`. +Neither tie broken so far was broken that way: `coding_general` opened up when +`feedback.py` folded in real traffic, and `docs_writing` opened up on six more +benchmark passes. Both were samples, not settings. `coding_refactor` and +`debugging` are still flat at 1.00 on 2-3 samples each — which is now a state +this project has mistaken for a measurement once. + +Current spread by category, widest first: + +| category | spread | +|---|---| +| `tool_use_agentic` | 0.33 - 1.00 | +| `summarization` | 0.60 - 1.00 | +| `reasoning_math` | 0.67 - 1.00 | +| `docs_writing` | 0.66 - 0.97 | +| `general_chat` | 0.80 - 1.00 | +| `translation` | 0.85 - 1.00 | +| `coding_general` | 0.86 - 1.00 | +| `coding_refactor`, `debugging` | flat at 1.00 | **What does discriminate is tool use, arithmetic traps, and prose.** `deepseek-v4-flash` scores 1.00 on all three coding categories yet **0.33 on @@ -330,10 +513,42 @@ It over-reaches for tools, which is exactly the failure mode that matters in an agent loop. The router now avoids it for those categories while still picking it for coding. -Everything currently reads `source='self_eval_thin'`: real measurement, but -below `self_eval_min_samples` (2-3 tasks per category per run). Re-run -`eval_proficiency.py` to accumulate — scores fold into a running mean rather -than replacing, so samples add up across runs. +Most rows still read `source='self_eval_thin'` (118 of 132): real +measurement, but below `self_eval_min_samples` at 2-3 tasks per category per +run. The 14 that have crossed it are all `docs_writing`, from the six extra +passes above. Two paths thicken it, and they are complementary — re-run +`eval_proficiency.py` to accumulate benchmark samples, or just use the router +and let `feedback.py` fold in real outcomes. Both fold into a running mean +rather than replacing, so samples add up across runs. + +### A score is only as fresh as the row it was copied to + +Proficiency is a property of the weights, not the queue, so the eval harness +scores one row per family and `propagate_to_variants` copies the result onto +the serving variants — `kimi-k3-flex` gets `kimi-k3`'s number, because no +benchmark rates a `-flex` row separately. + +That copy used to happen **exactly once per variant, ever.** The guard skipped +any row with `self_eval_samples > 0`, meaning "measured directly, do not +overwrite" — but inheritance copies the sample count too, so after the first +propagation an inherited row was indistinguishable from a measured one and was +never refreshed again. `kimi-k3-flex` sat at 0.85/n=2 while `kimi-k3` moved to +0.973/n=11. + +`proficiency.inherited_from` records the provenance that was missing, and the +**migration** was the delicate half, not the fix: `ADD COLUMN` gives every +existing row NULL, which reads as "measured here", so shipping the guard alone +would have permanently frozen the exact rows it exists to unfreeze. The +backfill infers provenance from the harness's own selection rule rather than +guessing — `eval_identities` only ever evaluates standard rows plus flex rows +with **no** standard equivalent, so a flex row that has one was never a +candidate for direct evaluation, whatever its sample count claims. Everything +else keeps NULL, which fails safe: NULL means "do not overwrite", so no real +measurement can be lost to a wrong guess. + +Confirmed on the live database, and on the catalog's one genuine exception — +`glm-5.2` is canary, so `glm-5.2-flex` is the routable row the harness scores +directly, and its NULL is correct. ### Harness bugs this shook out @@ -361,6 +576,66 @@ fails. It immediately caught a check where the expected value was simply wrong — which would have docked every model on a task and been indistinguishable from genuine difficulty. +## Tool competence is read from the request, not guessed at + +Neither local classifier can identify agentic work. Asked to label six +unambiguous tool-use prompts ("read the config then update the manifest", +"run the tests and fix what fails"), `qwen3.5` got 2/6 and `mistral-nemo` +1/6 — and `mistral-nemo`'s misses collapse to `general_chat`, which is also +the configured `fallback_category`, so qwen3.5's crashes land in the same +place. + +That mattered because `tool_use_agentic` has the widest proficiency spread in +the table (0.33-1.00) and `deepseek-v4-flash` — the current winner on coding — +sits at the bottom of it. + +**The fix was not a better classifier.** Whether tools are on the table is +stated in the request: every agent client sends a `tools` array, and +`chat_completions` never looked at it. Reading it is exact and free. + +It is applied as a **hard filter**, not a category override, and the +distinction is load-bearing. The question is not "is this task agentic" but +"can this model be trusted with tools that exist". The recorded failure is +precisely the second one: `deepseek-v4-flash` was given a *non*-agentic prompt +("it is 1:20pm and my meeting is at 3pm, how many minutes away?", both times +supplied) and called two tools rather than subtracting. A model that +over-reaches is a hazard on every request where tools are available, whatever +a classifier would have labelled the task. + +So `routing.min_tool_proficiency` drops any candidate whose measured +`tool_use_agentic` score is below it, but only when the request carries tools: + +| request | winner on `coding_general` @ 50k | +|---|---| +| no tools | `deepseek-v4-flash` ($0.0024) | +| tools present | `qwen3.6-35b` ($0.0041) | + +Verified live through `/v1/chat/completions` with identical bodies differing +only by the `tools` array. The cost of safety here is 1.7x on that route, +paid only where tools exist. + +**It is currently set to `null`, i.e. OFF**, deliberately and pending +experiment. opencode sends `tools` on essentially every request, so with the +filter on, `deepseek-v4-flash` is excluded from ordinary agent traffic and its +~7x cost advantage goes unused; with it off, that advantage applies and a +model measured at 0.33 on tool use handles requests where tools are on the +table. Which is right is an empirical question and the benchmark cannot +answer it — the 0.33 comes from 3 tasks. + +What settles it is `POST /outcome`: run with the filter off, let real pass/fail +reports accumulate, and compare `deepseek-v4-flash`'s `tool_use_agentic` +proficiency before and after. That is the one signal here that knows whether +the work actually worked, and `feedback.py` folds client outcomes in both +directions, so success counts too. + +0.5 sits in the empty band between the only two values the catalog holds +(0.33 and 1.00), so it is not fitted to either. A model with **no** measured +tool score is unproven rather than proven bad and is not dropped — the same +rule as the tier-1 context gate. Config load refuses a +`routing.tool_use_category` that is not a real category, because a name +matching nothing yields NULL for every row and NULL means "do not +disqualify": the filter would silently stop filtering. + ## Tier is an iteration budget, not just a floor A tier used to mean only "do not route below this". It now also buys @@ -477,25 +752,54 @@ truncating the answer — are recorded but excluded. ## The classifier is the latency floor Every routed request pays a full local classification round-trip before a -single upstream token is requested. Measured on `qwen3.5:latest`: +single upstream token is requested, so the classifier model is the single +biggest lever on interactive latency. -| | latency | -|---|---| -| cold (model not resident) | **43s+** — a reload can exceed even the 120s ceiling | -| warm, simple prompt | **~4s** | -| warm, prompt that triggers a long thinking trace | **up to ~25s** | +**The default is `mistral-nemo:12b`, and it replaced `qwen3.5:latest` on +measurement.** Same prompts, same system prompt, `temperature: 0`, cold load +excluded — 14 unambiguous category cases and 5 tier probes: + +| | `qwen3.5` | `mistral-nemo:12b` | +|---|---|---| +| category correct | 10/14 | 9/14 | +| tier correct | 1/5 (2 of them crashed) | **3/5** | +| **hard failures** | **4 of 19 calls (21%)** | **0** | +| latency mean / median / max | 6.6 / 5.5 / **15.6s** | 1.7 / 1.8 / **1.9s** | + +Category accuracy is a wash. What decides it is the failure column and the +tail. Every one of those 4 failures is the runaway-thinking-trace mode below: +~15s spent to produce no JSON, which degrades to `source: "fallback"` — tier +2, `general_chat`. A reasoning model is the wrong tool for a job whose entire +output is ~45 tokens of JSON. + +`mistral-nemo` does not reason by default, which is why its tail is flat: its +slowest call (1.94s) is faster than qwen3.5's median. End-to-end `/route` +went ~10s -> **~1.7s**. Both scored 5/5 on the local verification task, so +the verifier moved with it and only one model stays resident. Four settings keep this usable, each fixing a failure seen in practice: - **`max_retries=0` on the classifier client.** The OpenAI SDK retries twice by default, so `timeout_seconds` silently became a 3x wall-clock bound — a request hung past 250s on a 120s setting and logged nothing. -- **`max_output_tokens: 1024`.** qwen3.5 is a reasoning model and will - otherwise emit an unbounded chain of thought. That cascades: Ollama keeps - generating after the client gives up *and* serializes per model, so one - runaway request queues every later request behind it and the timeouts - spread. 256 was too tight — the trace consumed the budget and the model was - truncated before emitting any JSON, which surfaced as an empty response. +- **`max_output_tokens: 1024`.** Bounds a REASONING model's chain of thought. + Unbounded, it cascades: Ollama keeps generating after the client gives up + *and* serializes per model, so one runaway request queues every later + request behind it and the timeouts spread. 256 was too tight — the trace + consumed the budget and the model was truncated before emitting any JSON, + which surfaced as an empty response. Does not bind for `mistral-nemo`; kept + because it costs nothing unused and is the only guard if a reasoning model + is swapped back in. +- **`max_input_chars: 8000`.** The classifier decides a category and a tier; + it does not need the document. Feeding it one is harmful, not merely + wasteful — on a ~20k-token prompt `qwen3.5` spent 28.7s and returned empty + while `mistral-nemo` spent 41.8s echoing the input back inside its JSON. + Both land on `source: "fallback"`, i.e. 30-40s of local inference buying the + answer an instant failure would have given. Clamped to head + tail (the + instruction sits at one end or the other; never the middle), the same + prompts classify correctly in ~2.2s. Nothing is lost: + `chat_completions` measures the real conversation with + `estimate_prompt_tokens` and takes the larger value. - **`fallback_tier` / `fallback_category`.** A classifier that times out, errors, or returns garbage now degrades to a configured mid tier flagged `source: "fallback"` instead of returning 502/503. The caller is a coding @@ -506,10 +810,71 @@ Four settings keep this usable, each fixing a failure seen in practice: then tier 1 on consecutive calls and routed to two different models. But ~10s of local overhead on every message is a real tax for an interactive -agent, where the upstream answer itself may take 2s. Unaddressed options: -keep Ollama resident (`OLLAMA_KEEP_ALIVE`), classify once per session rather -than per message, cache by prompt hash, use a smaller classifier, or skip -classification for short prompts. This is a design call, not a tuning one. +agent, where the upstream answer itself may take 2s. Still unaddressed: +classify once per session rather than per message, cache by prompt hash, or +skip classification for short prompts. + +### "Local" means your hardware, not this machine + +The premise is that a local LLM classifies the task before a cloud model +answers it. That does not require the GPU to be in the machine you are typing +on, and usually it isn't — most developers already have WireGuard or a VPN +back to a home lab. So the normal shape is router and editor on the laptop, +Ollama on the workstation, `classifier.base_url` pointing across the tunnel. + +`classifier.base_url` / `api_key_env` / `model` take any OpenAI-compatible +endpoint. Verified end to end against a non-loopback address, classifier and +verifier both. + +Ollama binds `127.0.0.1` by default, so this fails with connection-refused +until the serving host applies `deploy/ollama-over-vpn.conf`. Bind it to the +VPN address, not `0.0.0.0`: Ollama has no auth of any kind, so anything that +can reach the port can run inference and enumerate your models, and `0.0.0.0` +publishes it on whatever wifi the laptop is sitting on. + +A cloud endpoint also satisfies that interface, and measured better than the +local one — worth knowing before assuming local is the cheap side. Five +prompts, same system prompt, `temperature: 0`: + +| | local `qwen3.5` (RTX 6000) | NeuralWatt `deepseek-v4-flash` | +|---|---|---| +| mean latency | **11.58s** (4.95-15.76) | **1.02s** | +| categories agreed with the label | 2 of 4 | **5 of 5** | +| hard failures | 1 of 5 (empty after 15.76s) | 0 | +| energy per call | ~7e-05 kWh, on your meter | 1.17e-05 kWh attributed | +| cost per 1,000 calls | electricity + 6.6GB resident | **$0.093** (0.19% of quota) | + +11x faster, more accurate, and less attributed energy, on a machine with a +24GB card sitting idle. The one hard failure is the documented +runaway-thinking-trace mode — `qwen3.5` spent 15.76s and emitted no JSON, +degrading to `source: "fallback"`, i.e. a silent tier-2 guess. +`deepseek-v4-flash` does not reason by default, so that mode does not exist +for it. End-to-end `/route` goes ~10s -> **~1.0s**. + +Local classification was assumed to be the cheap option because local compute +felt free. It is not free, it is just unbilled — the same substitution the +cost axis had to unlearn. The shipped default is still local Ollama, because +switching spends quota and that is a deployment choice, not a code one. + +### The verifier follows, but only to another Ollama + +The local LLM check speaks Ollama's **native** `/api/chat` (the only way to +set `think: False`), so it follows the classifier across a VPN but not to a +cloud provider. It used to derive its URL by stripping `/v1` off +`classifier.base_url`, which meant moving the classifier at all would have +pointed it at `/api/chat`. It now has its own +`verification.base_url` and `verification.model`. + +`verification.model` may be null only while both run on one host. Config load +**refuses** the null once the hostnames differ, because the failure is silent: +observed directly, with the classifier on NeuralWatt the verifier POSTed +`deepseek-v4-flash` to `localhost:11434`, 404'd, caught it, logged "local +verification unavailable" and recorded no sample. Verification would have +looked enabled while producing nothing. + +Structural verification needs no model at all — it is pure Python — so +`verification.local_llm_enabled: false` leaves a host with no local inference +fully functional, minus the refusal/incoherence class of failure. Note also that opencode sends **~32K prompt tokens** of system prompt and tool definitions on a trivial request, so the measured-size floor in @@ -543,13 +908,25 @@ that request was two orders of magnitude low. 4. **Session-directory attribution picks the wrong directory.** The opencode plugin now genuinely reports — 31 accepted, 3 refused as ambiguous, 22 `succeeded` / 4 `failed` from real test runs, all folded in. But the - "most frequent path" heuristic resolved the session to - `.venv/lib/python3.14/site-packages/c2pa` 24 times versus - `~/Sources/fieldwitness` 22, because reading a dependency's source - outweighed editing the project. It degrades safely — an ambiguous session - is refused rather than misattributed, which is why the 3 × 409 — but the - heuristic needs to weight *writes* over reads, or anchor on the client's - cwd instead of a path histogram. + "most frequent path" heuristic resolved one session to a **dependency's + source directory** inside the project's own virtualenv (24 mentions) + rather than the project being edited (22), because reading a library's + source outweighed writing the code under test. It degrades safely — an + ambiguous session is refused rather than misattributed, which is what the + 3 × 409 were — but the heuristic needs to weight *writes* over reads, or + anchor on the client's cwd instead of a path histogram. + +5. **Local energy is not on the ledger.** The router meters what NeuralWatt + bills and reports, but the electricity its own classifier and verifier burn + on local hardware is invisible to it. The whole premise is spending cheap + local compute to avoid wasting expensive cloud compute, and right now only + one side of that trade is measured — which is how "local is free" survived + as an assumption long enough to be wrong (a hosted classifier measured + faster, more accurate and lower attributed energy than the local one). + Closing it means metering local draw (nvidia-smi / RAPL / a smart plug) and + pricing it against a real tariff — utility rate data, ideally the user's own + plan, including time-of-use bands. That would also make the local-vs-cloud + comparison an actual number rather than a shrug. ## Known open questions @@ -563,19 +940,49 @@ that request was two orders of magnitude low. - Three models still fail a split-half stability check at 7 samples. Is the instability real (variable serving conditions) or an artifact of when the sweep ran? Re-sweeping at a different hour would tell. -- The tier-1 composites sit within 0.009 of each other across a 40x cost - range, because min-max compresses once one candidate is far cheaper than - the rest. Ranking is right, but margins are thin — proficiency data will - swing these easily, which is the intent. +- Answered, and the question no longer parses: tier-1 composites used to sit + within 0.009 of each other because min-max normalization compressed them. + There is no composite any more — ranking is quality first, cost as the + tiebreak inside `quality_tolerance` — so nothing normalizes and nothing + compresses. +- Answered: the eval set exists (`evals/tasks.yaml`, 23 tasks, four scoring + kinds) and `tests/test_task_set.py` keeps it honest. The open part is + narrower now — `coding_refactor` and `debugging` are still flat at 1.00 + across every model, so those tasks discriminate nothing and either need + hardening again or should be conceded as non-discriminating. **Try samples + before hardening.** `docs_writing` looked flat at the top too, and six more + passes spread it 0.66-0.97 without touching a task; two samples per model is + not enough to tell a saturated task from an unsampled one. - How much context-assembly (RAG-style retrieval) belongs in the classifier step vs. a separate pre-step? Leaning decoupled, undecided. -- Self-run eval set: minimum viable task set to start proficiency scoring - without it becoming its own maintenance burden? - Should `eco_score` use real-time grid carbon intensity per request or a stable per-model average? Currently the latter, from the reference sweep. `grid_carbon_intensity` and `grid_id` are logged per observation, so this stays answerable from data without a re-run. +## Config is strict: an unknown key is an error + +Pydantic ignores extra keys by default, which means a typo or a misplaced +setting loads cleanly, does nothing, and still looks configured. Every config +model now inherits `StrictModel` (`extra="forbid"`), so both of these fail at +load rather than silently: + +``` +verification.max_input_chars # right key, wrong section +routing.min_tool_proficency # sic +``` + +This is not hypothetical. `max_input_chars` shipped into the `verification:` +block instead of `classifier:` and was accepted and discarded — it happened to +match the code default, so behaviour was correct and the file was a lie. +Editing it would have done nothing. + +The corollary worth keeping: **every knob belongs in `config.yaml`, not only +in a Pydantic default.** A default the file never mentions is invisible to +anyone tuning it. `classifier.outcome_attribution_window_seconds` was removed +in the same pass — it was declared, never read, and shadowed the +`verification` one that actually is. + ## Setup ```bash @@ -589,12 +996,22 @@ python config.py # sanity-check config loads python -m uvicorn dispatcher:app --reload ``` -Ollama must be running locally with the classifier model pulled — the name -must match `classifier.model` in `config.yaml`: +Then set what is deployment-specific in `config.yaml`: `classifier.model` and +`classifier.base_url` for your Ollama, and `objective.plan_kwh_per_period` to +your own plan's quota (it is reported in `/health` as burn against the +allowance; it does not gate anything). + +Ollama must be reachable with the classifier model pulled — the name must +match `classifier.model` in `config.yaml`: ```bash -ollama pull qwen3.5:latest +ollama pull mistral-nemo:12b ``` +It does not have to be on this machine. To use one across a VPN, point +`classifier.base_url` and `verification.base_url` at it and apply +`deploy/ollama-over-vpn.conf` on the serving host — Ollama binds loopback-only +by default and will otherwise refuse. + `requirements.txt` is pinned. Recreating the venv with the old `>=` ranges jumped openai 2.53 -> 3.0 and httpx -> httpx2 without warning; a service that restarts on boot shouldn't change its dependency tree underneath itself. @@ -607,8 +1024,9 @@ resolves. ## Run as a service -`deploy/` holds three systemd **user** units — see `deploy/README.md` for -install and operation. In short: +`deploy/` holds five systemd **user** units (dispatcher, plus a timer and a +oneshot service each for the poller and the seed sweep) and one drop-in for a +*system* Ollama — see `deploy/README.md` for install and operation. In short: ```bash echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env @@ -626,12 +1044,17 @@ The service holds a billable API key and has **no auth of its own**. Loopback bind is the only thing standing between the open internet and your allowance; add auth before widening `--host`. +The same applies to an Ollama shared over a VPN — it has no auth either, so +`deploy/ollama-over-vpn.conf` binds it to the VPN address rather than +`0.0.0.0`, which would publish it on whatever network the client happens to +be on. + ## Pointing a coding agent at it The `/v1` endpoints are OpenAI-compatible, so any normal client works — opencode, an SDK, plain curl. Repo-local `opencode.json` is already wired up, -so `cd ~/Sources/6krrt && opencode` routes by default. For global use, merge -`provider.llm-router` into `~/.config/opencode/opencode.json`. +so running `opencode` from a clone of this repo routes by default. For global +use, merge `provider.llm-router` into `~/.config/opencode/opencode.json`. | model name | behavior | |---|---| diff --git a/README.md b/README.md index f59f35e..c5bb3b3 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,21 @@ # Local LLM Model Router -A router that uses a local model (served via Ollama on an RTX 6000, 24GB) to -classify incoming coding/documentation tasks — category, tier, required -context size — and dispatch each task to the cheapest/best-fit open-weight -model on **Neuralwatt Cloud**, weighted by cost, ecological impact -(Neuralwatt exposes real energy-per-request data), and per-category -proficiency. +A router that uses a local model (served via Ollama) to classify incoming +coding/documentation tasks — category, tier, required context size — and +dispatch each task to the cheapest/best-fit open-weight model on **Neuralwatt +Cloud**, weighted by cost, per-category proficiency, and a per-request energy +ceiling. + +The local model does the classifying, so it wants a GPU, but not necessarily +*your* GPU — `classifier.base_url` takes any OpenAI-compatible endpoint, so +the usual shape is the router on your laptop and Ollama on a workstation +across a VPN. See [Setup](#setup). + +**Numbers in this README are measurements, not specifications.** They come +from one deployment against one provider account, and the catalog, prices, +grid intensity and pool load all move. They are here because the reasoning +behind a design choice is worth more than the choice, and re-running the +measurement is how you check whether it still holds for you. Neuralwatt is the only provider. The `provider` column and the `(model_id, provider)` primary key stay so a second provider can be added @@ -16,10 +26,10 @@ later without a migration. | Dimension | Detail | |---|---| | **Cost model** | Per-kWh, not per-token. Flat $8.00/kWh measured across the catalog. | -| **Latency** | ~4 s warm classification, up to ~120 s on cold start. Dispatcher response is unaffected — verification runs async. | +| **Latency** | Classification is the floor: ~5-15 s warm on a local reasoning model, up to ~120 s cold. A cloud classifier measured ~1 s. Verification is async and never blocks the response. | | **Quality** | Quality is the objective; cost is a per-request kWh ceiling plus a tiebreak. Eco is logged but no longer optimized. Proficiency blends external benchmarks with a self-run eval harness. | | **Verification** | Every response structurally checked (free) + local LLM spot-check on long prose answers (~6 s, never blocks). Failures fold back into proficiency via `feedback.py`. | -| **Fault tolerance** | Classifier failure degrades to a mid-tier fallback rather than 502/503. Retry count is zero to prevent silent 3× time-outs. | +| **Fault tolerance** | Classifier failure degrades to a mid-tier fallback rather than 502/503. SDK retry count is zero, to prevent `timeout_seconds` silently becoming a 3× wall-clock bound. | | **API surface** | OpenAI-compatible `/v1` endpoints, streaming chunk proxy with SSE telemetry scraping. | ## Verification Pipeline @@ -109,7 +119,7 @@ Key behaviors: ``` ┌─────────────────────┐ - incoming task ───▶│ Local Classifier │ Ollama (qwen3.5:latest) + incoming task ───▶│ Local Classifier │ Ollama (mistral-nemo:12b) │ - task_category │ ~4s warm, ~120s cold cap │ - task_tier │ temperature: 0, max 1024 tokens │ - required_context │ max_retries: 0 (silent 3× cap guard) @@ -163,19 +173,19 @@ Key behaviors: | Layer | Technology | |---|---| -| **Language** | Python 3 (IO-bound provider APIs; iteration speed matters more than raw speed) | +| **Language** | Python 3.10+ (IO-bound provider APIs; iteration speed matters more than raw speed) | | **Framework** | FastAPI + uvicorn | | **Database** | SQLite (`router.db`) — decision table, energy observations, proficiency, verifications | -| **Local Classification** | Ollama (OpenAI-compatible at `localhost:11434/v1`) | -| **Local Model** | `qwen3.5:latest` | +| **Local Classification** | Ollama, OpenAI-compatible — `localhost:11434/v1` or an Ollama across your VPN | +| **Local Model** | `classifier.model` — `mistral-nemo:12b` by default; any Ollama model works | | **Cloud Provider** | Neuralwatt only | | **Config** | `config.yaml` loaded & validated by Pydantic (`config.py`) | | **OpenAI Client** | `openai==3.0.0` (official SDK) | | **HTTP** | `requests` for poller, `httpx` (via openai/uvicorn) | -| **Testing** | `pytest` — 207 tests across 11 files | +| **Testing** | `pytest` — 356 tests across 19 files, all offline | | **Config Files** | `config.yaml`, `leaderboards.yaml`, `evals/tasks.yaml` | | **Deployment** | systemd user units (`.service` + `.timer` files in `deploy/`) | -| **Integration** | Pre-configured in `opencode.json` — `cd` into repo uses router by default | +| **Integration** | `opencode.json` in the repo routes through it by default; any OpenAI-compatible client works | ## Pinned Dependencies (requirements.txt) @@ -212,7 +222,8 @@ restarts on boot shouldn't change its dependency tree underneath itself. | **`verification.py`** | Two-layer check: structural parse (always) + local LLM spot-check (async, size-gated). Never executes model output | Pure | | **`feedback.py`** | Folds observed verification failures into `proficiency`; routing learns from real traffic | Yes (DB) | | **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) | -| **`config.py`** | YAML loader + Pydantic validators (weights sum to 1, valid tiers, etc.) | Yes (file) | +| **`iteration.py`** | Retry budget per tier, and matching the retry to the failure kind | Pure | +| **`config.py`** | YAML loader + Pydantic validators (blend weights sum to 1, valid tiers, endpoints separately addressable) | Yes (file) | ## Decision Table Schema (SQLite) @@ -270,6 +281,7 @@ parses them into `access_level` and `routing.allowed_access_levels` (default | `self_eval_samples` | INTEGER | Evidence count for blending threshold | | `blended_score` | REAL | Computed: `w_lb × lb + w_se × se` | | `source` | TEXT | `blended` \| `self_eval` \| `self_eval_thin` \| `leaderboard` | +| `inherited_from` | TEXT | Model this row was copied from, NULL if measured directly | | `last_updated` | TEXT | ISO8601 | **Category set** (9 categories, defined in `config.yaml`): @@ -302,8 +314,14 @@ exclude it can. **Proficiency inheritance:** `propagate_to_variants` copies evaluated `scores` to equivalent serving variants (same weights, same reasoning -setting, same context pool), but only when the variant has no measured -data itself. A `-fast` row is **not** equivalent to its `-standard` sibling. +setting, same context pool), but never over a row that was measured +directly. A `-fast` row is **not** equivalent to its `-standard` sibling. + +"Measured directly" is `inherited_from IS NULL`, not `self_eval_samples > 0` +— inheritance copies the sample count too, so sample count alone cannot tell +an inherited row from a measured one, and using it meant a variant inherited +exactly once and then froze forever. `ensure_columns()` adds the column and +backfills provenance on databases that predate it. ### `energy_observations` — per-request telemetry @@ -355,11 +373,43 @@ failure — it means the checker had nothing to say, not that the model failed. ## Weighted Scoring -Three hard filters are applied **before** scoring (not weighted — outright disqualification): +Four hard filters are applied **before** scoring (not weighted — outright disqualification): 1. `effective_context_window ≥ required_context_tokens` 2. `tier ≥ required_tier` (from classifier) 3. Serving class compatible with request's `latency_tolerance`; `access_level` reachable +4. `tool_use_agentic` proficiency ≥ `routing.min_tool_proficiency`, **only when + the request carries a `tools` array** (ships disabled — `null`) +5. `supports_vision = 1` when the request carries `image_url` parts; `NULL` + fails closed (an unknown flag means the capability cannot be confirmed) +6. `supports_json_mode = 1` when `response_format.type` is `json_object` or + `json_schema`; `NULL` also fails closed + +Filters 4–6 are read from the request body rather than inferred. A `tools` +array states whether tool definitions are on the table, `image_url` parts state +whether vision is needed, and `response_format` states whether JSON mode is +needed. Local classifiers identified unambiguous tool-use prompts only 1-2 +times in 6, so inferring capability requirements does not work; the request +states them exactly and for free. + +The tool gate is a **quality measurement** filter: a model with no measured +`tool_use_agentic` score is admitted, because "unproven" is not "proven bad". +Only a measured score below `routing.min_tool_proficiency` is dropped. The +vision and JSON-mode gates are **capability flag** filters. There every +routable catalog row has `supports_tools = 1`, so a flag gate would be inert; +vision and JSON mode are not universal, and an absent or `NULL` flag means +"cannot confirm the capability", so the model is dropped. A wrong guess on a +capability flag is a guaranteed provider 400. + +The tool filter ships **off** (`null`), because agent clients send `tools` on +nearly every request, so enabling it excludes the cheapest model from ordinary +agent traffic. Whether that trade is worth it is an empirical question best +settled with `POST /outcome` data rather than a 3-task benchmark score. Set it +to `0.5` to turn it on. The vision and JSON-mode gates ship **on**, because a +wrong guess is a guaranteed 400. + +Config is strict (`extra="forbid"`): a misspelled or misplaced key fails at +load instead of being silently ignored. ``` 1. drop candidates whose measured energy exceeds objective.max_energy_per_request @@ -371,53 +421,78 @@ Three hard filters are applied **before** scoring (not weighted — outright dis | Setting | Value | Notes | |---|---|---| | `quality_tolerance` | 0.10 | Measurement noise, not preference: scores rest on 2-3 samples, so smaller gaps are sampling variation | -| `max_energy_per_request` | null | Per-request kWh ceiling. The plan is a 6.25 kWh quota — a wall, not a bill | -| `plan_kwh_per_period` | 6.25 | Reported in `/health` as burn against the allowance | +| `assumed_cache_rate` | 0.917 | Share of prompt tokens served from the provider's prefix cache. Agent clients resend the conversation each turn, so most of it hits. Measured token-weighted over 40.7M tokens; measure your own against the provider's session view | +| `assumed_completion_tokens` | 500 | Completion length assumed when pricing a candidate | +| `max_energy_per_request` | null | Per-request kWh ceiling — a wall, not a bill. Null disables it | +| `plan_kwh_per_period` | 6.25 | **Set this to your own plan's quota.** Reported in `/health` as burn against the allowance; it does not gate anything | This replaced a weighted blend (cost 0.4 / eco 0.2 / proficiency 0.4). Measurement retired it: turning the cost weight from 0.4 to **zero** changed the winner in only 2 of 6 categories, so the blend was never steering on quality — while 60% of every decision adjudicated fractions of a cent. -**Why cost ≠ list price:** Neuralwatt bills flat $8.00/kWh. The catalog's -`input_per_million` / `output_per_million` are **not** what gets charged. -Measured: `cost_usd / energy_kwh = 8.00` across every model. A model listing -at $4/1M can cost 10× more than one at $15/1M on the same prompt. +**Cost is priced per request, from catalog token prices scaled to the +request's shape** — prompt size, `assumed_completion_tokens`, +`assumed_cache_rate`. It is deliberately *not* a benchmark average. + +Neuralwatt bills flat per-kWh rather than per-token, so list price is not what +gets charged. Scoring used measured billed cost from a fixed 400-token +reference sweep for exactly that reason — and that was wrong for real traffic, +because the ranking depends on the workload's *shape*, not just the model. On +a 400-token prompt one model looked 3.2× cheaper than another; on a realistic +70k-token prompt the same pair inverted and the second was 5.0× cheaper. The +provider's attribution ratio moves with prompt size, so a fixed-shape +benchmark cannot rank models for a workload of a different shape. + +List price is still not what is billed, but billing is capped at a multiple of +it, so it tracks the real ordering and bounds it — and it is free, needs no +sweep, and refreshes whenever the poller runs. **Why cost ≠ eco:** Cost tracks energy (kWh), but carbon is energy × grid -intensity. Grid intensity spans ~49 gCO2/kWh (`FI`) to ~442 (`US-MIDA-PJM`) — roughly a 9× spread, -and it moves: the provider's own 24h blended figure was 145.4. The models disagree: `glm-5.2-fast` is +intensity. Grid intensity spanned ~49 gCO2/kWh (`FI`) to ~442 +(`US-MIDA-PJM`) when measured — and it moves with time of day. The models +disagree: `glm-5.2-fast` is 2nd cheapest but 6th cleanest; `kimi-k3-flex` draws 3.7× less energy than `kimi-k2.7-code` while emitting 3.6× more carbon. Collapsing them picks a side. -### What routing actually does today — expect one model +### What routing actually returns, and why it moves -Run `/route` across all 9 categories and 3 tiers and every one of those 27 -decisions currently returns **`qwen3.6-35b`**. That is not a bug and not a -misconfiguration, so it is worth stating plainly before you go looking for -one. +Sweeping `/route` across every category and tier is the fastest way to see +whether your data is doing anything. On the deployment this was written +against, 9 categories × 3 tiers currently yields **5 distinct winners** +(`qwen3.6-35b`, `gemma-4-31b`, `deepseek-v4-flash`, `kimi-k3`, +`kimi-k3-fast`). -On this catalog `qwen3.6-35b` is **Pareto-dominant**: cheapest *and* cleanest -in the routable set, while scoring within 0.15 of the best model on -proficiency. On `summarization` it costs 7× less and emits 65× less carbon -than `kimi-k3`, which beats it 1.00 to 0.85 on quality — no defensible -weighting picks `kimi-k3` there. One model winning is the correct answer to -the question the weights ask. +That number is a diagnostic, not a target, and it is worth knowing what each +outcome means: -Proficiency still does work underneath: runner-up ordering reorders by -category (`kimi-k3-fast` climbs to 3rd on `docs_writing` and `summarization` -where it scores 1.00, and leaves the top four elsewhere). The axis is live; -it simply cannot overturn a leader that wins on two axes at once. +- **One winner everywhere** is a legitimate answer, not a misconfiguration. + It happened here: with cost, eco and proficiency all populated, one model + was Pareto-dominant — cheapest *and* cleanest in the routable set while + scoring within `quality_tolerance` of the best. No defensible weighting + picks anything else. If you see this, check whether the leader really is + dominant before reaching for the config. +- **Winners that change with context size** are the hard filters working. + A model is dropped once `required_context_tokens` exceeds its window, so a + long session can change model mid-conversation. Past the largest window, + `/v1/chat/completions` returns 422 naming the constraint rather than + silently truncating. +- **Winners that change by category** mean proficiency is live. That is the + only category-dependent term, so until the `proficiency` table has data, + `task_category` cannot change a decision at all — the classifier computes + it and the router pays for it for nothing. -The spread widens when the leader stops being eligible. Past ~94,196 tokens -of context `qwen3.6-35b` is filtered out and `glm-5.2-fast` takes over; past -~790K nothing qualifies and `/v1/chat/completions` returns 422 naming the -constraint rather than truncating. So on a large codebase you will see the -model change mid-session — that is the context filter working. +The spread here widened for two reasons worth copying: cost became a +per-request estimate rather than a benchmark average, and tier stopped being +inferred from price alone. Both had been quietly excluding a cheap +large-context model from every request above tier 1. -If a different balance is wanted, the lever is `weights` in `config.yaml`, -not more data. +If you want a different balance, the levers are `objective.quality_tolerance` +(how big a quality gap must be before it outranks cost) and +`objective.max_energy_per_request` (a hard ceiling). There is no weight to +tune — quality is the objective and cost is the tiebreak, which replaced an +earlier weighted blend. ## API Endpoints @@ -455,6 +530,11 @@ structural verification verdict surfaces in the `X-Router-Verification` header so a client can inspect it without parsing the response body. Valid values: `ok`, `truncated`, `malformed`, `unverifiable`, `none`. +**Capability 422s**: When no model survives the hard filters, the 422 names the +active constraints. That now includes "vision-capable model" or +"json-mode-capable model" when the request carried images or a JSON-mode +`response_format`, alongside the existing context/tier/latency/tool reasons. + ## Quick Usage ```bash @@ -478,8 +558,65 @@ curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application # Admit flex rows for overnight/async work curl -s -X POST localhost:8080/route -H 'content-type: application/json' \ -d '{"task":"nightly code review","latency_tolerance":"batch"}' + +# Image URL request (routed only to vision-capable catalog rows) +curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \ + -d '{"model":"auto","messages":[{"role":"user","content":[{"type":"text","text":"Describe this"},{"type":"image_url","image_url":{"url":"data:image/gif;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs="}}]}]}' + +# JSON-mode request via response_format +curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \ + -d '{"model":"auto","messages":[{"role":"user","content":"Return a JSON object with field answer"}],"response_format":{"type":"json_object"}}' ``` +## Logging and Traceability + +One `route` line per request says what was decided; one `dispatch` line says +what it cost. Both carry a trace id, and `rid`/`sess` are columns in +`energy_observations`, so a journal line pivots to its database row and back. + +``` +route id=r9116d9 cat=coding_refactor tier=2 ctx=500 src=classifier tools=0 + latency=interactive cand=8 pick=deepseek-v4-flash est_usd=0.00016296 ms=1868 +dispatch id=r9116d9 model=deepseek-v4-flash rid=chatcmpl-... sess=c090d751 + p_tok=59511 c_tok=415 kwh=4.8e-05 usd=0.000783 verdict=ok total_ms=5692 +``` + +At `debug` it also says *why* — the classifier's answer and timing, every +candidate that was dropped and by which filter, and the ranking with scores: + +``` +classify id=r9116d9 cat=coding_refactor tier=2 confidence=0.95 ms=1863 +filter id=r9116d9 model=gemma-4-31b reason=tier(1<2) +filter id=r9116d9 model=glm-5.2-short reason=access_level(preview) +filter id=r9116d9 model=kimi-k3-flex reason=latency_class(flex) +rank id=r9116d9 pos=0 model=deepseek-v4-flash prof=1 est_usd=0.00016296 +``` + +**No conversation text is logged at any level**, prompt or answer — prompts +here run 60k–150k tokens and the journal is on disk. A test enforces it. + +Set the level in `config.yaml` (`logging.level`), or override it without +touching a tracked file: + +```bash +systemctl --user edit llm-router # Environment="LLM_ROUTER_LOG_LEVEL=debug" +systemctl --user restart llm-router +``` + +Reading it back: + +```bash +journalctl --user -u 'llm-router*' -f # quote the glob in zsh +journalctl --user -u llm-router -f -o cat # message only +journalctl --user -u llm-router -p warning # severity actually filters +journalctl --user -u llm-router --grep 'chatcmpl-abc123' # one request's trail +journalctl --user -u llm-router --grep ' id=r9116d9' # every stage of it +``` + +`-p` works because the service emits journald priority prefixes when — and only +when — systemd owns its stderr. Run uvicorn in a terminal and the lines come +out clean. + ## Scheduled Jobs (systemd) Five user units. See `deploy/README.md` for full install/operate instructions. @@ -487,17 +624,23 @@ Five user units. See `deploy/README.md` for full install/operate instructions. | Unit | Schedule | What it runs | |---|---|---| | `llm-router.service` | Continuous | FastAPI dispatcher | -| `llm-router-poller.timer` | 2 min after boot, then every 2 h | `poller.py` → `tier.py` | -| `llm-router-seed.timer` | Every 6 h | Small `seed_energy.py` sweep | +| `llm-router-poller.timer` | 2 min after boot, then every 2 h | triggers the poller unit | +| `llm-router-poller.service` | oneshot | `poller.py` → `tier.py` | +| `llm-router-seed.timer` | Every 6 h | triggers the seed unit | +| `llm-router-seed.service` | oneshot | small `seed_energy.py` sweep | **The poller timer is load-bearing, not optional.** `freshness.stale_after_days` is 3 with `exclude_stale: true` — an unpolled catalog marks every row stale in 3 days and the router returns zero candidates for everything. **The seed timer spans time.** Energy attribution drifts with pool load -across hours (~50× for `deepseek-v4-flash` between sweeps), so a single -sweep measures one moment. The median has to span time — every 6 h sweep -accumulates into a time-weighted median automatically. +across hours (~50× for one model between sweeps), so a single sweep measures +one moment. The median has to span time — every 6 h sweep accumulates into a +median-across-time automatically. + +This now feeds `eco` only. Cost is priced per request from catalog prices, so +routing no longer depends on the sweep at all; disabling this timer costs you +carbon figures, not routing quality. ## Self-Eval Harness (`eval_proficiency.py`) @@ -517,13 +660,40 @@ python eval_proficiency.py --dry-run # plan only ## Classifier Reliability Notes -Several tuning decisions keep the classifier from cascading failures: +The classifier is the one blocking LLM call on the request path, so it sets +the latency floor for every routed request. `classifier.base_url` / +`api_key_env` / `model` accept any OpenAI-compatible endpoint — a local +Ollama, an Ollama on another machine across a VPN, or a cloud model. + +**Pick a non-reasoning model.** The classifier's entire output is ~45 tokens +of JSON, and a reasoning model spends its budget getting there. Replacing +`qwen3.5` with `mistral-nemo:12b` on identical prompts, cold load excluded: + +| | `qwen3.5` | `mistral-nemo:12b` | +|---|---|---| +| category correct | 10/14 | 9/14 | +| tier correct | 1/5 (2 crashed) | 3/5 | +| **hard failures** | **4 of 19 calls** | **0** | +| latency mean / max | 6.6s / **15.6s** | 1.7s / **1.9s** | + +Accuracy is a wash; the failure column and the tail are not. Every failure was +an unbounded thinking trace producing no JSON, which degrades to +`source: "fallback"` after ~15s. End-to-end `/route` went ~10s → ~1.7s. + +Several settings keep it from cascading failures: + +- **`max_input_chars: 8000`** clamps what the classifier is shown to head + + tail. It decides a category and a tier; it does not need the document, and + feeding it one is harmful rather than wasteful — on a ~20k-token prompt one + model returned empty after 28.7s and the other spent 41.8s echoing the input + into its JSON. Clamped, the same prompts classify in ~2.2s. + - **`max_retries: 0`** on the classifier client. The OpenAI SDK retries twice by default, so `timeout_seconds` silently became a 3× wall-clock bound — a request could hang past 250s on a 120s setting with no log. - **`max_output_tokens: 1024`** bounds a reasoning model's chain of thought. - Without it, `qwen3.5` emits unbounded traces — Ollama keeps generating after + Without it, a reasoning model emits unbounded traces — Ollama keeps generating after the client gives up AND serializes per model, so one runaway request queues every later request behind it. 256 was tried first and was too tight; the trace would consume the whole budget and the model would be truncated before @@ -539,27 +709,39 @@ Several tuning decisions keep the classifier from cascading failures: ## Pointing a Coding Agent at It -The `/v1` endpoints are OpenAI-compatible. Repo-local `opencode.json` is -already wired up (`cd ~/Sources/6krrt && opencode` routes by the router by -default). For global use, merge `provider.llm-router` into +The `/v1` endpoints are OpenAI-compatible, so any normal client works — +opencode, an SDK, plain curl. A repo-local `opencode.json` is included, so +running `opencode` from a clone of this repo routes through the router by +default. For global use, merge its `provider.llm-router` block into `~/.config/opencode/opencode.json`. Virtual model names: - `auto` → router picks, interactive mode (flex rows excluded) - `auto:batch` → router picks, admits flex rows for async work +**opencode image support**: The repo's `opencode.json` declares +`"modalities": {"input": ["text", "image"]}` for every `llm-router` model. +This is required: opencode strips image parts client-side unless the provider +model declares image input. Without it, images never reach the router at all. +For global use, make sure the merged `~/.config/opencode/opencode.json` entry +carries the same modality block. + ## Testing ```bash -python -m pytest # run all 207 tests +python -m pytest # 356 tests python -m pytest --cov # with coverage ``` +No test calls a provider or a local model — the pure modules take rows and +config as arguments, so the suite runs offline on a clean checkout. + + | Test file | What it covers | |---|---| | `test_scoring.py` | `normalize_inverted`, `composite_score`, None-handling | | `test_routing.py` | Hard filters, candidate selection & ranking across all 9 categories × 3 tiers | -| `test_tiering.py` | Tier resolver precedence: override → reasoning → cost → mid | +| `test_tiering.py` | Tier resolver precedence: override → reasoning → cost + context window → mid | | `test_apply_tiering.py` | DB tiering pass, sanity guards | | `test_poller_parsing.py` | Serving class, base model, access level parsing | | `test_proficiency.py` | Blending rule, accumulation | @@ -568,9 +750,17 @@ python -m pytest --cov # with coverage | `test_task_set.py` | Reference solutions validating every `code` task's checks & `exact` answers | | `test_verification.py` | Structural checks on every language, verifier edge cases, local-LLM gating | | `test_feedback.py` | Failure identification, idempotency, attribution filtering | +| `test_iteration.py` | Retry budget per tier, and matching the retry to the failure kind | +| `test_session_identity.py` | Outcome attribution: session matching, ambiguity refusal | +| `test_config_endpoints.py` | Classifier and verifier are separately addressable; guards on the split | ## Setup +**You need:** a Neuralwatt API key, Python 3.10+ (suite verified on 3.10 and +3.14), and an Ollama reachable from wherever this runs with a classifier model +pulled. Nothing else is assumed about the host — routing itself is SQLite and +arithmetic. + ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt @@ -578,15 +768,50 @@ sqlite3 router.db < schema.sql cp .env.example .env # fill in NEURALWATT_API_KEY python poller.py # populate the catalog python tier.py # resolve tiers -python seed_energy.py # seed reference energy observations (~5 per model) +python config.py # sanity-check config loads python -m uvicorn dispatcher:app --reload ``` -Ollama must be running locally: +Then edit `config.yaml` for your own setup — at minimum: + +| Key | Why | +|---|---| +| `classifier.model` | must match a model `ollama list` reports | +| `classifier.base_url` | where that Ollama actually is | +| `objective.plan_kwh_per_period` | your plan's quota; `/health` reports burn against it | +| `objective.assumed_cache_rate` | 0.917 was measured from one client's traffic (40.7M tokens). Check yours against the provider's per-session cache-hit figures | + +`python seed_energy.py` is optional. It sweeps a fixed reference workload to +populate `eco`, which is logged but is not an objective — routing works +without it. It costs real money and quota, so it is not in the path above. + +### Where Ollama lives + ```bash -ollama pull qwen3.5:latest +ollama pull mistral-nemo:12b # or whatever you set as classifier.model ``` +It does not have to be on the machine running the router; the box with the +GPU usually isn't the laptop. To use one across a VPN, point **both** +endpoints at it: + +```yaml +classifier: + base_url: "http://:11434/v1" +verification: + base_url: "http://:11434" # same host, so `model` can stay null +``` + +and apply `deploy/ollama-over-vpn.conf` on the serving host — Ollama binds +`127.0.0.1` by default and will otherwise refuse. Bind it to the VPN address +rather than `0.0.0.0`: Ollama has no authentication, so anything reaching the +port can run inference and enumerate your models. + +Both endpoints move together because the verifier speaks Ollama's *native* +API and cannot follow the classifier to a cloud provider. Config load refuses +the case where they are on different hosts and `verification.model` is null, +because that combination fails silently. + ## Known Limitations & Open Items - **Leaderboard priors unfilled** — `leaderboards.yaml` ships empty. `python leaderboard.py --check` @@ -596,14 +821,22 @@ ollama pull qwen3.5:latest - **Three models unsettled** — split-half stability at 7 samples: `kimi-k2.7-code-fast` (29×), `kimi-k3` (14×), `glm-5.2-flex` (2.2×). More samples needed before their positioning is trustworthy. -- **Verification observes but never intervenes** — failures are detected and - fed back into `proficiency` automatically (`verification.py` → - `feedback.py`), so the router does learn which models fail on real work. - What is missing is action: nothing retries or escalates on a failed check. - That is deliberate — whether auto-escalation pays should be decided from - the observed failure rate, which is now being collected rather than - guessed at. -- **No auth** — the service holds a billable API key with no - authentication of its own. Loopback binds only. +- **Retry does not reach streaming** — a failed structural check now buys + corrective attempts scaled by tier (`iteration.py`), but only on the + non-streaming path. Once bytes have gone to the client there is nothing to + take back, and buffering to allow correction would cost streaming itself. + `POST /outcome` is the answer for streamed traffic: the client reports + afterwards, so it works identically either way. +- **No auth** — the service holds a billable API key with no authentication + of its own. It binds loopback for that reason; widening the bind needs an + auth layer first. The same applies to an Ollama shared over a VPN, which + also has no auth — bind it to the VPN address, never `0.0.0.0`. +- **Local energy is not accounted for** — the router measures what the cloud + provider bills and reports, but the electricity its own classifier and + verifier burn on your hardware is invisible to it. Since the whole premise + is spending cheap local compute to avoid wasting expensive cloud compute, + that half of the ledger is currently taken on faith. Closing it means + metering local draw and pricing it against a real tariff rather than + assuming local is free. - **Context assembly (RAG)** is out of scope — the classifier sees the full conversation but does not perform document/code retrieval. diff --git a/capabilities.py b/capabilities.py new file mode 100644 index 0000000..d25c750 --- /dev/null +++ b/capabilities.py @@ -0,0 +1,81 @@ +"""Detect request-side capability requirements from an OpenAI-format body. + +Like ``routing.py``, ``scoring.py`` and ``tiering.py``, this module is free of +I/O: the request body comes in as a dict and capabilities come out as a value +object. ``dispatcher.py`` owns the parsing of the HTTP request into that dict. + +Detection is read from the request body, not inferred by a classifier. This +mirrors the existing ``tools_present`` / ``min_tool_proficiency`` philosophy in +``routing.py``: a local classifier cannot reliably identify agentic or visual +work — asked to label unambiguous tool-use prompts it got 1-2 of 6, and vision +is the same class of problem. But the request states both exactly, in the +``tools`` array and in an ``image_url`` part, and reading them is free. When +the capability matters, it is a hard filter (a model that cannot be trusted +with tools that exist is not a candidate), never a weighted preference. + +``has_reasoning_request`` is the one flag that is informational only: it is +recorded so dispatcher can observe it, but never fed to a routing filter. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +# OpenAI response_format.types that require the model to support JSON mode. +JSON_MODE_TYPES = frozenset({"json_object", "json_schema"}) + + +@dataclass(frozen=True) +class RequestCapabilities: + has_images: bool = False + require_json_mode: bool = False + tools_present: bool = False + has_reasoning_request: bool = False + + +def detect_capabilities(body: dict[str, Any]) -> RequestCapabilities: + """Read the capability requirements stated by an OpenAI-format request body. + + ``has_images`` scans ALL messages, not just the last: vision is a property + of the whole conversation, and an image in an earlier turn still has to be + understood by whoever answers the latest one. A message whose ``content`` is + a list contributes True if any part is a dict whose ``type`` is + ``image_url``. String content and non-image parts never trigger it; a raw + base64 string sitting in a text part is not an image_url and must not be + matched. + + ``require_json_mode`` reads ``response_format.type`` against + ``JSON_MODE_TYPES``. ``text`` and an absent ``response_format`` read False. + + ``tools_present`` is truthy exactly when the body carries a ``tools`` array. + + ``has_reasoning_request`` is True if the body carries either a + ``reasoning_effort`` key or a ``reasoning`` key. This flag is recorded for + observation only and is never consumed by a routing filter. + """ + has_images = _any_message_has_images(body.get("messages") or []) + response_format = body.get("response_format") + require_json_mode = ( + isinstance(response_format, dict) + and response_format.get("type") in JSON_MODE_TYPES + ) + return RequestCapabilities( + has_images=has_images, + require_json_mode=require_json_mode, + tools_present=bool(body.get("tools")), + has_reasoning_request="reasoning_effort" in body or "reasoning" in body, + ) + + +def _any_message_has_images(messages: list[Any]) -> bool: + for message in messages: + if not isinstance(message, dict): + continue + content = message.get("content") + if not isinstance(content, list): + continue + for part in content: + if isinstance(part, dict) and part.get("type") == "image_url": + return True + return False diff --git a/config.py b/config.py index fd7cd90..a7c09ce 100644 --- a/config.py +++ b/config.py @@ -4,19 +4,37 @@ Loads and validates config.yaml for the local LLM router. Usage: from config import load_config cfg = load_config("config.yaml") - cfg.weights.cost # etc. + cfg.objective.quality_tolerance # etc. """ from __future__ import annotations from pathlib import Path from typing import Optional +from urllib.parse import urlparse import yaml -from pydantic import BaseModel, field_validator, model_validator +from pydantic import BaseModel, ConfigDict, field_validator, model_validator -class Objective(BaseModel): +class StrictModel(BaseModel): + """Base for every config section: an unknown key is an error. + + Pydantic ignores extra keys by default, which makes a typo or a + misplaced setting silently do nothing while the file still loads and + still looks configured. That is not hypothetical here — `max_input_chars` + was written into the `verification:` block instead of `classifier:`, + where it was accepted, ignored, and had no effect. It happened to carry + the same value as the code default, so nothing visibly broke; editing it + would simply have done nothing. + + Anyone tuning this file needs a wrong key to say so. + """ + + model_config = ConfigDict(extra="forbid") + + +class Objective(StrictModel): """What the router optimizes: quality, bounded by cost. Replaced a three-way weighted blend. See config.yaml for why — briefly, @@ -25,7 +43,7 @@ class Objective(BaseModel): """ quality_tolerance: float = 0.10 - assumed_cache_rate: float = 0.84 + assumed_cache_rate: float = 0.917 assumed_completion_tokens: int = 500 max_energy_per_request: Optional[float] = None plan_kwh_per_period: Optional[float] = None @@ -47,10 +65,44 @@ class Objective(BaseModel): return v -class ContextConfig(BaseModel): +class ContextOverride(StrictModel): + """Per-model context handling, for a row whose real limits are known. + + Typed rather than a bare ``dict`` so a typo INSIDE an override is an error + too. That is the whole point of StrictModel, and it was not true here: the + block validated, nothing read it, and config.yaml shipped a worked example + for it -- so anyone who followed that example got silence. + + Both fields are optional; whichever is absent falls back to the global. + """ + + safety_factor: Optional[float] = None + output_reserve_tokens: Optional[int] = None + + @field_validator("safety_factor") + @classmethod + def factor_in_range(cls, v: Optional[float]) -> Optional[float]: + if v is not None and not (0.0 < v <= 1.0): + raise ValueError( + "context.per_model_overrides[...].safety_factor must be in (0, 1]" + ) + return v + + @field_validator("output_reserve_tokens") + @classmethod + def reserve_not_negative(cls, v: Optional[int]) -> Optional[int]: + if v is not None and v < 0: + raise ValueError( + "context.per_model_overrides[...].output_reserve_tokens must be >= 0" + ) + return v + + +class ContextConfig(StrictModel): safety_factor: float default_output_reserve_tokens: int - per_model_overrides: dict = {} + # Read by poller.ModelRow.effective_context_window. + per_model_overrides: dict[str, ContextOverride] = {} @field_validator("safety_factor") @classmethod @@ -60,7 +112,7 @@ class ContextConfig(BaseModel): return v -class ProficiencyConfig(BaseModel): +class ProficiencyConfig(StrictModel): self_eval_min_samples: int leaderboard_weight: float self_eval_weight: float @@ -77,7 +129,7 @@ class ProficiencyConfig(BaseModel): return self -class TieringConfig(BaseModel): +class TieringConfig(StrictModel): cheap_completion_max: float tier1_context_max: float = float("inf") model_tiers: dict[str, int] @@ -108,9 +160,20 @@ class TieringConfig(BaseModel): return v -class RoutingConfig(BaseModel): +class RoutingConfig(StrictModel): allowed_access_levels: list[str] default_latency_tolerance: str + # Applied only when the REQUEST carries tool definitions. None disables it. + min_tool_proficiency: Optional[float] = 0.5 + tool_use_category: str = "tool_use_agentic" + + # Whether a request carrying image parts is hard-restricted to vision-capable + # models. A wrong guess here is a guaranteed 400, so this gates by default + # and routing fails closed when the catalog flag is unknown. + require_vision: bool = True + # Whether a response_format requiring json_object/json_schema is hard-restricted + # to JSON-mode-capable models. Same guaranteed-failure argument. + require_json_mode: bool = True @field_validator("allowed_access_levels") @classmethod @@ -136,15 +199,54 @@ class RoutingConfig(BaseModel): return v -class VerificationConfig(BaseModel): +class VerificationConfig(StrictModel): local_llm_enabled: bool = True min_completion_tokens: int = 600 timeout_seconds: int = 60 max_output_tokens: int = 1024 outcome_attribution_window_seconds: int = 120 + # The local checker's OWN endpoint, no longer derived from the + # classifier's. It speaks Ollama's NATIVE API (/api/chat, think=False), + # which no cloud provider offers, so it must keep pointing at an Ollama + # instance even when classification has been moved off this machine. + base_url: str = "http://localhost:11434" + # None means "whatever the classifier uses", which is correct only while + # both run on the same local Ollama. Set it explicitly once they diverge. + model: Optional[str] = None -class EscalationConfig(BaseModel): +class LocalVisionConfig(StrictModel): + enabled: bool = False + base_url: str = "http://localhost:11434/v1" # OpenAI-compatible (classifier shape) + api_key_env: Optional[str] = None + model: str = "qwen3-vl:4b" + timeout_seconds: int = 60 + max_images: int = 4 + max_image_bytes: int = 9 * 1024 * 1024 # 9 MiB, Ollama default cap + + @field_validator("timeout_seconds") + @classmethod + def timeout_positive(cls, v: int) -> int: + if v <= 0: + raise ValueError("local_vision.timeout_seconds must be > 0") + return v + + @field_validator("max_images") + @classmethod + def images_positive(cls, v: int) -> int: + if v <= 0: + raise ValueError("local_vision.max_images must be > 0") + return v + + @field_validator("max_image_bytes") + @classmethod + def image_bytes_positive(cls, v: int) -> int: + if v <= 0: + raise ValueError("local_vision.max_image_bytes must be > 0") + return v + + +class EscalationConfig(StrictModel): enabled: bool max_tier: int min_confidence_before_bump: float @@ -152,7 +254,7 @@ class EscalationConfig(BaseModel): preemptive_on_low_confidence: bool = False -class IterationConfig(BaseModel): +class IterationConfig(StrictModel): """A tier's budget for corrective attempts after a verification failure.""" enabled: bool = True @@ -173,41 +275,58 @@ class IterationConfig(BaseModel): return v -class FreshnessConfig(BaseModel): +class FreshnessConfig(StrictModel): stale_after_days: int exclude_stale: bool exclude_deprecated: bool -class DatabaseConfig(BaseModel): +class DatabaseConfig(StrictModel): path: str -class ClassifierConfig(BaseModel): +class ClassifierConfig(StrictModel): provider: str base_url: str + # Env var holding the API key, for a classifier served by a provider that + # actually checks one. None means unauthenticated, which is the local + # Ollama case — it ignores the key entirely but the SDK requires one. + api_key_env: Optional[str] = None model: str + # Ceiling on the text handed to the classifier. 0 disables clamping. + max_input_chars: int = 8000 timeout_seconds: int temperature: float = 0.0 max_output_tokens: int = 1024 - outcome_attribution_window_seconds: int = 120 fallback_tier: int = 2 fallback_category: str = "general_chat" response_format: str system_prompt: str -class DispatchProvider(BaseModel): +class DispatchProvider(StrictModel): base_url: str api_key_env: str -class LoggingConfig(BaseModel): +class LoggingConfig(StrictModel): + # log_path is gone. Nothing ever wrote a file: the dispatcher logs to + # stderr and systemd captures that to the journal, so the setting named a + # destination that did not exist. log_energy_observations: bool - log_path: str + # LLM_ROUTER_LOG_LEVEL overrides this at runtime — see logs.resolve_level. + level: str = "info" + + @field_validator("level") + @classmethod + def level_known(cls, v: str) -> str: + known = ("debug", "info", "warning", "error") + if v.strip().lower() not in known: + raise ValueError(f"logging.level must be one of {known}, got {v!r}") + return v.strip().lower() -class RouterConfig(BaseModel): +class RouterConfig(StrictModel): objective: Objective context: ContextConfig tiers: dict[int, str] @@ -215,6 +334,7 @@ class RouterConfig(BaseModel): proficiency: ProficiencyConfig routing: RoutingConfig verification: VerificationConfig = VerificationConfig() + local_vision: LocalVisionConfig = LocalVisionConfig() escalation: EscalationConfig iteration: IterationConfig = IterationConfig() freshness: FreshnessConfig @@ -223,6 +343,57 @@ class RouterConfig(BaseModel): dispatch_providers: dict[str, DispatchProvider] logging: LoggingConfig + @model_validator(mode="after") + def tool_use_category_is_a_real_category(self) -> "RouterConfig": + """The tool filter joins on a category name; a typo would disable it. + + A name that matches nothing produces NULL for every row, and NULL + means "unproven, do not disqualify" — so the filter would silently + pass everything. Failing at load beats a guard that quietly stops + guarding. + """ + if self.routing.min_tool_proficiency is None: + return self + if self.routing.tool_use_category not in self.proficiency.categories: + raise ValueError( + f"routing.tool_use_category " + f"({self.routing.tool_use_category!r}) is not in " + f"proficiency.categories — the tool-competence filter would " + f"join against nothing and silently pass every model." + ) + return self + + @model_validator(mode="after") + def verifier_model_is_stated_once_the_hosts_differ(self) -> "RouterConfig": + """A remote classifier must not lend its model name to the verifier. + + ``verification.model`` falling back to ``classifier.model`` is correct + only while both point at the same Ollama. Once classification moves + off-host the fallback names a model the local Ollama has never heard + of, and the failure is SILENT: the verifier 404s, catches it, logs + "local verification unavailable" and records no sample. Verification + would appear to be on while producing nothing. + + Caught by pointing the classifier at NeuralWatt and watching the + verifier POST ``deepseek-v4-flash`` to localhost:11434. Failing at + config load instead means the misconfiguration is impossible rather + than merely documented. + """ + if not self.verification.local_llm_enabled or self.verification.model: + return self + classifier_host = urlparse(self.classifier.base_url).hostname + verifier_host = urlparse(self.verification.base_url).hostname + if classifier_host != verifier_host: + raise ValueError( + "verification.model must be set explicitly when the classifier " + f"runs on a different host ({classifier_host} vs " + f"{verifier_host}). It would otherwise fall back to " + f"classifier.model ({self.classifier.model!r}), which the local " + "Ollama does not serve — and the verifier fails silently. " + "Set verification.model, or verification.local_llm_enabled: false." + ) + return self + def load_config(path: str | Path = "config.yaml") -> RouterConfig: path = Path(path) @@ -232,14 +403,32 @@ def load_config(path: str | Path = "config.yaml") -> RouterConfig: return RouterConfig(**raw) +def summary_lines(cfg: RouterConfig) -> list[str]: + """What ``python config.py`` prints. + + A function rather than inline prints so a test can pin the attribute names. + The previous version read ``cfg.weights``, which had been replaced by + ``cfg.objective`` -- so the setup step documented in both README.md and + CLAUDE.md said "Config loaded OK" and then died with AttributeError on a + config that had in fact loaded perfectly. + """ + return [ + "Config loaded OK", + f" objective: quality_tolerance={cfg.objective.quality_tolerance}, " + f"max_energy_per_request={cfg.objective.max_energy_per_request}, " + f"plan_kwh_per_period={cfg.objective.plan_kwh_per_period}", + f" categories: {cfg.proficiency.categories}", + f" classifier: {cfg.classifier.model} @ {cfg.classifier.base_url}", + f" verifier: {cfg.verification.model or cfg.classifier.model} " + f"@ {cfg.verification.base_url}" + + ("" if cfg.verification.local_llm_enabled else " (disabled)"), + f" tool filter: min_tool_proficiency={cfg.routing.min_tool_proficiency}", + f" dispatch providers: {list(cfg.dispatch_providers)}", + ] + + if __name__ == "__main__": import sys cfg_path = sys.argv[1] if len(sys.argv) > 1 else "config.yaml" - cfg = load_config(cfg_path) - print("Config loaded OK") - print(f" weights: cost={cfg.weights.cost} eco={cfg.weights.eco} " - f"proficiency={cfg.weights.proficiency}") - print(f" categories: {cfg.proficiency.categories}") - print(f" classifier: {cfg.classifier.model} @ {cfg.classifier.base_url}") - print(f" dispatch providers: {list(cfg.dispatch_providers.keys())}") + print("\n".join(summary_lines(load_config(cfg_path)))) diff --git a/config.yaml b/config.yaml index 9d0146d..70b016a 100644 --- a/config.yaml +++ b/config.yaml @@ -28,10 +28,18 @@ objective: # to the actual request agree with the live measurement, cost nothing, and # need no sweep. - # Share of prompt tokens served from the provider's prefix cache. Measured - # from real traffic: 1.9M of 2.2M prompt tokens, 84%. Agent clients resend - # the whole conversation each turn, so most of it is a cache hit. - assumed_cache_rate: 0.84 + # Share of prompt tokens served from the provider's prefix cache. Agent + # clients resend the whole conversation each turn, so most of it is a hit. + # + # Measured token-weighted across 50 sessions and 40.7M tokens (2026-08-23): + # 91.7% overall, 92.6% on the sessions above 400k tokens, which are the ones + # that carry the cost. The previous 0.84 came from 2.2M tokens of much + # earlier traffic. + # + # Raising it changed NO winner in any of the nine categories at 60k context + # — checked before editing. It is here because the number should be true, + # not because the routing needed it. + assumed_cache_rate: 0.917 # Completion length assumed when pricing a request. Real sessions here median # around 200-400 completion tokens against enormous prompts. @@ -55,10 +63,22 @@ objective: context: safety_factor: 0.75 # fraction of advertised context treated as usable default_output_reserve_tokens: 4096 - # If a model has a known-good override, list it here; otherwise the global - # safety_factor above applies. + # Per-model exceptions to the two settings above, for a row whose real + # limits you have measured. The global factor has to hold for the whole + # catalog, so it is deliberately pessimistic; a model you have actually + # pushed to its limit deserves its own number. + # + # Both keys are optional and each falls back to the global on its own. + # An override of 0 reserve tokens means zero, not "unset". + # + # per_model_overrides: + # qwen3.6-35b: + # safety_factor: 0.85 + # output_reserve_tokens: 8192 + # + # Takes effect on the next `python poller.py` -- effective_context_window is + # computed at poll time, not per request. per_model_overrides: {} - # example: per_model_overrides: { "qwen3.6-35b": { safety_factor: 0.85 } } tiers: # Maps a tier number to a human label, purely for logging/dashboards. @@ -150,15 +170,91 @@ routing: # interactive, so a request has to opt in via latency_tolerance. default_latency_tolerance: interactive # 'interactive' | 'batch' - # There is deliberately no flex discount knob. Cost scoring reads the mean - # cost actually billed for the reference workload (see seed_energy.py), and - # a flex row's measured cost already is its flex cost. + # Minimum tool_use_agentic proficiency required of a model when the REQUEST + # carries tool definitions. A filter, not a weight, because it is a + # capability requirement rather than a preference. + # + # It does NOT ask whether the task is agentic — it asks whether the model + # can be trusted with tools that are on the table. The measured failure is + # the second one: deepseek-v4-flash scores 0.33 here, and the recorded case + # is a NON-agentic prompt ("it is 1:20pm, my meeting is at 3pm, how many + # minutes?") where it called two tools instead of subtracting. A model that + # over-reaches is a hazard wherever tools exist, not only where a classifier + # would say "agentic" — which it cannot do anyway: asked to identify six + # unambiguous tool-use prompts, the local models managed 2/6 and 1/6. + # Whether tools are present is stated in the request body. Read it. + # + # 0.5 sits in the empty band between the only two values the catalog + # currently holds (0.33 and 1.00), so it is not fitted to either. A model + # with NO measured tool score is unproven rather than proven bad, and is not + # dropped. + # + # CURRENTLY DISABLED (null) pending experimentation. The trade being + # measured: opencode sends `tools` on essentially every request, so with the + # filter on, deepseek-v4-flash is excluded from ordinary agent traffic and + # the ~7x cost advantage on coding routes goes unused. With it off, that + # advantage applies — and a model measured at 0.33 on tool use handles + # requests where tools are on the table. + # + # What would settle it is outcome data, not another benchmark: run with it + # off, let POST /outcome report real pass/fail, and compare + # tool_use_agentic proficiency for deepseek before and after. That is the + # one signal here that knows whether the work actually worked. + min_tool_proficiency: null + tool_use_category: tool_use_agentic + + # Request-side capability gates. These read the request body (image parts, + # response_format) and hard-restrict to models whose catalog row declares the + # capability. Unlike min_tool_proficiency these are NOT proficiency gates — + # a wrong guess is a guaranteed provider 400, so they gate by default and + # fail closed when the catalog flag is unknown. + require_vision: true + require_json_mode: true + + # There is deliberately no flex discount knob. Whether a flex row is usable + # at all is a hard filter above (latency_tolerance), not a price adjustment + # -- being held server-side during peak is a latency property, and the + # catalog advertises flex and standard at the same token price anyway. + +local_vision: + # Local Ollama vision fallback. Used ONLY when a request carries image + # parts and routing finds NO cloud model that supports_vision — the cloud + # catalog currently excludes the cost leader (deepseek) on vision, so a + # fallback is what keeps image requests working instead of 422ing. + # It speaks the OpenAI-compatible /v1 surface, so this is an Ollama endpoint + # and the model must be pulled (`ollama pull qwen3-vl:4b`) on that host. + enabled: true + base_url: "http://localhost:11434/v1" + api_key_env: null + model: "qwen3-vl:4b" + timeout_seconds: 60 + max_images: 4 + max_image_bytes: 9437184 verification: - # Structural checks (parse the code, never run it) are free and always on. + # Structural checks (parse the code, never run it) are free, pure Python and + # always on — they need no model and run anywhere, down to an RPi. # This section governs the LOCAL LLM check, which is not free. + # + # Set local_llm_enabled: false on a host with no usable local inference. + # Structural checking and POST /outcome both keep working; only the + # refusal/incoherence class of failure stops being caught. local_llm_enabled: true + # This check speaks Ollama's NATIVE API (/api/chat with think=False), which + # no cloud provider offers, so it is configured SEPARATELY from the + # classifier rather than derived from it. Deriving it meant that pointing + # classification anywhere else sent these requests to /api/chat. + # + # Point it at the SAME Ollama as the classifier when that is across a VPN; + # leave the model null in that case, since the hosts then match. + base_url: "http://localhost:11434" + # null means "whatever classifier.model is", correct only while both run on + # the same local Ollama — config load REFUSES the null once the hosts differ, + # because the fallback would name a model this Ollama has never heard of and + # the verifier would fail silently. + model: "mistral-nemo:12b" + # Only check answers this large. Measured on real traffic: a local check # costs ~15% of a median 193-token answer, so it would only pay if such # answers failed more than ~15% of the time. At 1,500 completion tokens the @@ -191,10 +287,28 @@ database: path: "router.db" classifier: - # Local Ollama instance doing task classification / context sizing. + # The local LLM that classifies each task before a cloud model answers it. + # "Local" means your hardware, not necessarily this machine — the box with + # the GPU is usually not the laptop you are typing on. It speaks plain + # OpenAI-compatible chat completions, so point it wherever Ollama lives: + # + # same machine base_url: http://localhost:11434/v1 + # over a VPN base_url: http://:11434/v1 + # (serving host needs deploy/ollama-over-vpn.conf; Ollama + # binds loopback-only by default and will refuse) + # + # Any OpenAI-compatible endpoint works, so a cloud model can classify too — + # set api_key_env for one that checks a key. Worth knowing before assuming + # local is the cheap option: on five prompts NeuralWatt's deepseek-v4-flash + # classified in 1.02s mean against qwen3.5's 11.58s on an RTX 6000, agreed + # with the label 5/5 against 2/4, and cost $0.093 per thousand calls. Local + # inference is not free, it is unbilled. provider: "ollama" base_url: "http://localhost:11434/v1" - model: "qwen3.5:latest" # must match a model `ollama list` reports + # Unset means unauthenticated, which is the Ollama case. Name the env var + # holding the key when the endpoint actually checks one. + api_key_env: null + model: "mistral-nemo:12b" # must match a model `ollama list` reports # A cold Ollama took 43s to answer the first classification, which blew the # old 30s ceiling and returned 503 to the client. Warm it is ~5s. The # ceiling is for a cold model load, not the steady state. @@ -205,21 +319,36 @@ classifier: # prompt is untraceable. temperature: 0 # Hard cap on the classifier's generation, to bound a failure mode that - # cascades: qwen3.5 is a reasoning model and will sometimes emit a long - # chain of thought, blowing past timeout_seconds — and Ollama keeps - # generating after the client gives up AND serializes per model, so one - # runaway request queues every later request behind it. + # cascades on REASONING models: they emit a long chain of thought, blow past + # timeout_seconds, and Ollama keeps generating after the client gives up AND + # serializes per model, so one runaway request queues every later one behind + # it. Measured on qwen3.5, which failed this way on 4 of 19 calls — each one + # a 15s wait ending in a silent fallback. # - # Must leave room for the thinking trace, not just the ~45-token answer. At - # 256 the trace consumed the whole budget on some prompts and the model was - # truncated before emitting any JSON at all, which read as an empty - # response. 1024 bounds the runaway while leaving the answer reachable. + # The current default (mistral-nemo) does not reason, so this does not bind + # for it. Kept anyway: it costs nothing when unused and is the only thing + # standing between a swapped-in reasoning model and that cascade. If you do + # swap one in, note 256 was too tight — the trace consumed the whole budget + # and the model was truncated before emitting any JSON. max_output_tokens: 1024 # Where routing lands when the classifier times out, errors, or returns # something unparseable. A local model being slow should degrade routing, # not refuse the request — the caller is a coding agent that would rather # have a mid-tier answer than a 502. + # Ceiling on the text handed to the classifier (head + tail, middle elided). + # It decides a category and a tier; it does not need the document, and + # feeding it one is harmful rather than merely wasteful. Measured on a ~20k + # token prompt: qwen3.5 spent 28.7s and returned empty (budget consumed by + # its reasoning trace), mistral-nemo spent 41.8s echoing the input back + # inside its JSON. Both land on source: "fallback" — the same answer an + # instant failure gives, after 30-40s of local inference. + # + # Nothing is lost: chat_completions measures the real conversation with + # estimate_prompt_tokens and takes the larger value, so required_context + # never depends on what the classifier saw. 0 disables clamping. + max_input_chars: 8000 + fallback_tier: 2 fallback_category: general_chat response_format: "json" # ask Ollama to constrain output to valid JSON @@ -247,5 +376,26 @@ dispatch_providers: api_key_env: "NEURALWATT_API_KEY" logging: + # Whether to write a row to energy_observations for every completion. Off + # means no cost accounting, no reference sweep and no /outcome attribution, + # so leave it on unless you are debugging. + # + # There is no log_path. Nothing ever wrote a file -- the dispatcher prints to + # stderr and the systemd unit hands that to the journal (`journalctl --user + # -u llm-router -f`), so the setting named a destination that did not exist. log_energy_observations: true - log_path: "router.log" + + # debug | info | warning | error. + # + # info gives one line per request: what it was classified as, which model won, + # what it cost, how long each stage took. debug adds why -- every candidate + # that was dropped and by which filter, the ranking with scores, the + # classifier's raw reply. It logs no conversation text at any level; prompts + # here run 60k-150k tokens and the journal is on disk. + # + # LLM_ROUTER_LOG_LEVEL overrides this, so a running service can be turned up + # without editing a tracked file: + # + # systemctl --user edit llm-router # Environment="LLM_ROUTER_LOG_LEVEL=debug" + # systemctl --user restart llm-router + level: info diff --git a/deploy/README.md b/deploy/README.md index f7e5a5f..8bea6d8 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -22,11 +22,18 @@ environment, so the API key has to come from a file. ```bash # 1. The key. User units don't see your shell env, so .env is required. -cd ~/Sources/6krrt +cd /path/to/this/repo echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env -# 2. Install and start -cp deploy/llm-router*.{service,timer} ~/.config/systemd/user/ +# 2. Install, pointing the units at wherever you actually cloned this. +# The shipped units say %h/llm-router; %h is systemd's expansion for your +# home directory, so only the part after it needs changing. Getting this +# wrong fails at start with status=200/CHDIR rather than anything obvious. +REPO=$(pwd) +mkdir -p ~/.config/systemd/user +for u in deploy/llm-router*.{service,timer}; do + sed "s|%h/llm-router|${REPO}|g" "$u" > ~/.config/systemd/user/"$(basename "$u")" +done systemctl --user daemon-reload systemctl --user enable --now llm-router.service llm-router-poller.timer llm-router-seed.timer @@ -43,9 +50,11 @@ systemctl --user list-timers 'llm-router*' ```bash systemctl --user status llm-router.service -journalctl --user -u llm-router.service -f # request log -journalctl --user -u llm-router-poller.service # catalog refreshes -systemctl --user restart llm-router.service # after editing config.yaml +journalctl --user -u 'llm-router*' -f # everything, live (quote the glob) +journalctl --user -u llm-router -f -o cat # the request log, message only +journalctl --user -u llm-router -p warning # fallbacks, retries, refusals +journalctl --user -u llm-router-poller.service # catalog refreshes +systemctl --user restart llm-router.service # after editing config.yaml systemctl --user start llm-router-poller.service # force a refresh now ``` @@ -53,6 +62,41 @@ systemctl --user start llm-router-poller.service # force a refresh now restart. The catalog is read per-request, so a poller run takes effect immediately. +### Turning up the logs + +`logging.level` in config.yaml is the documented setting, but flipping it means +editing a tracked file. For a running service use a drop-in instead: + +```bash +systemctl --user edit llm-router # Environment="LLM_ROUTER_LOG_LEVEL=debug" +systemctl --user restart llm-router +``` + +`info` gives one `route` and one `dispatch` line per request — category, tier, +model chosen, cost, latency. `debug` adds every candidate that was dropped and +by which filter, plus the ranking with scores. No conversation text is logged +at any level. + +Every line carries a trace id, and the `dispatch` line carries the provider's +completion id and the session fingerprint, both of which are columns in +`energy_observations`: + +```bash +journalctl --user -u llm-router --grep ' id=r9116d9' # one request, all stages +journalctl --user -u llm-router --grep 'chatcmpl-abc123' # from a DB row back to its decision +``` + +Severity filtering works because the service prefixes its lines with journald +priorities when systemd owns its stderr (`SyslogLevelPrefix` is on by default). +A foreground `uvicorn` prints them clean, so the same binary is readable either +way. + +**The oneshot units buffer.** `poller.py` and `seed_energy.py` print progress +with plain `print()`, and Python block-buffers stdout when it is not a +terminal, so their output arrives in one dump at exit rather than +progressively. Add `Environment="PYTHONUNBUFFERED=1"` to those units if you +want to watch a sweep as it runs. + ## A note on the bind address `--host 127.0.0.1` is deliberate. The service holds a billable API key and @@ -61,10 +105,55 @@ your allowance. `ProtectHome=read-only` plus a `ReadWritePaths` exception for the repo limits the blast radius on the filesystem, but nothing limits spend. Putting this on a LAN address needs an auth layer first. +## Using an Ollama on another machine + +The local LLM does the classifying; it does not have to be on the machine you +are typing on, and usually the GPU isn't. Most developers already have +WireGuard or a VPN back to a home lab, so the normal shape is router and +editor on the laptop, Ollama on the workstation. + +On the **serving** host (the one with the GPU): + +```bash +sudo mkdir -p /etc/systemd/system/ollama.service.d +sudo cp deploy/ollama-over-vpn.conf \ + /etc/systemd/system/ollama.service.d/override.conf +# set OLLAMA_HOST to that host's own VPN address — `ip -4 -o addr show` +sudo nano /etc/systemd/system/ollama.service.d/override.conf +sudo systemctl daemon-reload && sudo systemctl restart ollama +``` + +On the **client** host, in `config.yaml`: + +```yaml +classifier: + base_url: "http://:11434/v1" +verification: + base_url: "http://:11434" # same host, so `model` can stay null +``` + +Both must move together. `verification` speaks Ollama's native `/api/chat` +and used to derive its URL from the classifier's; it no longer does, so +pointing only the classifier across the tunnel leaves the verifier talking to +a `localhost` Ollama that may not exist. Config load refuses the combination +where `verification.model` is null and the two hosts differ, because that +failure is otherwise silent — the verifier 404s, catches it, and records no +sample while appearing to be enabled. + +Bind Ollama to the **VPN address, not `0.0.0.0`**. It has no authentication of +any kind: anything that reaches the port can run inference, enumerate your +models and pull new ones. Same reasoning as the dispatcher's loopback bind +above. + +A cloud endpoint works too — set `classifier.api_key_env` to the env var +holding its key. On a five-prompt comparison NeuralWatt's `deepseek-v4-flash` +classified in 1.02s mean against `qwen3.5`'s 11.58s on an RTX 6000. Local +inference is not free, it is unbilled. + ## Pointing opencode at it The repo-local `opencode.json` sets this up already, so running `opencode` -from inside `~/Sources/6krrt` uses the router by default. To use it from +from inside a clone of this repo uses the router by default. To use it from anywhere, merge the `provider.llm-router` block into `~/.config/opencode/opencode.json` and set `"model": "llm-router/auto"`. diff --git a/deploy/llm-router-poller.service b/deploy/llm-router-poller.service index b8a0887..39dd4f7 100644 --- a/deploy/llm-router-poller.service +++ b/deploy/llm-router-poller.service @@ -1,21 +1,21 @@ [Unit] Description=Refresh the NeuralWatt model catalog and re-resolve tiers -Documentation=file:%h/Sources/6krrt/CLAUDE.md +Documentation=file:%h/llm-router/CLAUDE.md After=network-online.target Wants=network-online.target [Service] Type=oneshot -WorkingDirectory=%h/Sources/6krrt -EnvironmentFile=%h/Sources/6krrt/.env +WorkingDirectory=%h/llm-router +EnvironmentFile=%h/llm-router/.env # poller.py refreshes the catalog; tier.py re-resolves tiers from it. Tiers # are derived from cost and reasoning fields the poll may have changed, so # they always run as a pair. -ExecStart=%h/Sources/6krrt/.venv/bin/python poller.py -ExecStart=%h/Sources/6krrt/.venv/bin/python tier.py +ExecStart=%h/llm-router/.venv/bin/python poller.py +ExecStart=%h/llm-router/.venv/bin/python tier.py NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=read-only -ReadWritePaths=%h/Sources/6krrt +ReadWritePaths=%h/llm-router diff --git a/deploy/llm-router-seed.service b/deploy/llm-router-seed.service index a8058db..db7e1a7 100644 --- a/deploy/llm-router-seed.service +++ b/deploy/llm-router-seed.service @@ -1,19 +1,19 @@ [Unit] Description=Sample the reference workload to keep cost/eco scoring current -Documentation=file:%h/Sources/6krrt/CLAUDE.md +Documentation=file:%h/llm-router/CLAUDE.md After=network-online.target Wants=network-online.target [Service] Type=oneshot -WorkingDirectory=%h/Sources/6krrt -EnvironmentFile=%h/Sources/6krrt/.env +WorkingDirectory=%h/llm-router +EnvironmentFile=%h/llm-router/.env # Fewer samples per run than a manual sweep, because the point is coverage # across TIME rather than depth at one moment — see the timer. -ExecStart=%h/Sources/6krrt/.venv/bin/python seed_energy.py --samples 3 +ExecStart=%h/llm-router/.venv/bin/python seed_energy.py --samples 3 NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ProtectHome=read-only -ReadWritePaths=%h/Sources/6krrt +ReadWritePaths=%h/llm-router diff --git a/deploy/llm-router.service b/deploy/llm-router.service index 3e27d01..548dad2 100644 --- a/deploy/llm-router.service +++ b/deploy/llm-router.service @@ -1,6 +1,6 @@ [Unit] Description=Local LLM model router (FastAPI dispatcher) -Documentation=file:%h/Sources/6krrt/CLAUDE.md +Documentation=file:%h/llm-router/CLAUDE.md # The classifier talks to Ollama on localhost and the dispatcher talks to # NeuralWatt, so both loopback and real network need to be up. Ollama is a # system service and cannot be ordered against from a user unit, so a failed @@ -12,11 +12,11 @@ Wants=network-online.target Type=exec # config.yaml, router.db and router.log are all referenced as relative paths, # so this has to be the repo root. -WorkingDirectory=%h/Sources/6krrt +WorkingDirectory=%h/llm-router # Holds NEURALWATT_API_KEY. Create it with: # echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env -EnvironmentFile=%h/Sources/6krrt/.env -ExecStart=%h/Sources/6krrt/.venv/bin/uvicorn dispatcher:app --host 127.0.0.1 --port 8080 +EnvironmentFile=%h/llm-router/.env +ExecStart=%h/llm-router/.venv/bin/uvicorn dispatcher:app --host 127.0.0.1 --port 8080 Restart=on-failure RestartSec=5s @@ -28,7 +28,7 @@ PrivateTmp=true ProtectSystem=strict ProtectHome=read-only # ...except the repo, which needs to be writable for router.db and router.log. -ReadWritePaths=%h/Sources/6krrt +ReadWritePaths=%h/llm-router [Install] WantedBy=default.target diff --git a/deploy/ollama-over-vpn.conf b/deploy/ollama-over-vpn.conf new file mode 100644 index 0000000..93491e4 --- /dev/null +++ b/deploy/ollama-over-vpn.conf @@ -0,0 +1,41 @@ +# Serve this host's Ollama to the other machines you work from. +# +# The premise of this router is that a local LLM classifies your task before a +# cloud model answers it. "Local" means your hardware, not necessarily the +# machine you are typing on — the box with the GPU usually is not the laptop. +# Most developers already have WireGuard or a VPN back to a home lab, so the +# normal shape is: router and editor on the laptop, Ollama on the workstation, +# classifier.base_url pointing across the tunnel. +# +# Ollama listens on 127.0.0.1:11434 by default, so that fails with +# connection-refused until this drop-in is applied on the SERVING host. +# +# Install (needs root — ollama.service is a system unit): +# sudo mkdir -p /etc/systemd/system/ollama.service.d +# sudo cp deploy/ollama-over-vpn.conf \ +# /etc/systemd/system/ollama.service.d/override.conf +# sudo nano /etc/systemd/system/ollama.service.d/override.conf # set the address +# sudo systemctl daemon-reload && sudo systemctl restart ollama +# +# Then on the client machine, in config.yaml: +# classifier.base_url: http://:11434/v1 +# verification.base_url: http://:11434 +# +# Bind to the VPN address, NOT 0.0.0.0. Ollama has no authentication of any +# kind: anything that can reach the port can run inference, enumerate your +# models and pull new ones. 0.0.0.0 also publishes it on whatever cafe or +# hotel wifi the laptop is sitting on. Same reasoning that keeps the +# dispatcher itself on loopback. +# +# Replace with this host's own VPN address — 10.x.x.x for WireGuard, +# 100.x.x.x for Tailscale. `ip -4 -o addr show` will tell you. + +[Service] +Environment="OLLAMA_HOST=10.0.0.1:11434" + +# Keep the model resident between requests. Without this, the first +# classification after an idle gap pays a cold load — measured at 43s, which +# exceeds even the 120s ceiling once the SDK's retries are counted. Serving a +# laptop makes this matter more, not less: requests arrive in bursts separated +# by long gaps, which is exactly the pattern that keeps evicting the model. +Environment="OLLAMA_KEEP_ALIVE=24h" diff --git a/dispatcher.py b/dispatcher.py index 5a85152..4fd0fb1 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -39,20 +39,29 @@ import os import hashlib import re import sqlite3 -import sys +import time from datetime import datetime, timezone from statistics import median from typing import Any, Literal, Optional import requests +import secrets from dotenv import load_dotenv from fastapi import BackgroundTasks, FastAPI, HTTPException -from fastapi.responses import JSONResponse, StreamingResponse +from fastapi.responses import JSONResponse, Response, StreamingResponse from openai import OpenAI, OpenAIError from pydantic import BaseModel, Field +import logs +from capabilities import detect_capabilities from config import RouterConfig, load_config -from routing import BATCH, INTERACTIVE, rank_candidates, select_candidates +from routing import ( + BATCH, + INTERACTIVE, + rank_candidates, + rejection_reason, + select_candidates, +) from iteration import attempts_allowed, plan_retry from verification import ( LOCAL_VERIFY_SYSTEM, @@ -122,6 +131,7 @@ BTU_PER_KWH = 3412.14 app = FastAPI(title="Local LLM Model Router", version="0.1.0") cfg: RouterConfig = load_config("config.yaml") +logs.configure(cfg.logging.level) # --- request/response models --------------------------------------------- @@ -138,6 +148,27 @@ class TaskRequest(BaseModel): "Defaults to routing.default_latency_tolerance." ), ) + tools_present: bool = Field( + False, + description=( + "Whether the caller has tool definitions on the table. Requires a " + "model with measured tool-use competence (routing." + "min_tool_proficiency). Set automatically from the `tools` array " + "on /v1/chat/completions; exposed here for testing and for " + "clients that route before they call." + ), + ) + has_images: bool = Field( + False, + description="Whether the request carries image parts; requires a " + "vision-capable model (routing.require_vision). Set automatically from " + "the request body on /v1/chat/completions.", + ) + require_json_mode: bool = Field( + False, + description="Whether the request's response_format needs JSON mode; " + "requires a JSON-mode-capable model (routing.require_json_mode).", + ) # Overrides, mostly for testing the router without the classifier in the loop. task_category: Optional[str] = None task_tier: Optional[int] = Field(None, ge=1, le=3) @@ -193,6 +224,15 @@ class DispatchResponse(BaseModel): # --- infrastructure ------------------------------------------------------- +def _ms(started: float) -> int: + """Elapsed milliseconds since a perf_counter mark, for the latency fields. + + Integers because nobody debugging a route cares about microseconds, and a + log line reads better without six decimal places on every timing. + """ + return int((time.perf_counter() - started) * 1000) + + def _db() -> sqlite3.Connection: conn = sqlite3.connect(cfg.database.path) conn.row_factory = sqlite3.Row @@ -200,16 +240,40 @@ def _db() -> sqlite3.Connection: def _classifier_client() -> OpenAI: - # Ollama ignores the key but the SDK requires one to be set. - # - # max_retries=0 matters: the SDK retries twice by default, so - # classifier.timeout_seconds silently becomes a 3x wall-clock bound and a - # cold model load can block a caller for six minutes on a 120s setting — - # observed as a request that hung past 250s and logged nothing. Retrying a - # local model that is busy loading only queues more work behind it, so the - # configured timeout should be the real one. + """The classification endpoint, which need not be local. + + Classification is the one blocking LLM call on the request path, and it + assumes the least about its host: it is a short prompt with a short JSON + answer. That makes it the piece worth moving when the machine running this + router has no GPU worth the name — an RPi or a laptop can route perfectly + well while something else does the classifying. + + ``api_key_env`` unset means unauthenticated, which is the local Ollama + case: it ignores the key but the SDK requires one to be set. + + max_retries=0 matters: the SDK retries twice by default, so + classifier.timeout_seconds silently becomes a 3x wall-clock bound and a + cold model load can block a caller for six minutes on a 120s setting — + observed as a request that hung past 250s and logged nothing. Retrying a + local model that is busy loading only queues more work behind it, so the + configured timeout should be the real one. + """ + key_env = cfg.classifier.api_key_env + if key_env: + api_key = os.environ.get(key_env) + if not api_key: + # Raised rather than passed through as a placeholder: a remote + # classifier with no key 401s on every request, and classify() + # would swallow that into a silent fallback tier forever. + raise HTTPException( + 503, + f"classifier.api_key_env={key_env!r} is set but {key_env} is " + "not in the environment.", + ) + else: + api_key = "ollama" return OpenAI( - base_url=cfg.classifier.base_url, api_key="ollama", max_retries=0 + base_url=cfg.classifier.base_url, api_key=api_key, max_retries=0 ) @@ -229,6 +293,40 @@ def _provider_client(provider: str) -> OpenAI: # --- classification ------------------------------------------------------- +def clamp_for_classifier(text: str, max_chars: int) -> str: + """Shrink classifier input to the part that carries the instruction. + + The classifier decides a category and a tier. It does not need the + document, only the instruction wrapped around it — and feeding it the + document is actively harmful, not merely wasteful. Measured on a ~20k-token + prompt, both local models failed and neither failed cleanly: + + | model | wall clock | result | + |---|---|---| + | qwen3.5 | 28.7s | `finish_reason=length`, empty content | + | mistral-nemo | 41.8s | echoed the input back inside the JSON | + + Both end as unparseable output, i.e. ~30-40s of local inference spent to + reach `source: "fallback"` — the same answer an immediate failure would + have given, minus the wait. + + Head AND tail, because the instruction sits at either end depending on how + the caller phrased it ("Translate this: " vs " — translate + this"), and the middle of a pasted document is the one part that never + carries it. + + Nothing is lost by clamping. ``required_context_tokens`` does not come + from this call in the path that matters — ``chat_completions`` measures + the real conversation with ``estimate_prompt_tokens`` and takes the + larger, precisely because the classifier's own estimate was measured two + orders of magnitude low. + """ + if max_chars <= 0 or len(text) <= max_chars: + return text + half = max_chars // 2 + return f"{text[:half]}\n\n[... {len(text) - max_chars} characters elided ...]\n\n{text[-half:]}" + + def classify(task: str, context: Optional[str]) -> Classification: """Ask the local model to categorize and size the task. @@ -243,15 +341,24 @@ def classify(task: str, context: Optional[str]) -> Classification: + "\n".join(f" - {c}" for c in categories) ) user_content = task if not context else f"{task}\n\n--- context ---\n{context}" + user_content = clamp_for_classifier(user_content, cfg.classifier.max_input_chars) client = _classifier_client() + started = time.perf_counter() try: resp = _classify_once(client, system_prompt, user_content) except (OpenAIError, ValueError, KeyError, TypeError, json.JSONDecodeError) as e: # A local model that is slow, restarting, or mid-thought must not take # the caller down with it. Routing degrades to a configured mid tier, # flagged as source='fallback' so the response says so. - print(f"classifier fallback ({type(e).__name__}: {e})", file=sys.stderr) + logs.warning( + "fallback", + error=type(e).__name__, + detail=str(e)[:200], + tier=cfg.classifier.fallback_tier, + cat=cfg.classifier.fallback_category, + ms=_ms(started), + ) return Classification( task_category=cfg.classifier.fallback_category, task_tier=cfg.classifier.fallback_tier, @@ -260,6 +367,17 @@ def classify(task: str, context: Optional[str]) -> Classification: source="fallback", ) + # The single blocking LLM call on the request path, so its latency is the + # floor under every routed request and worth seeing on its own. + logs.debug( + "classify", + cat=resp.task_category, + tier=resp.task_tier, + ctx=resp.required_context_tokens, + confidence=resp.confidence, + chars=len(user_content), + ms=_ms(started), + ) return resp @@ -327,18 +445,17 @@ def load_candidates(conn: sqlite3.Connection, category: str) -> list[dict]: Filtering happens in ``routing.py`` rather than in SQL so the hard filters stay in one testable place; the catalog is 19 rows. - ``cost`` and ``eco`` are both measured, and both come ONLY from the - fixed reference workload written by ``seed_energy.py`` - (``task_category='seed_reference'``). + ``eco`` and ``energy`` come ONLY from the fixed reference workload written + by ``seed_energy.py`` (``task_category='seed_reference'``). ``cost`` is + read here too, but as a FALLBACK: ``rank_candidates`` overwrites it with + ``routing.estimated_cost``, priced from catalog token prices scaled to the + request in hand. A fixed 400-token sweep ranks models backwards for real + traffic -- glm-5.2-fast looked 3.2x cheaper than deepseek-v4-flash on a toy + prompt and is 5.0x more expensive on a 70k one -- so the measured median + survives only for a row the catalog has no price for. - Measured, because the catalog's list price is not what is billed — - NeuralWatt charges per kWh, so on the same prompt kimi-k2.7-code-fast - ($4/1M listed) cost 10x more than kimi-k3-fast ($15/1M listed). Mean - observed ``cost_usd`` needs no billing formula in the code and already - accounts for the flex tier and the per-request price ceiling. - - Reference-only, because energy depends far more on the shape of a - request than on the model: across organic traffic on a single model, + Reference-only for the rest, because energy depends far more on the shape + of a request than on the model: across organic traffic on a single model, per-request energy spanned 19x purely because prompt and completion sizes differed. Averaging that in would rank models by what they happened to be asked. Organic observations stay in the table for cost @@ -374,14 +491,20 @@ def load_candidates(conn: sqlite3.Connection, category: str) -> list[dict]: """ rows = conn.execute( """ - SELECT m.*, p.blended_score AS proficiency + SELECT m.*, + p.blended_score AS proficiency, + tp.blended_score AS tool_proficiency FROM models m LEFT JOIN proficiency p ON p.model_id = m.model_id AND p.provider = m.provider AND p.category = ? + LEFT JOIN proficiency tp + ON tp.model_id = m.model_id + AND tp.provider = m.provider + AND tp.category = ? """, - (category,), + (category, cfg.routing.tool_use_category), ).fetchall() # Medians are computed here rather than in SQL — SQLite has no MEDIAN, and @@ -474,15 +597,34 @@ def route(req: TaskRequest) -> RouteResponse: finally: conn.close() - eligible = select_candidates( - rows, + filters = dict( required_context_tokens=classification.required_context_tokens, required_tier=classification.task_tier, latency_tolerance=latency_tolerance, allowed_access_levels=cfg.routing.allowed_access_levels, exclude_stale=cfg.freshness.exclude_stale, exclude_deprecated=cfg.freshness.exclude_deprecated, + min_tool_proficiency=( + cfg.routing.min_tool_proficiency if req.tools_present else None + ), + require_vision=cfg.routing.require_vision if req.has_images else False, + require_json_mode=( + cfg.routing.require_json_mode if req.require_json_mode else False + ), ) + eligible = select_candidates(rows, **filters) + if logs.enabled_for_debug(): + # "No model satisfies the hard filters" is otherwise a dead end with no + # explanation. One line per drop, naming the filter and its numbers, is + # the difference between a five-minute answer and an afternoon. + survivors = {r["model_id"] for r in eligible} + for row in rows: + if row["model_id"] not in survivors: + logs.debug( + "filter", + model=row["model_id"], + reason=rejection_reason(row, **filters), + ) ranked = rank_candidates( eligible, quality_tolerance=cfg.objective.quality_tolerance, @@ -493,6 +635,17 @@ def route(req: TaskRequest) -> RouteResponse: cache_rate=cfg.objective.assumed_cache_rate, ) + if logs.enabled_for_debug(): + for position, row in enumerate(ranked[:5]): + logs.debug( + "rank", + pos=position, + model=row["model_id"], + prof=row["proficiency_score"], + est_usd=row.get("cost"), + kwh=row.get("energy"), + ) + return RouteResponse( classification=classification, latency_tolerance=latency_tolerance, @@ -502,6 +655,47 @@ def route(req: TaskRequest) -> RouteResponse: ) +def log_decision( + decision: RouteResponse, + *, + tools: bool, + ms: int, + src: Optional[str] = None, + ctx_src: str = "classifier", +) -> None: + """The one INFO line per request that says what the router decided. + + Emitted by the ENDPOINTS rather than by ``route()``, because + ``chat_completions`` routes twice when the measured conversation turns out + larger than the classifier's estimate -- which, with opencode sending ~32K + of system prompt, is essentially every request. Logging inside route() + would double every line for no new information. + + That second pass is also why ``src`` can be overridden. The re-route passes + category and tier back in, so the resulting Classification reads + ``source='override'`` -- which in a log line means "the client chose this", + and the client did no such thing: the classifier ran, and only the context + figure was replaced. ``ctx_src`` records that separately, so the line says + what actually happened rather than what the second call looked like. + """ + selected = decision.selected + logs.info( + "route", + cat=decision.classification.task_category, + tier=decision.classification.task_tier, + ctx=decision.classification.required_context_tokens, + ctx_src=ctx_src, + src=src or decision.classification.source, + tools=tools, + latency=decision.latency_tolerance, + cand=decision.candidates_considered, + pick=selected.model_id if selected else None, + est_usd=selected.cost if selected else None, + prof=selected.proficiency_score if selected else None, + ms=ms, + ) + + # --- energy / cost accounting -------------------------------------------- class Telemetry(BaseModel): @@ -622,14 +816,20 @@ def run_local_verification( # thought and empty content, and raising the cap to 8,192 just bought more # thought. With think=False it answers in 25 tokens, and returned the same # verdict on 3/3 repeats of four labelled cases. - url = cfg.classifier.base_url.rstrip("/") + # + # This endpoint is configured separately from the classifier's and is NOT + # derived from it. It used to be, and that silently coupled two unrelated + # decisions: pointing classification at a cloud provider would have sent + # these requests to /api/chat, which does not exist. + url = cfg.verification.base_url.rstrip("/") if url.endswith("/v1"): url = url[: -len("/v1")] + verify_model = cfg.verification.model or cfg.classifier.model try: resp = requests.post( f"{url}/api/chat", json={ - "model": cfg.classifier.model, + "model": verify_model, "think": False, "stream": False, "format": "json", @@ -650,13 +850,12 @@ def run_local_verification( resp.raise_for_status() raw = ((resp.json() or {}).get("message") or {}).get("content") or "" except (requests.RequestException, ValueError) as e: - print(f"local verification unavailable ({type(e).__name__})", file=sys.stderr) + logs.warning("verify_unavailable", error=type(e).__name__, model=verify_model) return check = interpret_local_verdict(parse_verdict_json(raw)) if check is None: - print("local verification returned nothing usable; no sample recorded", - file=sys.stderr) + logs.warning("verify_unusable", model=verify_model) return log_verification( @@ -753,6 +952,9 @@ def health(): "scoring": scoring_coverage(), "classifier_reachable": classifier_ok, "classifier_model": cfg.classifier.model, + # The tier labels, documented as being for logging and dashboards. + # /health is the dashboard surface, so this is where they reach one. + "tiers": cfg.tiers, "providers": list(cfg.dispatch_providers), "api_keys_present": { name: bool(os.environ.get(p.api_key_env)) @@ -822,7 +1024,9 @@ def scoring_coverage() -> dict: if missing_energy: warnings.append( f"{len(missing_energy)}/{total} routable models have no reference-workload " - f"observations — cost and eco score the neutral 0.5 for them. " + f"observations — eco scores the neutral 0.5 for them and " + f"objective.max_energy_per_request cannot bound them. (Cost is " + f"unaffected: it is priced per request from catalog prices.) " f"Run: python seed_energy.py --samples 7" ) if missing_proficiency: @@ -849,8 +1053,14 @@ def quota_burn() -> Optional[dict]: This counts only what the ROUTER saw. Traffic that bypasses it — the eval harness, ad-hoc scripts, a client pointed straight at the provider — is - invisible here, so treat this as a floor on real consumption. Measured - against the provider's own 24h figure it accounted for ~86% of energy. + invisible here, so treat this as a floor on real consumption. + + How good a floor, measured against the provider's own 24h session view: + **96.9%** on 2026-08-23 (0.1543 kWh metered against 0.1593 kWh charged, + 567 calls, 40.7M tokens). An earlier check put it at ~86%; the gap closed + as more traffic moved through the router rather than around it. The + remainder is mostly `eval_proficiency.py`, which calls the provider + directly and writes no observation. """ if not cfg.objective.plan_kwh_per_period: return None @@ -860,7 +1070,7 @@ def quota_burn() -> Optional[dict]: """ SELECT COALESCE(SUM(energy_kwh), 0) kwh, COUNT(*) n FROM energy_observations - WHERE observed_at > datetime('now', '-30 days') + WHERE julianday(observed_at) > julianday('now', '-30 days') """ ).fetchone() finally: @@ -917,6 +1127,17 @@ def _most_recent_if_unambiguous(conn: sqlite3.Connection): long ago and makes every report look ambiguous. Observed directly: at 30 minutes, a real opencode run was refused because of test traffic from earlier in the same session of work. + + ``julianday()`` on both sides, NOT a string comparison against + ``datetime('now', ...)``. ``observed_at`` is written by + ``datetime.now(timezone.utc).isoformat()``, so it carries a 'T' separator + (``2026-08-23T00:20:04.577131+00:00``) while ``datetime()`` returns a space + (``2026-08-23 00:18:04``). 'T' sorts after ' ', so once the dates match the + time of day never participates and the window silently becomes "everything + today". Measured on the live DB: 14 rows across 2 sessions matched where + the correct comparison matched 0 -- which is what produced the spurious + 409s, and made this setting inert at every value. ``poller.mark_stale`` + already had the right form. """ window = cfg.verification.outcome_attribution_window_seconds recent = conn.execute( @@ -925,7 +1146,7 @@ def _most_recent_if_unambiguous(conn: sqlite3.Connection): FROM energy_observations WHERE request_id IS NOT NULL AND task_category != ? - AND observed_at > datetime('now', '-{int(window)} seconds') + AND julianday(observed_at) > julianday('now', '-{int(window)} seconds') ORDER BY id DESC LIMIT 50 """, (SEED_CATEGORY,), @@ -942,6 +1163,10 @@ def _most_recent_if_unambiguous(conn: sqlite3.Connection): def report_outcome(report: OutcomeReport): """Record whether a completion actually worked. + Logged either way. A report that lands is the only ground-truth signal the + router gets; one that is refused is the client discovering its reports go + nowhere, which is worth seeing from this side too. + This is the only ground truth the router can get. Everything else it records is a proxy: structural checks know whether code *parses*, the local checker guesses whether prose *looks* right, and neither knows @@ -958,6 +1183,7 @@ def report_outcome(report: OutcomeReport): inflate scores if counted. 'succeeded' from a client means the work worked. """ + logs.new_trace() conn = _db() try: if report.request_id: @@ -991,6 +1217,12 @@ def report_outcome(report: OutcomeReport): conn.close() if row is AMBIGUOUS: + logs.warning( + "outcome_refused", + reason="ambiguous", + source=report.source, + ok=report.ok, + ) raise HTTPException( 409, "More than one conversation has been routed recently, so this " @@ -1001,6 +1233,12 @@ def report_outcome(report: OutcomeReport): ) if row is None: + logs.warning( + "outcome_refused", + reason="unknown_request", + rid=report.request_id, + source=report.source, + ) # Deliberately a 404 rather than a silent accept: a client whose # reports go nowhere should find out, not quietly train nothing. raise HTTPException( @@ -1021,6 +1259,14 @@ def report_outcome(report: OutcomeReport): verdict="succeeded" if report.ok else "failed", detail=" | ".join(x for x in (report.detail, report.source) if x)[:300], ) + logs.info( + "outcome", + verdict="succeeded" if report.ok else "failed", + model=row["model_id"], + rid=row["request_id"], + cat=row["task_category"], + source=report.source, + ) return { "recorded": True, "request_id": row["request_id"], @@ -1033,7 +1279,16 @@ def report_outcome(report: OutcomeReport): @app.post("/route", response_model=RouteResponse) def route_endpoint(req: TaskRequest): """Classify and pick a model without calling it.""" - return route(req) + logs.new_trace() + started = time.perf_counter() + decision = route(req) + log_decision( + decision, + tools=req.tools_present, + ms=_ms(started), + ctx_src="caller" if req.required_context_tokens is not None else "classifier", + ) + return decision # --- OpenAI-compatible surface ------------------------------------------- @@ -1105,12 +1360,17 @@ def session_directory(messages: list[dict]) -> Optional[str]: return max(winners, key=len) if winners else None -def estimate_prompt_tokens(messages: list[dict]) -> int: +def estimate_prompt_tokens( + messages: list[dict], tools: Optional[list[dict]] = None +) -> int: """Floor estimate of a conversation's token count, from its characters. Deliberately crude. Its only job is to stop a long conversation being routed to a model that cannot hold it when the classifier lowballs its - own input size. + own input size. Tool definitions ride along in ``upstream_body`` on every + call, so they count against the same context window and belong in this + estimate too -- omitting them undercounts every tool-carrying request, + which is nearly all agent traffic. """ chars = 0 for m in messages: @@ -1122,6 +1382,8 @@ def estimate_prompt_tokens(messages: list[dict]) -> int: for part in content: if isinstance(part, dict) and isinstance(part.get("text"), str): chars += len(part["text"]) + if tools: + chars += len(json.dumps(tools)) return chars // CHARS_PER_TOKEN @@ -1140,6 +1402,233 @@ def _last_user_text(messages: list[dict]) -> str: return "" +def _count_images(messages: list[dict]) -> int: + """The number of image_url parts across the whole conversation. + + Scans all messages, not just the last: any image in an earlier turn still + has to be understood by whoever answers the latest one, exactly as in + ``detect_capabilities``. A part counts only when it is a dict whose type + is ``image_url``, mirroring the detection rule. + """ + count = 0 + for m in messages: + content = m.get("content") + if not isinstance(content, list): + continue + for part in content: + if isinstance(part, dict) and part.get("type") == "image_url": + count += 1 + return count + + +def _image_payload_bytes(messages: list[dict]) -> int: + """Total base64 payload length across all image_url parts. + + The "length" is the raw length of the payload string carried by each part, + whatever its shape. A part may spell the data inline in ``image_url.url`` + or in a separate ``detail``/``data`` field; this counts the bytes of the + image_url value so an oversized upload is skipped rather than POSTed to a + local model that would reject it. + """ + total = 0 + for m in messages: + content = m.get("content") + if not isinstance(content, list): + continue + for part in content: + if not isinstance(part, dict) or part.get("type") != "image_url": + continue + parts = part.get("image_url") + if isinstance(parts, dict): + # Inline base64 lives in `url` as data:image/...;base64,. + total += len(parts.get("url", "")) + elif isinstance(parts, str): + total += len(parts) + return total + + +def _local_vision_data_uris_ok(messages: list[dict]) -> bool: + """Whether every image_url part is an inline ``data:`` URI, not a URL. + + The local vision model (typically Ollama on this host) is trusted to read + the pixels of an inline image, but pointing it at an arbitrary ``http`` URL + would let a request make the local model fetch internal or private + resources the caller cannot reach directly — SSRF by indirection. OpenAI's + ``image_url`` schema allows either a base64 ``data:`` URI or a remote URL, + so a URL is a real possibility, not a theoretical edge. The fallback is + therefore refused unless EVERY part is a data URI; a single remote/unknown + URL declines the fallback (falls through to the 422) rather than handing a + network-fetching capability to an unauthenticated caller. + """ + for m in messages: + content = m.get("content") + if not isinstance(content, list): + continue + for part in content: + if not isinstance(part, dict) or part.get("type") != "image_url": + continue + value = part.get("image_url") + if isinstance(value, dict): + value = value.get("url", "") + if not isinstance(value, str) or not value.startswith("data:"): + return False + return True + + +def _run_local_vision(messages: list[dict], lv_cfg) -> Optional[str]: + """Ask a local vision model to caption the request's images, if it can. + + Runs only when routing found no cloud vision candidate and the local + vision path is enabled — the fallback that keeps a multimodal request + answerable without spending a cloud completion on a model that would 400. + Returns None on any failure or budget refusal so the caller falls through + to the 422; a hidden failure must not masquerade as a 200. + + The messages passed are the ORIGINAL list that still carries the image_url + parts — the local model reads the pixels, so the parts cannot be stripped. + + No remote URL is ever forwarded: every image part must be an inline + ``data:`` URI, or the fallback is declined (see ``_local_vision_data_uris_ok``). + """ + if not _local_vision_data_uris_ok(messages): + logs.warning("local_vision_skip", reason="remote_url") + return None + if _count_images(messages) > lv_cfg.max_images: + logs.warning( + "local_vision_skip", reason="too_many_images", + n=_count_images(messages), max=lv_cfg.max_images, + ) + return None + if _image_payload_bytes(messages) > lv_cfg.max_image_bytes: + logs.warning( + "local_vision_skip", reason="oversized_image", + bytes=_image_payload_bytes(messages), max=lv_cfg.max_image_bytes, + ) + return None + url = f"{lv_cfg.base_url.rstrip('/')}/chat/completions" + headers = {} + if lv_cfg.api_key_env: + key = os.environ.get(lv_cfg.api_key_env) + if not key: + logs.warning("local_vision_skip", reason="missing_api_key") + return None + headers["Authorization"] = f"Bearer {key}" + try: + resp = requests.post( + url, + headers=headers, + json={ + "model": lv_cfg.model, + "messages": messages, + "temperature": 0, + "max_tokens": 2048, + "stream": False, + }, + timeout=lv_cfg.timeout_seconds, + ) + except requests.RequestException as e: + logs.warning("local_vision_skip", reason="request", error=type(e).__name__) + return None + if resp.status_code != 200: + # Deliberately no response body in the log: an upstream error can echo + # request content or headers back, and the journal must not become a + # place reflected prompt/image data lands. Status alone is enough to + # see that the fallback could not answer. + logs.warning( + "local_vision_skip", reason="status", status=resp.status_code, + ) + return None + try: + payload = resp.json() + except ValueError: + logs.warning("local_vision_skip", reason="unparseable") + return None + content = (payload.get("choices") or [{}])[0].get("message", {}).get("content") + # A 200 with empty or non-string content is a failed answer, not a success. + # Returning ``None`` lets the caller fall through to the 422 so an empty + # local caption cannot masquerade as a 200 — the same "no hidden failures" + # contract as every other early return above. + if not isinstance(content, str) or not content.strip(): + return None + return content + + +def _local_vision_response(content: str, *, streaming: bool) -> Response: + """Shape a local vision answer into an OpenAI completion, streamed or not. + + ``streaming`` diverges where the delivery differs: buffered answers get the + structural-verification header so a client can see the verdict, streamed + answers cannot carry headers after the first chunk so they just emit the + SSE events. + """ + payload = { + "id": f"local-vision-{secrets.token_hex(6)}", + "object": "chat.completion", + "created": int(time.time()), + "model": cfg.local_vision.model, + "choices": [ + { + "index": 0, + "message": {"role": "assistant", "content": content}, + "finish_reason": "stop", + } + ], + "usage": {"prompt_tokens": 0, "completion_tokens": 0}, + } + logs.info( + "dispatch", + model=cfg.local_vision.model, + provider="local", + category="general_chat", + stream=streaming, + ) + if not streaming: + verdict = verify_response(content, "stop", has_tool_calls=False).verdict + return JSONResponse( + content=payload, + headers={ + "X-Router-Model": cfg.local_vision.model, + "X-Router-Verification": verdict, + }, + ) + + def stream(): + yield f"data: {json.dumps({'id': payload['id'], 'object': 'chat.completion.chunk', 'created': payload['created'], 'model': payload['model'], 'choices': [{'index': 0, 'delta': {'role': 'assistant', 'content': content}, 'finish_reason': None}]})}\n\n".encode() + yield f"data: {json.dumps({'id': payload['id'], 'object': 'chat.completion.chunk', 'created': payload['created'], 'model': payload['model'], 'choices': [{'index': 0, 'delta': {}, 'finish_reason': 'stop'}], 'usage': payload['usage']})}\n\n".encode() + yield b"data: [DONE]\n\n" + + return StreamingResponse(stream(), media_type="text/event-stream") + + +def _check_pinned_capabilities(model_id: str, caps) -> None: + """Reject a pinned model that cannot satisfy the request's capabilities. + + Reads the ``models`` row by ``model_id`` and fails closed on any missing + or NULL capability flag, matching the routing gate: a capability that + cannot be confirmed is treated as absent, because dispatching to a model + that might lack it is a guaranteed provider 400. + """ + conn = _db() + try: + row = conn.execute( + "SELECT supports_vision, supports_json_mode FROM models " + "WHERE model_id = ? AND provider = 'neuralwatt'", + (model_id,), + ).fetchone() + finally: + conn.close() + if caps.has_images and (row is None or row["supports_vision"] != 1): + raise HTTPException( + 422, + f"{model_id} does not support vision; the request carries image parts", + ) + if caps.require_json_mode and (row is None or row["supports_json_mode"] != 1): + raise HTTPException( + 422, + f"{model_id} does not support json mode; the response_format requires it", + ) + + @app.get("/v1/models") def list_models(): """The routable catalog, in OpenAI's list shape. @@ -1201,10 +1690,18 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): coding agent still renders tokens as they arrive; the telemetry is scraped from the trailing SSE comments on the way past. """ + logs.new_trace() + started = time.perf_counter() messages = body.get("messages") or [] if not messages: raise HTTPException(400, "messages is required") + # What this request needs, read from the body once and used by both the + # routed and pass-through branches. Stated by the caller, not guessed at: + # a tools array, an image_url part and a response_format each say exactly + # whether a capability is required. + caps = detect_capabilities(body) + requested = body.get("model") or ROUTER_MODEL # Some clients (opencode among them) send the model as `provider/model`. # Real ids can contain slashes too (`deepseek-ai/DeepSeek-V4-Flash`), so @@ -1214,41 +1711,111 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): if wants_routing: requested = bare + logs.debug( + "request", + asked=requested, + routed=wants_routing, + stream=bool(body.get("stream")), + tools=caps.tools_present, + msgs=len(messages), + capped=body.get("max_tokens"), + ) + if wants_routing: latency = BATCH if requested == ROUTER_MODEL_BATCH else INTERACTIVE + tools_present = caps.tools_present decision = route( TaskRequest( task=_last_user_text(messages), latency_tolerance=latency, + tools_present=tools_present, + has_images=caps.has_images, + require_json_mode=caps.require_json_mode, # Take whichever is larger: what the classifier thinks it # needs, or what the conversation actually measures. required_context_tokens=None, ) ) - measured = estimate_prompt_tokens(messages) + # The classifier's own verdict, before the re-route below rewrites the + # Classification's source to 'override'. + classified_src = decision.classification.source + ctx_src = "classifier" + measured = estimate_prompt_tokens(messages, tools=body.get("tools")) if measured > decision.classification.required_context_tokens: + ctx_src = "measured" decision = route( TaskRequest( task=_last_user_text(messages), latency_tolerance=latency, + tools_present=tools_present, + has_images=caps.has_images, + require_json_mode=caps.require_json_mode, task_category=decision.classification.task_category, task_tier=decision.classification.task_tier, required_context_tokens=measured, ) ) if decision.selected is None: + if ( + caps.has_images + and cfg.local_vision.enabled + ): + fallback = _run_local_vision(messages, cfg.local_vision) + if fallback is not None: + return _local_vision_response( + fallback, streaming=bool(body.get("stream")) + ) + # Name every constraint that was applied, including the one the + # caller did not ask for explicitly. A 422 that omits the tool + # filter sends someone hunting through context and tier settings + # for a limit that came from the `tools` array. + limits = [ + f"tier >= {decision.classification.task_tier}", + f"context >= {decision.classification.required_context_tokens} tokens", + decision.latency_tolerance, + ] + if tools_present and cfg.routing.min_tool_proficiency is not None: + limits.append( + f"tool-use proficiency >= {cfg.routing.min_tool_proficiency} " + "(request carries tool definitions)" + ) + if caps.has_images: + limits.append( + "vision-capable model (request carries image(s))" + ) + if caps.require_json_mode: + limits.append( + "json-mode-capable model (response_format requires it)" + ) + logs.warning("no_candidate", limits="; ".join(limits)) raise HTTPException( 422, - "No model satisfies the hard filters for this request " - f"(tier >= {decision.classification.task_tier}, context >= " - f"{decision.classification.required_context_tokens} tokens, " - f"{decision.latency_tolerance}).", + "No model satisfies the hard filters for this request (" + + ", ".join(limits) + + ").", ) + log_decision( + decision, + tools=tools_present, + ms=_ms(started), + src=classified_src, + ctx_src=ctx_src, + ) target = decision.selected.model_id provider = decision.selected.provider category = decision.classification.task_category else: target, provider, category = requested, "neuralwatt", "general_chat" + # Not a routing decision at all, and worth saying so plainly: a client + # pinned to one model gets none of the filtering or ranking below. + logs.info("passthrough", model=target, stream=bool(body.get("stream"))) + + # A pinned model id is dispatched as asked, but a pin that cannot + # possibly satisfy the request should fail with a clear 422 instead of + # an opaque 400 from the provider. The pin could never have worked + # anyway, so nothing is lost by rejecting early. + if caps.has_images or caps.require_json_mode: + _check_pinned_capabilities(requested, caps) settings = cfg.dispatch_providers[provider] api_key = os.environ.get(settings.api_key_env) @@ -1285,10 +1852,20 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): if cfg.iteration.enabled and wants_routing else 0 ) - alternatives = [ - (c.model_id, model_output_ceiling(c.model_id, provider)) - for c in decision.runners_up - ] + # Bound to the routed case. A pass-through request never produced a + # `decision`, so reading its runners-up raised NameError before the + # provider was ever called — every non-streaming request naming a real + # model id returned 500. Empty is also the right value: there is no + # ranking behind a model the caller named itself, and `budget` is 0 for + # a pass-through, so the loop breaks on its first pass regardless. + alternatives = ( + [ + (c.model_id, model_output_ceiling(c.model_id, provider)) + for c in decision.runners_up + ] + if wants_routing + else [] + ) current_model = target current_max_tokens = body.get("max_tokens") attempts_used = 0 @@ -1298,20 +1875,30 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): attempt_body = {**body, "model": current_model} if current_max_tokens is not None: attempt_body["max_tokens"] = current_max_tokens + upstream_started = time.perf_counter() resp = requests.post(url, headers=headers, json=attempt_body, timeout=600) + upstream_ms = _ms(upstream_started) if resp.status_code >= 400: + logs.error( + "upstream", + model=current_model, + status=resp.status_code, + detail=resp.text[:200], + ms=upstream_ms, + ) raise HTTPException(resp.status_code, resp.text[:500]) payload = resp.json() usage = payload.get("usage") or {} completion_tokens = usage.get("completion_tokens") request_id = payload.get("id") + telemetry = extract_telemetry(payload) log_observation( current_model, provider, category, request_id, session_key, session_dir, prompt_tokens=usage.get("prompt_tokens"), completion_tokens=completion_tokens, - telemetry=extract_telemetry(payload), + telemetry=telemetry, ) choice = (payload.get("choices") or [{}])[0] @@ -1333,6 +1920,22 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): ), ) + logs.info( + "dispatch", + model=current_model, + rid=request_id, + sess=session_key, + cat=category, + p_tok=usage.get("prompt_tokens"), + c_tok=completion_tokens, + kwh=telemetry.energy_kwh, + usd=telemetry.cost_usd, + verdict=result.verdict, + attempt=attempts_used + 1, + upstream_ms=upstream_ms, + total_ms=_ms(started), + ) + if not result.failed or attempts_used >= budget: break @@ -1350,7 +1953,13 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): break retry_trail.append(f"{result.verdict}:{plan.detail}") - print(f"iteration: {plan.detail} (was {result.verdict})", file=sys.stderr) + logs.warning( + "retry", + verdict=result.verdict, + reason=plan.reason, + to=plan.model_id, + detail=plan.detail, + ) if plan.reason == "malformed": alternatives = alternatives[1:] current_model = plan.model_id @@ -1382,7 +1991,16 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): }, ) + # Bound OUT here, not read inside. starlette resumes a StreamingResponse's + # generator through a threadpool, and every next() runs in a fresh COPY of + # the caller's context, so a ContextVar set inside is gone by the following + # resumption -- including the finally block that logs the dispatch line. + # Measured: it logged id=- until the trace was carried explicitly. This is + # the path all agent traffic takes. + slog = logs.bind() + def proxy(): + stream_started = time.perf_counter() collected: dict[str, dict] = {} usage: dict = {} # Accumulated so a streamed answer gets the same structural check as a @@ -1397,6 +2015,14 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): ) if upstream.status_code >= 400: detail = upstream.text[:500] + slog.error( + "upstream", + model=target, + status=upstream.status_code, + detail=detail[:200], + stream=True, + ms=_ms(stream_started), + ) yield f"data: {json.dumps({'error': {'message': detail}})}\n\n".encode() return try: @@ -1424,21 +2050,52 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): except json.JSONDecodeError: pass yield f"{raw}\n".encode() + except requests.exceptions.RequestException as exc: + # The upstream connection can die mid-stream (NeuralWatt closing + # early, a network blip). Left uncaught, this propagates out of + # the generator and Starlette surfaces it as an unhandled ASGI + # exception -- a full traceback in the log and the client's + # connection just cut dead with no [DONE]. Whatever was collected + # before the break still gets logged below via `finally`. + slog.error( + "upstream_stream_broken", + model=target, + rid=stream_request_id, + error=str(exc), + ms=_ms(stream_started), + ) + yield f"data: {json.dumps({'error': {'message': f'upstream stream interrupted: {exc}'}})}\n\n".encode() + yield b"data: [DONE]\n\n" finally: upstream.close() # Logged even on a client disconnect — the energy was spent. + telemetry = extract_telemetry(collected) log_observation( target, provider, category, stream_request_id, session_key, session_dir, prompt_tokens=usage.get("prompt_tokens"), completion_tokens=usage.get("completion_tokens"), - telemetry=extract_telemetry(collected), + telemetry=telemetry, ) streamed = "".join(content_parts) result = verify_response( streamed, finish_reason, has_tool_calls=stream_tool_calls ) ct = usage.get("completion_tokens") + slog.info( + "dispatch", + model=target, + rid=stream_request_id, + sess=session_key, + cat=category, + stream=True, + p_tok=usage.get("prompt_tokens"), + c_tok=ct, + kwh=telemetry.energy_kwh, + usd=telemetry.cost_usd, + verdict=result.verdict, + total_ms=_ms(stream_started), + ) log_verification( target, provider, category, stream_request_id, kind="structural", verdict=result.verdict, detail=result.detail, @@ -1468,7 +2125,15 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): @app.post("/dispatch", response_model=DispatchResponse) def dispatch_endpoint(req: TaskRequest): + logs.new_trace() + started = time.perf_counter() decision = route(req) + log_decision( + decision, + tools=req.tools_present, + ms=_ms(started), + ctx_src="caller" if req.required_context_tokens is not None else "classifier", + ) if decision.selected is None: raise HTTPException( 422, @@ -1485,6 +2150,7 @@ def dispatch_endpoint(req: TaskRequest): if req.context: messages.insert(0, {"role": "system", "content": req.context}) + upstream_started = time.perf_counter() try: # with_raw_response because the energy and cost blocks sit outside the # OpenAI schema and the parsed object drops them. @@ -1492,6 +2158,13 @@ def dispatch_endpoint(req: TaskRequest): model=selected.model_id, messages=messages ) except OpenAIError as e: + logs.error( + "upstream", + model=selected.model_id, + error=type(e).__name__, + detail=str(e)[:200], + ms=_ms(upstream_started), + ) raise HTTPException(502, f"Dispatch to {selected.model_id} failed: {e}") payload = json.loads(raw.text) @@ -1513,6 +2186,18 @@ def dispatch_endpoint(req: TaskRequest): choices = payload.get("choices") or [{}] content = (choices[0].get("message") or {}).get("content") or "" + logs.info( + "dispatch", + model=selected.model_id, + rid=payload.get("id"), + p_tok=prompt_tokens, + c_tok=completion_tokens, + kwh=telemetry.energy_kwh, + usd=telemetry.cost_usd, + upstream_ms=_ms(upstream_started), + total_ms=_ms(started), + ) + return DispatchResponse( route=decision, content=content, diff --git a/eval_proficiency.py b/eval_proficiency.py index ef7c206..ba77ff7 100644 --- a/eval_proficiency.py +++ b/eval_proficiency.py @@ -31,6 +31,7 @@ import argparse import json import os import re +import secrets import sqlite3 import subprocess import sys @@ -96,14 +97,47 @@ def strip_fences(text: str) -> str: return code.strip() +def parse_check_results(stdout: str, marker: str) -> dict[int, bool]: + """Read the harness's own verdicts out of a script's stdout. + + Keyed by check INDEX rather than counted, which bounds the total at the + number of checks even if a line somehow repeats, and drops anything that + is not exactly one of the harness's own three-token lines. + """ + results: dict[int, bool] = {} + for line in stdout.splitlines(): + parts = line.split() + if len(parts) != 3 or parts[0] != marker or parts[2] not in ("PASS", "FAIL"): + continue + try: + index = int(parts[1]) + except ValueError: + continue + results[index] = parts[2] == "PASS" + return results + + def score_code(model_output: str, checks: list[str]) -> tuple[float, str]: """Execute the model's code and eval each check against it. Returns (fraction of checks passing, detail). Partial credit is deliberate: a function correct on three of four cases is genuinely better than one that fails everything, and a binary score throws that away. + + The verdict lines carry a per-run nonce because this harness runs the + model's code as ``__main__``, and models routinely append a demo block. One + that prints the word PASS -- `print("self-test:", f(2) == 4 and "PASS")` -- + was counted as a passing check by the old substring count, and a two-check + task scored 1.50. The model cannot predict the nonce, so it cannot vote on + its own work. That was the fourth harness bug here to score the rig rather + than the model; this one inflated instead of deflating, which is why it + went unnoticed. """ + if not checks: + return 0.0, "no checks" + code = strip_fences(model_output) + marker = f"CHECK-{secrets.token_hex(8)}" harness = ( HARNESS_PREAMBLE + code @@ -115,7 +149,7 @@ def score_code(model_output: str, checks: list[str]) -> tuple[float, str]: + " _ok = bool(eval(_c))\n" + " except Exception:\n" + " _ok = False\n" - + " print('CHECK', _i, 'PASS' if _ok else 'FAIL')\n" + + f" print({marker!r}, _i, 'PASS' if _ok else 'FAIL')\n" ) with tempfile.TemporaryDirectory() as tmp: script = Path(tmp) / "harness.py" @@ -131,9 +165,8 @@ def score_code(model_output: str, checks: list[str]) -> tuple[float, str]: except subprocess.TimeoutExpired: return 0.0, "timeout" - passed = proc.stdout.count("PASS") - if not checks: - return 0.0, "no checks" + results = parse_check_results(proc.stdout, marker) + passed = sum(1 for i in range(len(checks)) if results.get(i)) if passed == 0 and proc.returncode != 0: # Distinguish "wrote broken code" from "wrote code that fails cases" first_line = (proc.stderr or "").strip().splitlines()[-1:] or [""] @@ -378,21 +411,32 @@ def eval_identities(conn: sqlite3.Connection, cfg: RouterConfig) -> list[dict]: ] -ALTERNATE_JUDGE = "qwen3.6-35b" +# Tried in order when the configured judge belongs to the model under test. +# A LIST rather than one name, because a single alternate can itself be that +# model: with `--judge-model qwen3.6-35b`, the old guard fired on qwen3.6-35b +# and handed back qwen3.6-35b -- the exact conflict it exists to prevent, and +# silent, because self-judged scores look like any other judge output. +ALTERNATE_JUDGES = ("kimi-k3", "qwen3.6-35b", "gemma-4-31b") -def judge_for(model_id: str, default_judge: str) -> str: - """Pick a judge that is not the model being judged. +def judge_for(model_id: str, default_judge: str) -> Optional[str]: + """Pick a judge that is not the model being judged, or None. A model scoring its own prose is a known bias, and the default judge is - itself in the evaluated set. Swapping in an alternate for its own family - costs nothing and removes the obvious conflict. + itself in the evaluated set. Matching is on the FAMILY, so a -fast row does + not get judged by its reasoning-on sibling either -- same weights. + + Returns None when every candidate shares the model's family, so the caller + skips the task. No sample beats a self-graded one, which is the same rule + score_judge already follows for a judge that returned nothing usable. """ from poller import parse_base_model_id - if parse_base_model_id(model_id) == parse_base_model_id(default_judge): - return ALTERNATE_JUDGE - return default_judge + family = parse_base_model_id(model_id) + for candidate in (default_judge, *ALTERNATE_JUDGES): + if parse_base_model_id(candidate) != family: + return candidate + return None def main() -> int: @@ -426,11 +470,11 @@ def main() -> int: print("nothing to run", file=sys.stderr) return 1 - judged = sum(1 for t in tasks if t["kind"] == "judge") + judge_tasks = sum(1 for t in tasks if t["kind"] == "judge") print( f"{len(identities)} models x {len(tasks)} tasks = " f"{len(identities) * len(tasks)} calls" - + (f" (+{len(identities) * judged} judge calls)" if judged else "") + + (f" (+{len(identities) * judge_tasks} judge calls)" if judge_tasks else "") ) if args.dry_run: for i in identities: @@ -482,10 +526,17 @@ def main() -> int: elif kind == "tool": score, detail = score_tool(tool_calls, task) elif kind == "judge": + judge_model = judge_for(model_id, args.judge_model) + if judge_model is None: + print( + f" {task['id']:30s} --- no judge outside " + f"{model_id}'s family, skipped" + ) + continue judged = score_judge( settings.base_url, api_key, - judge_for(model_id, args.judge_model), + judge_model, task, text, ) diff --git a/logs.py b/logs.py new file mode 100644 index 0000000..2ad317b --- /dev/null +++ b/logs.py @@ -0,0 +1,277 @@ +"""Structured logging for the dispatcher. + +The service had no logger. It had four ``print(..., file=sys.stderr)`` calls, +all of them failure paths, so a request that WORKED said nothing at all — not +the category, the tier, the candidates, the model chosen, the cost, or the +latency. Watching a live agent session showed only uvicorn's access line, which +does not even name the model that served it. + +Three things this module provides, in order of how much they matter: + +**A trace id per request.** One request touches classification, filtering, +ranking, dispatch, verification and possibly a retry, and until now nothing +tied those together. The id is carried in a ContextVar so helpers read it +without every signature growing a parameter, and it is printed on every line. + +**logfmt, not prose.** ``route id=r7f3a91 cat=coding_general pick=...`` reads +fine in ``journalctl`` and greps without a JSON parser. Prose reads better once +and aggregates never. + +**Real journald priorities.** systemd strips a ```` prefix off a log line and +records the message at that priority (``SyslogLevelPrefix`` defaults to true), +which is what makes ``journalctl -p warning`` mean something. Without it every +line lands at PRIORITY 6 and severity cannot be filtered at all. + +The prefix is emitted ONLY when ``$JOURNAL_STREAM`` is set, which systemd +exports when it owns our stderr and a terminal does not — so running uvicorn in +a foreground shell prints clean lines rather than a literal ``<6>`` on each one. +""" + +from __future__ import annotations + +import contextvars +import logging +import os +import secrets +import sys +from typing import Any, Optional, TextIO + +LOGGER_NAME = "llm_router" +LEVEL_ENV = "LLM_ROUTER_LOG_LEVEL" + +log = logging.getLogger(LOGGER_NAME) + +LEVELS = { + "debug": logging.DEBUG, + "info": logging.INFO, + "warning": logging.WARNING, + "error": logging.ERROR, +} + +# syslog priorities, which is what journald speaks. 5 (notice) and 1 (alert) +# are unused: nothing here is between info and warning, or above error. +PRIORITIES = { + logging.DEBUG: 7, + logging.INFO: 6, + logging.WARNING: 4, + logging.ERROR: 3, + logging.CRITICAL: 2, +} + +NO_TRACE = "-" + +_trace: contextvars.ContextVar[str] = contextvars.ContextVar("trace_id", default=NO_TRACE) + + +# --- trace ids ------------------------------------------------------------ + +def new_trace() -> str: + """Start a new request trace and return its id.""" + trace_id = "r" + secrets.token_hex(3) + _trace.set(trace_id) + return trace_id + + +def current_trace() -> str: + return _trace.get() + + +def set_trace(trace_id: str) -> None: + """Adopt an existing trace id in this context. + + Needed on the streaming path. A StreamingResponse's generator is iterated + from a different context than the endpoint that built it, so the ContextVar + set in the endpoint is NOT visible inside the generator — it must capture + the id as a local and re-set it here, or every streamed request logs no id + at all. That is the path all agent traffic takes. + """ + _trace.set(trace_id) + + +# --- rendering ------------------------------------------------------------ + +def render_value(value: Any) -> str: + """One logfmt value: unambiguous, single-line, cheap to read.""" + if value is None: + return "-" + if isinstance(value, bool): + return "1" if value else "0" + if isinstance(value, float): + # Six significant figures. Enough for microdollars and kWh in + # scientific notation, without 0.0007830000000000001. + return f"{value:.6g}" + text = str(value) + if text == "": + return '""' + if any(c in text for c in ' "\n\t='): + text = text.replace("\\", "\\\\").replace('"', '\\"') + text = text.replace("\n", " ").replace("\t", " ") + return f'"{text}"' + return text + + +def render(name: str, fields: dict[str, Any], trace_id: Optional[str] = None) -> str: + parts = [name, f"id={trace_id or current_trace()}"] + parts += [f"{key}={render_value(value)}" for key, value in fields.items()] + return " ".join(parts) + + +def _emit( + level: int, name: str, fields: dict[str, Any], trace_id: Optional[str] = None +) -> None: + """Emit one logfmt line, if this level is enabled. + + The level check comes first so DEBUG formatting costs nothing when DEBUG is + off — this runs on every request. + """ + if log.isEnabledFor(level): + log.log(level, "%s", render(name, fields, trace_id)) + + +def event(level: int, name: str, **fields: Any) -> None: + _emit(level, name, fields) + + +def debug(name: str, **fields: Any) -> None: + _emit(logging.DEBUG, name, fields) + + +def info(name: str, **fields: Any) -> None: + _emit(logging.INFO, name, fields) + + +def warning(name: str, **fields: Any) -> None: + _emit(logging.WARNING, name, fields) + + +def error(name: str, **fields: Any) -> None: + _emit(logging.ERROR, name, fields) + + +class Bound: + """An emitter pinned to one trace id, for code the ContextVar cannot reach. + + A StreamingResponse's generator is resumed by starlette through a + threadpool, and every ``next()`` runs in a FRESH COPY of the caller's + context — so a trace id set on one resumption is gone by the next, and the + ``finally`` block that logs the dispatch line sees nothing. Measured: the + streamed dispatch line logged ``id=-`` until this existed. + + Carrying the id explicitly is the only thing that survives that, and + streaming is the path all agent traffic takes. + """ + + __slots__ = ("trace_id",) + + def __init__(self, trace_id: str): + self.trace_id = trace_id + + def debug(self, name: str, **fields: Any) -> None: + _emit(logging.DEBUG, name, fields, self.trace_id) + + def info(self, name: str, **fields: Any) -> None: + _emit(logging.INFO, name, fields, self.trace_id) + + def warning(self, name: str, **fields: Any) -> None: + _emit(logging.WARNING, name, fields, self.trace_id) + + def error(self, name: str, **fields: Any) -> None: + _emit(logging.ERROR, name, fields, self.trace_id) + + +def bind(trace_id: Optional[str] = None) -> Bound: + """Pin an emitter to a trace id, defaulting to the current one.""" + return Bound(trace_id or current_trace()) + + +def enabled_for_debug() -> bool: + """For callers that must do real work to produce debug fields.""" + return log.isEnabledFor(logging.DEBUG) + + +# --- configuration -------------------------------------------------------- + +class PriorityFormatter(logging.Formatter): + """Plain message, optionally prefixed with a journald priority.""" + + def __init__(self, prefix: bool): + super().__init__("%(message)s") + self.prefix = prefix + + def format(self, record: logging.LogRecord) -> str: + message = super().format(record) + if not self.prefix: + return message + return f"<{PRIORITIES.get(record.levelno, 6)}>{message}" + + +def under_journald(stream: Optional[TextIO] = None) -> bool: + """Whether this stream really is the journal. + + systemd sets $JOURNAL_STREAM to the "device:inode" of the stream it + connected, and the variable is INHERITED by every child — including one + whose stderr has since been redirected to a file or a pipe. Presence alone + is therefore a false positive, and it fired immediately: a foreground + uvicorn started from a systemd-managed session, with its output redirected + to a file, wrote a literal <7> on every line. + + Comparing the value against the actual fd is what systemd documents, and it + is the only check that answers the real question. + """ + expected = os.environ.get("JOURNAL_STREAM") + if not expected: + return False + device, _, inode = expected.partition(":") + try: + stat = os.fstat((stream if stream is not None else sys.stderr).fileno()) + return stat.st_dev == int(device) and stat.st_ino == int(inode) + except (AttributeError, OSError, ValueError): + # No fileno at all (a StringIO under test), or an unparseable value. + return False + + +def resolve_level(configured: str) -> int: + """The env var wins, so a live service can be debugged without a repo edit. + + config.yaml is the documented home of the setting -- every knob belongs + there -- but flipping it means editing a tracked file and leaving a stray + diff. LLM_ROUTER_LOG_LEVEL goes in a systemd drop-in instead. + """ + override = os.environ.get(LEVEL_ENV) + if override: + level = LEVELS.get(override.strip().lower()) + if level is not None: + return level + print( + f"{LEVEL_ENV}={override!r} is not one of {sorted(LEVELS)}; " + f"falling back to logging.level={configured!r}", + file=sys.stderr, + ) + return LEVELS.get((configured or "info").strip().lower(), logging.INFO) + + +def configure( + level: str = "info", + *, + stream: Optional[TextIO] = None, + journald: Optional[bool] = None, +) -> logging.Logger: + """Install the one handler this logger gets. Idempotent. + + ``propagate`` is off: uvicorn configures its own loggers and a root handler + installed by anything else would print every line a second time. + """ + resolved = resolve_level(level) + log.setLevel(resolved) + log.propagate = False + for existing in list(log.handlers): + log.removeHandler(existing) + + target = stream if stream is not None else sys.stderr + handler = logging.StreamHandler(target) + handler.setLevel(resolved) + handler.setFormatter( + PriorityFormatter(under_journald(target) if journald is None else journald) + ) + log.addHandler(handler) + return log diff --git a/opencode.json b/opencode.json index 300f2a5..ff215e1 100644 --- a/opencode.json +++ b/opencode.json @@ -15,6 +15,12 @@ "limit": { "context": 782324, "output": 16384 + }, + "modalities": { + "input": [ + "text", + "image" + ] } }, "auto:batch": { @@ -22,6 +28,77 @@ "limit": { "context": 782324, "output": 16384 + }, + "modalities": { + "input": [ + "text", + "image" + ] + } + }, + "deepseek-v4-flash": { + "name": "Deepseek V4 Flash (pin)", + "limit": { + "context": 1048560, + "output": 16384 + }, + "modalities": { + "input": [ + "text", + "image" + ] + } + }, + "gemma-4-31b": { + "name": "Gemma 4 31B (pin)", + "limit": { + "context": 262128, + "output": 16384 + }, + "modalities": { + "input": [ + "text", + "image" + ] + } + }, + "kimi-k2.7-code": { + "name": "Kimi K2.7 Code (pin)", + "limit": { + "context": 262128, + "output": 16384 + }, + "modalities": { + "input": [ + "text", + "image" + ] + } + }, + "kimi-k3": { + "name": "Kimi K3 (pin)", + "limit": { + "context": 1048560, + "output": 16384 + }, + "modalities": { + "input": [ + "text", + "image" + ] + } + }, + "qwen3.6-35b": { + "name": "Qwen 3.6 35B (pin)", + "limit": { + "context": 131056, + "output": 16384 + }, + "modalities": { + "input": [ + "text", + "image" + ] } } } diff --git a/poller.py b/poller.py index b14a999..f82f99f 100644 --- a/poller.py +++ b/poller.py @@ -127,12 +127,34 @@ class ModelRow: deprecated: bool def effective_context_window(self, cfg: RouterConfig) -> Optional[int]: + """Usable context, after the safety factor and an output reserve. + + ``context.per_model_overrides`` wins where it is set, which is what it + is for: the global factor is a guess that has to hold for the whole + catalog, while a row someone has actually measured deserves its own + number. + + Compared with ``is not None`` rather than ``or``, so an override of 0 + reserve tokens means zero rather than silently falling through to the + default. Present-but-falsy is a trap this project's own eval set tests + models on; the router should not walk into it. + """ if not self.context_window: return None + override = cfg.context.per_model_overrides.get(self.model_id) + + factor = cfg.context.safety_factor + if override is not None and override.safety_factor is not None: + factor = override.safety_factor + # 11 of 19 catalog rows report no max_output_tokens, so the configured # reserve carries most of the catalog. - reserve = self.max_output_tokens or cfg.context.default_output_reserve_tokens - usable = int(self.context_window * cfg.context.safety_factor) - reserve + if override is not None and override.output_reserve_tokens is not None: + reserve = override.output_reserve_tokens + else: + reserve = self.max_output_tokens or cfg.context.default_output_reserve_tokens + + usable = int(self.context_window * factor) - reserve return max(usable, 0) diff --git a/proficiency_store.py b/proficiency_store.py index a33c966..179880c 100644 --- a/proficiency_store.py +++ b/proficiency_store.py @@ -23,13 +23,68 @@ def _now() -> str: return datetime.now(timezone.utc).isoformat() +def ensure_columns(conn: sqlite3.Connection) -> None: + """Add a column an older router.db predates. Idempotent, and cheap. + + schema.sql is CREATE TABLE IF NOT EXISTS, so it defines a NEW database and + silently does nothing to an existing one. Anything added later therefore + needs this, or the first write against a database created last week fails + with "no such column". + """ + columns = {row[1] for row in conn.execute("PRAGMA table_info(proficiency)")} + if "inherited_from" not in columns: + conn.execute("ALTER TABLE proficiency ADD COLUMN inherited_from TEXT") + conn.commit() + _backfill_inherited(conn) + + +def _backfill_inherited(conn: sqlite3.Connection) -> None: + """One-time: mark rows this harness could not have measured directly. + + ADD COLUMN gives every existing row NULL, which reads as "measured here" + and would leave exactly the rows this provenance was added for frozen + forever -- the migration would ship the fix and none of the repair. + + Provenance that was never recorded cannot be recovered in general. It can + be for the rows that matter, and not by guessing: ``eval_identities`` + selects ``latency_class='standard'`` rows, plus flex rows that have NO + standard equivalent. So a flex row WITH a standard equivalent was never a + candidate for direct evaluation, whatever its sample count says. That is + the harness's own selection rule read backwards. + + Anything else keeps NULL, which is the safe direction: it means "do not + overwrite", so a real measurement is never lost to this. + """ + pairs = conn.execute( + """ + SELECT v.model_id, v.provider, s.model_id + FROM models v + JOIN models s + ON s.base_model_id = v.base_model_id + AND s.provider = v.provider + AND s.reasoning_mode = v.reasoning_mode + AND s.context_variant = v.context_variant + WHERE v.latency_class = 'flex' AND s.latency_class = 'standard' + """ + ).fetchall() + for variant_id, provider, source_id in pairs: + conn.execute( + """ + UPDATE proficiency SET inherited_from = ? + WHERE model_id = ? AND provider = ? AND inherited_from IS NULL + """, + (source_id, variant_id, provider), + ) + conn.commit() + + def _read_row( conn: sqlite3.Connection, model_id: str, provider: str, category: str ) -> Optional[sqlite3.Row]: conn.row_factory = sqlite3.Row return conn.execute( """ - SELECT leaderboard_score, self_eval_score, self_eval_samples + SELECT leaderboard_score, self_eval_score, self_eval_samples, inherited_from FROM proficiency WHERE model_id = ? AND provider = ? AND category = ? """, @@ -46,7 +101,9 @@ def _write( leaderboard_score: Optional[float], self_eval_score: Optional[float], self_eval_samples: int, + inherited_from: Optional[str] = None, ) -> None: + ensure_columns(conn) blended, source = blend( leaderboard_score, self_eval_score, @@ -59,14 +116,16 @@ def _write( """ INSERT INTO proficiency ( model_id, provider, category, leaderboard_score, - self_eval_score, self_eval_samples, blended_score, source, last_updated - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + self_eval_score, self_eval_samples, blended_score, source, + inherited_from, last_updated + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(model_id, provider, category) DO UPDATE SET leaderboard_score = excluded.leaderboard_score, self_eval_score = excluded.self_eval_score, self_eval_samples = excluded.self_eval_samples, blended_score = excluded.blended_score, source = excluded.source, + inherited_from = excluded.inherited_from, last_updated = excluded.last_updated """, ( @@ -78,6 +137,7 @@ def _write( self_eval_samples, blended, source, + inherited_from, _now(), ), ) @@ -102,6 +162,8 @@ def set_leaderboard( score, existing["self_eval_score"] if existing else None, existing["self_eval_samples"] if existing else 0, + # A prior says nothing about where the self-eval half came from. + existing["inherited_from"] if existing else None, ) @@ -120,6 +182,14 @@ def add_self_eval( ``self_eval_samples`` keeps meaning "how much evidence stands behind this", which is what the blending threshold is gating on. """ + # Clamped at the one place every write passes through, which is what this + # module exists to be. score_judge already bounded its output; score_code + # did not, and a harness bug let a model's own demo output push a two-check + # task to 1.50. A blended_score above 1.0 raises `best` in + # rank_candidates and shifts every other candidate's quality band, so the + # damage is not confined to the row that carries the bad number. + scores = [min(1.0, max(0.0, s)) for s in scores] + existing = _read_row(conn, model_id, provider, category) prev_score = existing["self_eval_score"] if existing else None prev_samples = existing["self_eval_samples"] if existing else 0 @@ -150,8 +220,15 @@ def propagate_to_variants( alone. Inheriting across those would attribute reasoning-on quality to a reasoning-off row. - Anything already measured directly keeps its own scores; inheritance is - the fallback, never an overwrite. + Anything measured DIRECTLY keeps its own scores; inheritance is the + fallback, never an overwrite. A row that was inherited before is refreshed, + which needs ``inherited_from`` to tell the two apart -- both carry + self_eval_samples > 0, so the old "samples > 0 means leave it alone" test + could not distinguish them and froze every variant at its first + inheritance. Observed: kimi-k3 moved from 0.85 (n=2) to 0.957 (n=7) while + kimi-k3-flex sat at 0.85 with a week-old timestamp, and the run reported + "propagated 0 inherited rows". Flex rows serve `auto:batch`, so those + requests were ranking on scores their family had left behind. Returns the number of rows written. """ @@ -184,8 +261,13 @@ def propagate_to_variants( for variant in variants: for row in source_rows: existing = _read_row(conn, variant["model_id"], provider, row["category"]) - if existing and (existing["self_eval_samples"] or 0) > 0: - continue # measured directly; do not overwrite with the family's + measured_here = ( + existing is not None + and (existing["self_eval_samples"] or 0) > 0 + and not existing["inherited_from"] + ) + if measured_here: + continue # its own measurement outranks the family's _write( conn, cfg, @@ -195,6 +277,7 @@ def propagate_to_variants( row["leaderboard_score"], row["self_eval_score"], row["self_eval_samples"], + source_model_id, ) written += 1 return written diff --git a/routing.py b/routing.py index 10b8396..02c3f40 100644 --- a/routing.py +++ b/routing.py @@ -10,8 +10,10 @@ Two stages, in order: that cannot hold the context, is below the required tier, is stale, or is not actually reachable by this account is not a cost tradeoff to be weighed; it is not a candidate at all (design doc §3.2, §4). -2. **Weighted scoring** (``rank_candidates``) — order what survives by - ``w_cost*cost + w_eco*eco + w_prof*proficiency``. +2. **Ranking** (``rank_candidates``) — order what survives by quality, with + cost breaking ties inside ``quality_tolerance``. NOT a weighted blend: that + was measured to be nearly inert, since moving the cost weight from 0.4 to + zero changed the winner in only 2 of 6 categories. The flex filter is the one that is easy to get wrong. NeuralWatt's ``-flex`` rows are the same weights at the same advertised price, so on every scored @@ -31,7 +33,7 @@ INTERACTIVE = "interactive" BATCH = "batch" -def is_eligible( +def rejection_reason( row: dict, *, required_context_tokens: int, @@ -40,35 +42,103 @@ def is_eligible( allowed_access_levels: Sequence[str], exclude_stale: bool, exclude_deprecated: bool, -) -> bool: - """Whether a model row survives every hard filter. + min_tool_proficiency: float | None = None, + require_vision: bool = False, + require_json_mode: bool = False, +) -> str | None: + """Why this row is not a candidate, or None if it is one. + + Phrased as the reason rather than a bool so the debug log can say WHICH + filter dropped a model and with what numbers, without a second + implementation of these rules drifting away from this one. + Reasons are single tokens so a log line needs no quoting. A NULL ``effective_context_window`` fails the context filter: an unknown window cannot be shown to be large enough, and silently truncating mid-task is worse than routing elsewhere. + + ``min_tool_proficiency`` is set when the REQUEST carries tool definitions, + and is a capability requirement rather than a preference — which is why it + is a filter and not a term in the ranking. + + The distinction that matters: this does not ask whether the task is + "agentic". It asks whether the model can be trusted with tools that are + on the table. Those are different questions, and the measured failure is + the second one — `deepseek-v4-flash` scores 0.33 here, and the recorded + case is a *non*-agentic prompt ("it is 1:20pm, my meeting is at 3pm, how + many minutes?") where it called two tools instead of subtracting. A model + that over-reaches for tools is a hazard on every request where tools + exist, not only on the ones a classifier would label agentic. + + That also sidesteps the classifier entirely, which is the point: asked to + identify six unambiguous tool-use prompts, the local models managed 2/6 + and 1/6. Whether tools are present is stated in the request body. Reading + it is exact and free; inferring it is neither. """ eff_ctx = row.get("effective_context_window") - if eff_ctx is None or eff_ctx < required_context_tokens: - return False + if eff_ctx is None: + return "context(unknown)" + if eff_ctx < required_context_tokens: + return f"context({eff_ctx}<{required_context_tokens})" tier = row.get("tier") - if tier is None or tier < required_tier: - return False + if tier is None: + return "tier(unknown)" + if tier < required_tier: + return f"tier({tier}<{required_tier})" availability = row.get("availability") if exclude_stale and availability == "stale": - return False + return "stale" if exclude_deprecated and (availability == "deprecated" or row.get("deprecated")): - return False + return "deprecated" - if row.get("access_level", "public") not in allowed_access_levels: - return False + access_level = row.get("access_level", "public") + if access_level not in allowed_access_levels: + return f"access_level({access_level})" # Flex rows may be queued behind a capacity gap; only batch work opts in. if latency_tolerance == INTERACTIVE and row.get("latency_class") == "flex": - return False + return "latency_class(flex)" - return True + # Capability flags fail CLOSED on unknown, deliberately unlike the tool + # gate just below. A proficiency MEASUREMENT with no evidence is "unproven, + # not bad" -- admitting it loses nothing. But a capability FLAG that is + # absent means "cannot confirm the capability", and routing a request that + # needs vision or JSON mode to a model that might lack it is a guaranteed + # provider 400. Same precedent as the context(unknown) gate above. + if require_vision: + vision = row.get("supports_vision") + if vision is None: + return "vision(unknown)" + if not vision: + return "vision(unsupported)" + if require_json_mode: + jm = row.get("supports_json_mode") + if jm is None: + return "json_mode(unknown)" + if not jm: + return "json_mode(unsupported)" + + if min_tool_proficiency is not None: + tool_score = row.get("tool_proficiency") + # Absent evidence does not disqualify, consistently with the tier-1 + # context gate: a model nobody has evaluated for tool use yet is + # unproven, not proven bad. Only a measured score below the bar drops + # a candidate. + if tool_score is not None and tool_score < min_tool_proficiency: + return f"tool_proficiency({tool_score:g}<{min_tool_proficiency:g})" + + return None + + +def is_eligible(row: dict, **filters) -> bool: + """Whether a model row survives every hard filter. + + A thin wrapper so there is exactly one copy of the rules; see + ``rejection_reason`` for what they are and why. + """ + return rejection_reason(row, **filters) is None def select_candidates( @@ -80,6 +150,9 @@ def select_candidates( allowed_access_levels: Sequence[str], exclude_stale: bool, exclude_deprecated: bool, + min_tool_proficiency: float | None = None, + require_vision: bool = False, + require_json_mode: bool = False, ) -> list[dict]: """Apply every hard filter, preserving input order.""" return [ @@ -93,6 +166,9 @@ def select_candidates( allowed_access_levels=allowed_access_levels, exclude_stale=exclude_stale, exclude_deprecated=exclude_deprecated, + min_tool_proficiency=min_tool_proficiency, + require_vision=require_vision, + require_json_mode=require_json_mode, ) ] @@ -222,8 +298,14 @@ def rank_candidates( # while a real gap (tool_use_agentic spans 0.67) still decides outright. best = max(r["proficiency_score"] for r in ranked) - def band(r: dict) -> int: - return int((best - r["proficiency_score"]) / quality_tolerance) + def band(r: dict) -> float: + # 0 means "no band": rank strictly by quality, with cost breaking only + # exact ties. A legitimate setting -- "never trade quality for cost" -- + # and one the config validator has always accepted, while this line + # divided by it. + if quality_tolerance <= 0: + return -r["proficiency_score"] + return float(int((best - r["proficiency_score"]) / quality_tolerance)) ranked.sort( key=lambda r: ( diff --git a/schema.sql b/schema.sql index 375d8f9..b9c9a74 100644 --- a/schema.sql +++ b/schema.sql @@ -66,6 +66,13 @@ CREATE TABLE IF NOT EXISTS proficiency ( self_eval_samples INTEGER DEFAULT 0, blended_score REAL, -- computed: see blending rule in design doc source TEXT, -- 'leaderboard' | 'self_eval' | 'blended' + -- Which model this row's scores were COPIED from, or NULL if they were + -- measured on this row directly. Provenance, not decoration: without it + -- an inherited row is indistinguishable from a measured one (both carry + -- self_eval_samples > 0), so propagate_to_variants could not tell which + -- rows it was allowed to refresh and froze every variant permanently at + -- its first inheritance. + inherited_from TEXT, last_updated TEXT NOT NULL, PRIMARY KEY (model_id, provider, category), FOREIGN KEY (model_id, provider) REFERENCES models (model_id, provider) @@ -95,23 +102,32 @@ CREATE TABLE IF NOT EXISTS energy_observations ( task_category TEXT, prompt_tokens INTEGER, completion_tokens INTEGER, - -- energy_kwh is what NeuralWatt BILLS, but it is not a measure of model - -- efficiency: it equals avg_power_watts * duration_seconds * - -- attribution_ratio, where attribution_ratio is this request's share of a - -- shared multi-tenant GPU pool. Eight identical calls to one model inside - -- one minute varied 20x in energy_kwh and 22x in attribution_ratio, - -- correlation +0.997, while duration and power held steady. Ranking models - -- on this ranks how busy the provider was, not how efficient the model is. - energy_kwh REAL, -- response.energy.energy_kwh (attributed, noisy) + -- energy_kwh is what NeuralWatt BILLS. It equals avg_power_watts * + -- duration_seconds * attribution_ratio, where attribution_ratio is this + -- request's share of a shared multi-tenant GPU pool. Up close that term + -- looks like noise: eight identical calls to one model inside one minute + -- varied 20x, correlating +0.997 with the ratio while power and duration + -- held steady. + -- + -- It is not noise, and this is the ATTRIBUTED figure scoring reads. Across + -- the reference sweep the median ratio spans 750x BETWEEN models while the + -- typical spread WITHIN one is 1.8x, and the values are quantized (0.001, + -- 0.25, 0.5, 0.75) -- that is serving concurrency, a stable per-model + -- property. A model whose GPUs carry more concurrent requests genuinely + -- costs less per request. The median over repeated sweeps absorbs what + -- noise remains. + energy_kwh REAL, -- response.energy.energy_kwh (attributed) energy_btu REAL, -- energy_kwh * 3412.14, purely for comedic dashboard value -- The pre-attribution terms. avg_power_watts * duration_seconds is the -- pool's energy over the request, independent of how many other tenants - -- shared it, and it is ~10x more stable (1.85x spread vs 20x on the same - -- eight calls). That product is what scoring ranks on. + -- shared it. Scoring on that product was TRIED and is wrong: it discards + -- the 750x between-model signal above to suppress a 1.8x within-model one. + -- Kept as a diagnostic (dispatcher.gross_energy_kwh) so the decomposition + -- stays inspectable, not as a scoring input. avg_power_watts REAL, -- response.energy.avg_power_watts duration_seconds REAL, -- response.energy.duration_seconds - attribution_ratio REAL, -- kept so the noise term stays inspectable + attribution_ratio REAL, -- the share term, kept so the identity checks out -- Carbon is what design doc §4 actually scores eco on, and NeuralWatt -- reports it per-request rather than making us derive it. Grid intensity diff --git a/scoring.py b/scoring.py index 0210b1b..2b185aa 100644 --- a/scoring.py +++ b/scoring.py @@ -4,8 +4,18 @@ These functions are deliberately free of I/O: no DB reads, no file reads, no config import at runtime. Thresholds and weights are passed as arguments so the module stays testable and reusable. -Scoring model (design doc §4): - composite = w_cost*cost_score + w_eco*eco_score + w_prof*proficiency_score +What is still live here: ``proficiency_score``, which is the objective +``routing.rank_candidates`` orders on, and ``cost_score``, which it reports for +visibility without ranking on it. + +The weighted composite below is NOT the scoring model any more. It was +``w_cost*cost + w_eco*eco + w_prof*proficiency`` until measurement retired it: +moving the cost weight from 0.4 to zero changed the winner in only 2 of 6 +categories, so 40% of every decision was steering nothing, while eco was +optimizing a goal nobody had stated. Ranking is now quality first, cost as the +tiebreak inside ``quality_tolerance``, and a hard energy ceiling. ``eco_score`` +and ``composite_score`` survive as arithmetic with tests; nothing in the +router calls them. None-handling rule (consistent across cost/eco): candidates without data are excluded from the min/max range computation and receive a neutral 0.5. diff --git a/seed_energy.py b/seed_energy.py index e8b69c2..d3254ad 100644 --- a/seed_energy.py +++ b/seed_energy.py @@ -19,10 +19,24 @@ calls to one model inside one minute spanned 20x, correlating +0.997 with the attribution ratio while power and duration held steady. Two earlier sweeps of these same 13 models disagreed by up to 36x on that basis. -So routing scores on `power * duration` (see `dispatcher.gross_energy_kwh`), -which held to 1.85x across those same eight calls. The summary prints both -spreads side by side; if the billed column is wildly wider than the gross -column, that is the artifact, not the models. +Scoring reads the ATTRIBUTED figures, not `power * duration`. Ranking on the +pre-attribution product was tried and is wrong: the median attribution ratio +spans 750x between models against a 1.8x spread within one, so stripping it +discards the larger real signal to suppress the smaller noisy one. The summary +still prints both spreads side by side, because the gap between them is what +that reasoning rests on. + +What this sweep feeds today is `eco` and the per-request energy ceiling +(objective.max_energy_per_request). It no longer feeds `cost`: cost is priced +per request from catalog prices scaled to the request's shape, because a fixed +400-token workload ranks models backwards for real traffic. + +Coverage across TIME is the point of running it repeatedly. Attribution tracks +pool load and pool load tracks time of day -- between two sweeps hours apart, +deepseek-v4-flash moved ~50x and qwen3.6-35b ~7x the other way, enough to +invert their ranking. More samples inside one sweep measures one moment more +precisely; `load_candidates` takes the median over ALL seed_reference rows, so +repeated sweeps accumulate into a median across time for free. What lands in the table are real observations of real calls, identical in kind to what the dispatcher logs; they are simply generated deliberately @@ -77,14 +91,28 @@ def routable_models(conn: sqlite3.Connection, allowed_levels: list[str]) -> list return [dict(r) for r in rows] -def sample_once(base_url: str, api_key: str, model_id: str, timeout: int = 300) -> dict: +def sample_once( + base_url: str, + api_key: str, + model_id: str, + max_tokens: int = REFERENCE_MAX_TOKENS, + timeout: int = 300, +) -> dict: + """One reference call. ``max_tokens`` is a parameter because --max-tokens is. + + It used to be hardcoded while the banner printed whatever --max-tokens had + been passed, so the flag moved the report and not the request -- and the + resulting rows landed in the same seed_reference median as the 400-token + ones, quietly mixing two workload shapes in the one axis that exists to + hold the workload constant across models. + """ resp = requests.post( f"{base_url}/chat/completions", headers={"authorization": f"Bearer {api_key}"}, json={ "model": model_id, "messages": [{"role": "user", "content": REFERENCE_PROMPT}], - "max_tokens": REFERENCE_MAX_TOKENS, + "max_tokens": max_tokens, "temperature": 0, }, timeout=timeout, @@ -140,20 +168,29 @@ def main() -> int: results[model_id] = [] for i in range(args.samples): try: - payload = sample_once(settings.base_url, api_key, model_id) + payload = sample_once( + settings.base_url, api_key, model_id, args.max_tokens + ) except requests.RequestException as e: print(f" {model_id:26s} sample {i + 1}: FAILED {type(e).__name__}: {e}") continue usage = payload.get("usage") or {} telemetry = extract_telemetry(payload) + # Keyword arguments deliberately. log_observation grew request_id, + # session_key and session_dir in the middle of its signature and + # made the trailing three keyword-only; this call still passed six + # positionals, so every sweep died on TypeError after its first + # BILLED call -- and TypeError is not a RequestException, so the + # `except` below never caught it. log_observation( model_id, m["provider"], SEED_CATEGORY, - usage.get("prompt_tokens"), - usage.get("completion_tokens"), - telemetry, + payload.get("id"), + prompt_tokens=usage.get("prompt_tokens"), + completion_tokens=usage.get("completion_tokens"), + telemetry=telemetry, ) if telemetry.allowance_remaining_usd is not None: if allowance_start is None: diff --git a/tests/test_capabilities.py b/tests/test_capabilities.py new file mode 100644 index 0000000..e1ce860 --- /dev/null +++ b/tests/test_capabilities.py @@ -0,0 +1,84 @@ +"""Tests for capabilities.py — request-side capability detection. + +Each case proves one detection fires and that the others stay False, mirroring +the "rejects and does not over-reject" shape of test_routing.py. +""" + +from capabilities import detect_capabilities + + +def test_no_parts_yields_no_capabilities(): + caps = detect_capabilities({"messages": [{"content": "plain text"}]}) + assert caps == detect_capabilities({}) + assert not caps.has_images + assert not caps.require_json_mode + assert not caps.tools_present + assert not caps.has_reasoning_request + + +def test_image_url_part_sets_has_images(): + body = { + "messages": [ + {"content": [{"type": "image_url", "url": "https://x/img.png"}]} + ] + } + assert detect_capabilities(body).has_images is True + + +def test_image_in_an_earlier_message_counts(): + body = { + "messages": [ + {"content": [{"type": "image_url", "url": "https://x/1.png"}]}, + {"content": "now what do you make of it?"}, + ] + } + assert detect_capabilities(body).has_images is True + + +def test_non_image_parts_do_not_trigger(): + body = { + "messages": [ + { + "content": [ + {"type": "text", "text": "hello"}, + {"type": "tool_use", "id": "t1"}, + ] + } + ] + } + assert detect_capabilities(body).has_images is False + + +def test_string_content_does_not_trigger_images(): + # A raw base64 string in a text part is not an image_url part. + body = {"messages": [{"content": "data:image/png;base64,AA=="}]} + assert detect_capabilities(body).has_images is False + + +def test_json_object_response_format_sets_require_json_mode(): + body = {"response_format": {"type": "json_object"}} + assert detect_capabilities(body).require_json_mode is True + + +def test_json_schema_response_format_sets_require_json_mode(): + body = {"response_format": {"type": "json_schema", "json_schema": {}}} + assert detect_capabilities(body).require_json_mode is True + + +def test_text_or_missing_response_format_does_not_require_json_mode(): + assert detect_capabilities({}).require_json_mode is False + assert detect_capabilities({"response_format": {"type": "text"}}).require_json_mode is False + + +def test_tools_array_sets_tools_present(): + assert detect_capabilities({"tools": [{"type": "function"}]}).tools_present is True + # An empty array is no tools at all. + assert detect_capabilities({"tools": []}).tools_present is False + + +def test_reasoning_effort_sets_has_reasoning_request(): + assert detect_capabilities({"reasoning_effort": "high"}).has_reasoning_request is True + + +def test_reasoning_param_sets_has_reasoning_request(): + assert detect_capabilities({"reasoning": {"enabled": True}}).has_reasoning_request is True diff --git a/tests/test_chat_completions.py b/tests/test_chat_completions.py new file mode 100644 index 0000000..dd702df --- /dev/null +++ b/tests/test_chat_completions.py @@ -0,0 +1,807 @@ +"""Tests for the OpenAI-compatible surface — the endpoint every client uses. + +This file exists because it did not. `/v1/chat/completions` is how opencode, +an SDK and plain curl all reach the router, and it had zero tests, which is +how a NameError survived on the pass-through path: `alternatives` read +`decision.runners_up`, but `decision` is only bound inside `if wants_routing`, +so every non-streaming request naming a real model id returned 500 before the +provider was ever called. + +Nothing here touches the network. The classifier, the provider and the local +verifier are all stubbed, so the tests run in milliseconds and pin behaviour +rather than reachability. +""" + +import io +import json +import sqlite3 +from pathlib import Path + +import pytest +from starlette.testclient import TestClient + +import dispatcher +from dispatcher import Classification, app + +ROOT = Path(__file__).resolve().parent.parent +SCHEMA_SQL = (ROOT / "schema.sql").read_text() + +CHEAP = "cheap-model" +DEAR = "dear-model" + + +class FakeResponse: + """Just enough of requests.Response for both dispatcher paths.""" + + def __init__(self, payload=None, *, status_code=200, lines=None): + self.status_code = status_code + self._payload = payload or {} + self._lines = lines or [] + self.text = json.dumps(self._payload) + self.closed = False + + def json(self): + return self._payload + + def iter_lines(self, decode_unicode=False): + yield from self._lines + + def close(self): + self.closed = True + + +def completion(model, content="hello there", *, completion_tokens=12): + """A provider response shaped like NeuralWatt's, telemetry blocks included.""" + return { + "id": "chatcmpl-test-1", + "model": model, + "choices": [ + {"message": {"role": "assistant", "content": content}, + "finish_reason": "stop"} + ], + "usage": {"prompt_tokens": 31, "completion_tokens": completion_tokens}, + "energy": {"energy_kwh": 5.0e-05, "avg_power_watts": 400.0, + "duration_seconds": 1.4, "attribution_ratio": 0.25, + "carbon_g_co2eq": 2.4e-03, "carbon_source": "agent_cache", + "grid_id": "FI"}, + "cost": {"request_cost_usd": 4.0e-04}, + } + + +@pytest.fixture +def router(tmp_path, monkeypatch): + """The dispatcher pointed at a throwaway catalog, with nothing dialled out.""" + db_path = tmp_path / "test.db" + conn = sqlite3.connect(db_path) + conn.executescript(SCHEMA_SQL) + for model_id, completion_price, vision in ( + (CHEAP, 0.30, 1), + (DEAR, 9.00, 0), + ): + conn.execute( + """ + INSERT INTO models ( + model_id, provider, base_model_id, tier, context_window, + effective_context_window, max_output_tokens, + cost_per_1m_prompt, cost_per_1m_completion, + supports_vision, supports_json_mode, + latency_class, reasoning_mode, context_variant, + access_level, availability, last_updated + ) VALUES (?, 'neuralwatt', ?, 2, 262128, 192500, 16384, ?, ?, + ?, 1, 'standard', 'default', 'full', 'public', 'active', + '2026-08-22T00:00:00+00:00') + """, + (model_id, model_id, completion_price / 3, completion_price, vision), + ) + conn.commit() + conn.close() + + monkeypatch.setattr(dispatcher.cfg.database, "path", str(db_path)) + # No Ollama in the test environment, and the local check would otherwise + # fire as a background task and try to reach localhost:11434. + monkeypatch.setattr(dispatcher.cfg.verification, "local_llm_enabled", False) + # The local vision fallback is off unless a test opts in; without this it + # would fire for every image request and try to reach localhost. + monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", False) + monkeypatch.setenv("NEURALWATT_API_KEY", "test-key") + + calls = [] + + def fake_post(url, headers=None, json=None, stream=False, timeout=None): + calls.append({"url": url, "body": json, "stream": stream}) + if stream: + return FakeResponse(lines=STREAM_LINES) + return FakeResponse(completion(json["model"])) + + monkeypatch.setattr(dispatcher.requests, "post", fake_post) + monkeypatch.setattr( + dispatcher, "classify", + lambda task, context: Classification( + task_category="coding_general", task_tier=2, + required_context_tokens=100, confidence=0.9, + ), + ) + yield TestClient(app), calls, db_path + + +STREAM_LINES = [ + 'data: {"id":"chatcmpl-stream-1","choices":[{"delta":{"content":"hel"}}]}', + "", + 'data: {"id":"chatcmpl-stream-1","choices":[{"delta":{"content":"lo"},' + '"finish_reason":"stop"}],"usage":{"prompt_tokens":31,' + '"completion_tokens":9}}', + "", + ': energy {"energy_kwh": 5e-05, "carbon_g_co2eq": 0.0024, ' + '"carbon_source": "agent_cache"}', + ': cost {"request_cost_usd": 0.0004}', + "data: [DONE]", + "", +] + + +def _messages(text="write me a function"): + return [{"role": "user", "content": text}] + + +# --- the regression ------------------------------------------------------- + +def test_a_real_model_id_is_dispatched_rather_than_routed(router): + """The documented pass-through: 'any real model id — dispatched as asked'. + + This raised NameError on `decision` before the provider was ever called, + so the whole path 500'd while streaming clients never noticed. + """ + client, calls, _ = router + resp = client.post( + "/v1/chat/completions", + json={"model": DEAR, "messages": _messages()}, + ) + + assert resp.status_code == 200 + assert calls[0]["body"]["model"] == DEAR, "the caller's choice must survive" + assert resp.headers["X-Router-Model"] == DEAR + + +def test_a_real_model_id_does_not_pay_for_classification(router, monkeypatch): + """A named model has nothing to classify; the ~2s round-trip is skipped.""" + client, _, _ = router + + def explode(task, context): + raise AssertionError("classifier consulted for an explicit model id") + + monkeypatch.setattr(dispatcher, "classify", explode) + assert client.post( + "/v1/chat/completions", json={"model": CHEAP, "messages": _messages()} + ).status_code == 200 + + +# --- routing -------------------------------------------------------------- + +def test_auto_routes_and_reports_the_model_it_actually_used(router): + """Quality ties, so cost breaks it — and the client is told what ran.""" + client, calls, _ = router + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _messages()}, + ) + + assert resp.status_code == 200 + assert calls[0]["body"]["model"] == CHEAP + assert resp.headers["X-Router-Model"] == CHEAP + # The body must stay a valid OpenAI response naming the real model, not 'auto'. + assert resp.json()["model"] == CHEAP + + +def test_a_provider_prefixed_router_name_still_routes(router): + """opencode sends `llm-router/auto`; only the virtual names are stripped.""" + client, calls, _ = router + resp = client.post( + "/v1/chat/completions", + json={"model": "llm-router/auto", "messages": _messages()}, + ) + assert resp.status_code == 200 + assert calls[0]["body"]["model"] == CHEAP + + +def test_no_eligible_model_is_a_422_naming_the_filters(router, monkeypatch): + """A dead end must say which constraint killed it, not just 'no model'.""" + client, calls, _ = router + monkeypatch.setattr( + dispatcher, "classify", + lambda task, context: Classification( + task_category="coding_general", task_tier=3, + required_context_tokens=100, confidence=0.9, + ), + ) + resp = client.post( + "/v1/chat/completions", json={"model": "auto", "messages": _messages()} + ) + + assert resp.status_code == 422 + assert "tier >= 3" in resp.json()["detail"] + assert not calls, "nothing should be dispatched when nothing qualifies" + + +# --- streaming ------------------------------------------------------------ + +def test_a_stream_is_proxied_verbatim_including_the_telemetry_comments(router): + """NeuralWatt's energy/cost blocks are SSE comments; clients ignore them. + + They must still reach the client untouched — the router reads them on the + way past rather than buffering the stream to strip them. + """ + client, calls, _ = router + resp = client.post( + "/v1/chat/completions", + json={"model": DEAR, "messages": _messages(), "stream": True}, + ) + + assert resp.status_code == 200 + assert calls[0]["stream"] is True + body = resp.text + assert "hel" in body and "lo" in body + assert ': energy {"energy_kwh": 5e-05' in body + assert "data: [DONE]" in body + + +def test_a_streamed_call_still_logs_its_energy(router): + """Streaming is how every agent client talks; unlogged, it is most of the traffic.""" + client, _, db_path = router + client.post( + "/v1/chat/completions", + json={"model": DEAR, "messages": _messages(), "stream": True}, + ) + + conn = sqlite3.connect(db_path) + conn.row_factory = sqlite3.Row + row = conn.execute( + "SELECT model_id, request_id, energy_kwh, cost_usd, completion_tokens " + "FROM energy_observations ORDER BY id DESC LIMIT 1" + ).fetchone() + conn.close() + + assert row["model_id"] == DEAR + assert row["request_id"] == "chatcmpl-stream-1" + assert row["energy_kwh"] == pytest.approx(5e-05) + assert row["cost_usd"] == pytest.approx(4e-04) + assert row["completion_tokens"] == 9 + + +def test_a_dropped_upstream_connection_ends_the_stream_cleanly(router, monkeypatch): + """The upstream connection can die mid-stream (NeuralWatt closing early, a + network blip). Left uncaught, requests.exceptions.ChunkedEncodingError + propagated straight out of the generator, which Starlette surfaced as an + unhandled ASGI exception -- a full traceback in the log, and the client's + connection cut dead with no error payload or [DONE]. + """ + import requests as requests_module + + client, _, _ = router + + def broken_lines(): + yield 'data: {"id":"chatcmpl-stream-1","choices":[{"delta":{"content":"hel"}}]}' + raise requests_module.exceptions.ChunkedEncodingError("Response ended prematurely") + + def fake_post(url, headers=None, json=None, stream=False, timeout=None): + return FakeResponse(lines=broken_lines()) + + monkeypatch.setattr(dispatcher.requests, "post", fake_post) + + resp = client.post( + "/v1/chat/completions", + json={"model": DEAR, "messages": _messages(), "stream": True}, + ) + + assert resp.status_code == 200 + body = resp.text + assert "hel" in body + assert "upstream stream interrupted" in body + assert "data: [DONE]" in body + + +# --- what a request leaves in the journal --------------------------------- +# +# The reason this exists: watching a live session showed only uvicorn's access +# line, which does not even name the model that served it. + +@pytest.fixture +def logbuf(): + """Capture what would reach the journal, through the real formatter.""" + import logs + + buf = io.StringIO() + logs.configure("debug", stream=buf, journald=False) + yield buf + for handler in list(logs.log.handlers): + logs.log.removeHandler(handler) + + +def _lines(logbuf, event): + return [ln for ln in logbuf.getvalue().splitlines() if ln.startswith(f"{event} ")] + + +def _fields(line): + out = {} + for token in line.split(" ")[1:]: + if "=" in token: + key, _, value = token.partition("=") + out[key] = value + return out + + +def test_a_routed_request_logs_exactly_one_decision_line(router, logbuf): + client, _, _ = router + client.post("/v1/chat/completions", + json={"model": "auto", "messages": _messages()}) + + routes = _lines(logbuf, "route") + + assert len(routes) == 1, "routing twice for context must not log twice" + fields = _fields(routes[0]) + assert fields["pick"] == CHEAP + assert fields["cat"] == "coding_general" + assert fields["tier"] == "2" + assert int(fields["ms"]) >= 0 + + +def test_the_dispatch_line_carries_the_join_key(router, logbuf): + """rid is what pivots a journal line to its energy_observations row.""" + client, _, _ = router + client.post("/v1/chat/completions", + json={"model": "auto", "messages": _messages()}) + + fields = _fields(_lines(logbuf, "dispatch")[0]) + + assert fields["rid"] == "chatcmpl-test-1" + assert fields["model"] == CHEAP + assert fields["c_tok"] == "12" + + +def test_every_line_of_one_request_shares_a_trace_id(router, logbuf): + client, _, _ = router + client.post("/v1/chat/completions", + json={"model": "auto", "messages": _messages()}) + + ids = {_fields(ln)["id"] for ln in logbuf.getvalue().splitlines()} + + assert len(ids) == 1 + assert ids != {"-"} + + +def test_a_streamed_request_still_has_a_trace_id(router, logbuf): + """The trap: a StreamingResponse generator runs in a different context. + + Read from the ContextVar inside the generator it comes back empty, and + every streamed request -- which is all agent traffic -- logs a blank id. + """ + client, _, _ = router + client.post("/v1/chat/completions", + json={"model": DEAR, "messages": _messages(), "stream": True}) + + dispatched = _fields(_lines(logbuf, "dispatch")[0]) + + assert dispatched["id"] != "-" + assert dispatched["stream"] == "1" + assert dispatched["rid"] == "chatcmpl-stream-1" + + +def test_debug_says_which_filter_dropped_a_model(router, logbuf): + """'No model satisfies the hard filters' is otherwise a dead end.""" + client, _, db_path = router + conn = sqlite3.connect(db_path) + conn.execute( + """ + INSERT INTO models ( + model_id, provider, tier, context_window, effective_context_window, + latency_class, access_level, availability, last_updated + ) VALUES ('held-model', 'neuralwatt', 2, 262128, 192500, 'flex', + 'public', 'active', '2026-08-22T00:00:00+00:00') + """ + ) + conn.commit() + conn.close() + + client.post("/v1/chat/completions", + json={"model": "auto", "messages": _messages()}) + + dropped = {_fields(ln)["model"]: _fields(ln)["reason"] + for ln in _lines(logbuf, "filter")} + + assert dropped["held-model"] == "latency_class(flex)" + + +def test_a_pass_through_is_labelled_as_one(router, logbuf): + client, _, _ = router + client.post("/v1/chat/completions", + json={"model": DEAR, "messages": _messages()}) + + assert not _lines(logbuf, "route"), "nothing was routed" + assert _fields(_lines(logbuf, "passthrough")[0])["model"] == DEAR + + +def test_no_conversation_text_ever_reaches_the_log(router, logbuf): + """Prompts here run 60k-150k tokens and the journal is on disk. + + Asserted at DEBUG, the most verbose level, on both the request and the + answer. + """ + secret = "SQUAMOUS-EPHEMERAL-9137" + client, _, _ = router + + client.post("/v1/chat/completions", + json={"model": "auto", "messages": _messages(f"refactor {secret}")}) + + assert secret not in logbuf.getvalue() + # The stubbed provider answers "hello there"; that must not appear either. + assert "hello there" not in logbuf.getvalue() + + +def test_the_decision_line_reports_the_real_provenance(router, logbuf): + """The re-route for measured context must not read as a client override. + + chat_completions routes a second time when the conversation measures larger + than the classifier guessed, passing category and tier back in — which + makes the resulting Classification say source='override'. In a log line + that means "the client chose this", and the client chose nothing: the + classifier ran and only the context figure was replaced. + """ + client, _, _ = router + # ~1200 chars / 3 = 400 tokens measured, above the stubbed estimate of 100. + client.post("/v1/chat/completions", + json={"model": "auto", "messages": _messages("refactor " + "x " * 600)}) + + fields = _fields(_lines(logbuf, "route")[0]) + + assert fields["src"] == "classifier", "the classifier did decide this" + assert fields["ctx_src"] == "measured", "but the context came from measurement" + assert int(fields["ctx"]) > 100 + + +def test_a_caller_supplied_context_says_so(router, logbuf): + client, _, _ = router + client.post("/route", json={"task": "refactor this", + "task_category": "coding_general", + "task_tier": 2, + "required_context_tokens": 5000}) + + fields = _fields(_lines(logbuf, "route")[0]) + + assert fields["src"] == "override" + assert fields["ctx_src"] == "caller" + + +# --- capability gates: vision and JSON mode ------------------------------- + +def _image_messages(text="what is in this image?"): + """A user turn carrying an inline image_url part, OpenAI multimodal shape.""" + return [ + { + "role": "user", + "content": [ + {"type": "text", "text": text}, + { + "type": "image_url", + "image_url": {"url": "data:image/png;base64,AAAA"}, + }, + ], + } + ] + + +def _drop_cheap_by_tier(client_type, db_path): + """Make CHEAP ineligible so DEAR is the only remaining candidate.""" + conn = sqlite3.connect(db_path) + conn.execute("UPDATE models SET tier = 1 WHERE model_id = ?", (CHEAP,)) + conn.commit() + conn.close() + + +def _local_vision_fake(monkeypatch, router_calls, content="local caption", status=200): + """Point the local vision fallback at a fake that returns `content`.""" + def fake_post(url, headers=None, json=None, stream=False, timeout=None): + if "/chat/completions" in url: + router_calls.append({"url": url, "body": json, "stream": stream, "local": True}) + return FakeResponse( + { + "choices": [ + {"message": {"role": "assistant", "content": content}, + "finish_reason": "stop"} + ] + }, + status_code=status, + ) + router_calls.append({"url": url, "body": json, "stream": stream}) + return FakeResponse(completion(json["model"])) + + monkeypatch.setattr(dispatcher.requests, "post", fake_post) + + +def test_an_image_request_is_routed_to_a_vision_model(router): + """Image parts hard-restrict to vision-capable rows; CHEAP has vision.""" + client, calls, _ = router + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _image_messages()}, + ) + + assert resp.status_code == 200 + assert calls[0]["body"]["model"] == CHEAP + assert resp.headers["X-Router-Model"] == CHEAP + + +def test_an_image_request_excludes_a_non_vision_candidate(router): + """DEAR is eligible-tiered but lacks vision, so no candidate meets the ask.""" + client, calls, db_path = router + _drop_cheap_by_tier(dispatcher, db_path) + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _image_messages()}, + ) + + assert resp.status_code == 422 + assert "vision" in resp.json()["detail"] + assert not calls, "a non-vision model must never be dispatched for an image" + + +def test_no_vision_model_uses_local_fallback_when_enabled(router, monkeypatch): + """No cloud vision candidate, local vision on: the local model answers.""" + client, calls, db_path = router + _drop_cheap_by_tier(dispatcher, db_path) + monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True) + _local_vision_fake(monkeypatch, calls, content="local caption") + + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _image_messages()}, + ) + + assert resp.status_code == 200 + assert resp.json()["choices"][0]["message"]["content"] == "local caption" + local = [c for c in calls if c["url"].endswith("/chat/completions")] + assert local, "the local fallback should POST to its own /chat/completions" + parts = local[0]["body"]["messages"][0]["content"] + assert any( + isinstance(p, dict) and p.get("type") == "image_url" for p in parts + ), "the image_url part must survive into the local call" + + +def test_local_fallback_respects_streaming(router, monkeypatch): + client, calls, db_path = router + _drop_cheap_by_tier(dispatcher, db_path) + monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True) + _local_vision_fake(monkeypatch, calls, content="streamed caption") + + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _image_messages(), "stream": True}, + ) + + assert resp.status_code == 200 + assert "streamed caption" in resp.text + assert "data: [DONE]" in resp.text + + +def test_local_fallback_failure_still_422s(router, monkeypatch): + """A local vision failure must be visible, not swallowed into a 200.""" + client, calls, db_path = router + _drop_cheap_by_tier(dispatcher, db_path) + monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True) + _local_vision_fake(monkeypatch, calls, status=500) + + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _image_messages()}, + ) + + assert resp.status_code == 422 + assert "vision" in resp.json()["detail"] + + +def test_a_json_object_response_format_routes_to_a_json_capable_model(router): + """response_format json_object hard-restricts to JSON-mode-capable rows.""" + client, calls, _ = router + resp = client.post( + "/v1/chat/completions", + json={ + "model": "auto", + "messages": _messages(), + "response_format": {"type": "json_object"}, + }, + ) + + assert resp.status_code == 200 + assert calls[0]["body"]["model"] == CHEAP + + +def test_a_json_object_request_without_a_json_capable_model_422s(router): + client, calls, db_path = router + conn = sqlite3.connect(db_path) + conn.execute("UPDATE models SET supports_json_mode = 0") + conn.commit() + conn.close() + + resp = client.post( + "/v1/chat/completions", + json={ + "model": "auto", + "messages": _messages(), + "response_format": {"type": "json_object"}, + }, + ) + + assert resp.status_code == 422 + assert "json" in resp.json()["detail"] + assert not calls + + +def test_a_plain_text_request_is_unaffected_by_the_gates(router): + """No images, no response_format: routing is exactly as before.""" + client, calls, _ = router + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _messages()}, + ) + + assert resp.status_code == 200 + assert calls[0]["body"]["model"] == CHEAP + assert resp.headers["X-Router-Model"] == CHEAP + + +def test_a_pinned_non_vision_model_with_an_image_is_a_clear_422(router): + """A pin that cannot satisfy the request fails early with a named reason.""" + client, calls, _ = router + resp = client.post( + "/v1/chat/completions", + json={"model": DEAR, "messages": _image_messages()}, + ) + + assert resp.status_code == 422 + assert "vision" in resp.json()["detail"] + assert not calls, "no provider call should happen for an impossible pin" + + +def test_the_two_pass_reroute_passes_image_capability(router): + """The measured-context reroute must carry the image gate, not lose it.""" + client, calls, _ = router + # ~1200 chars / 3 = 400 measured tokens > the stubbed estimate of 100, so + # chat_completions reroutes — and the reroute must still require vision. + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _image_messages("refactor " + "x " * 600)}, + ) + + assert resp.status_code == 200 + assert resp.headers["X-Router-Model"] == CHEAP + assert calls[0]["body"]["model"] == CHEAP + + +def test_local_fallback_refuses_a_remote_image_url(router, monkeypatch): + """A remote http(s) URL is an SSRF vector by indirection and must not be + forwarded to the local model, even when the fallback is enabled.""" + client, calls, db_path = router + _drop_cheap_by_tier(dispatcher, db_path) + monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True) + _local_vision_fake(monkeypatch, calls, content="should not be used") + + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "describe this"}, + { + "type": "image_url", + "image_url": {"url": "http://internal/private.png"}, + }, + ], + } + ] + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": messages}, + ) + + assert resp.status_code == 422 + assert "vision" in resp.json()["detail"] + local = [c for c in calls if c.get("local")] + assert not local, "a remote image URL must never reach the local vision endpoint" + + +def test_local_fallback_wont_masquerade_an_empty_answer_as_200(router, monkeypatch): + """A 200 with empty content is a failed answer, and must fall through to 422 + rather than return an empty 200 — a hidden failure must not look like a win.""" + client, calls, db_path = router + _drop_cheap_by_tier(dispatcher, db_path) + monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True) + _local_vision_fake(monkeypatch, calls, content=" ") + + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _image_messages()}, + ) + + assert resp.status_code == 422 + assert "vision" in resp.json()["detail"] + + +def test_local_fallback_refuses_too_many_images(router, monkeypatch): + """An image count above the budget refuses the local call before spending it.""" + client, calls, db_path = router + _drop_cheap_by_tier(dispatcher, db_path) + monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True) + _local_vision_fake(monkeypatch, calls, content="caption") + + many = _image_messages() + for _ in range(dispatcher.cfg.local_vision.max_images + 1): + many[0]["content"].append({"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}}) + + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": many}, + ) + + assert resp.status_code == 422 + local = [c for c in calls if c.get("local")] + assert not local, "over-budget image count must skip the local call" + + +def test_local_fallback_refuses_when_api_key_env_is_missing(router, monkeypatch): + """A configured local vision api_key_env with no env var declines the call.""" + client, calls, db_path = router + _drop_cheap_by_tier(dispatcher, db_path) + monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True) + monkeypatch.setattr(dispatcher.cfg.local_vision, "api_key_env", "LOCAL_VISION_KEY") + monkeypatch.delenv("LOCAL_VISION_KEY", raising=False) + _local_vision_fake(monkeypatch, calls, content="caption") + + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _image_messages()}, + ) + + assert resp.status_code == 422 + local = [c for c in calls if c.get("local")] + assert not local, "a missing vision key must skip the local call" + + +def test_local_fallback_refuses_an_unparseable_response(router, monkeypatch): + """An unparseable local response body must not yield a 200.""" + client, calls, db_path = router + _drop_cheap_by_tier(dispatcher, db_path) + monkeypatch.setattr(dispatcher.cfg.local_vision, "enabled", True) + + def fake_post(url, headers=None, json=None, stream=False, timeout=None): + calls.append({"url": url, "body": json, "stream": stream, "local": True}) + return FakeResponse({}, status_code=200) + + monkeypatch.setattr(dispatcher.requests, "post", fake_post) + + resp = client.post( + "/v1/chat/completions", + json={"model": "auto", "messages": _image_messages()}, + ) + + assert resp.status_code == 422 + assert "vision" in resp.json()["detail"] + + +def test_a_pinned_non_json_model_with_json_response_format_422s(router): + """The JSON-mode arm of the pass-through check mirrors the vision one: a pin + whose model lacks JSON mode gets a clear 422, not a provider 400.""" + client, calls, db_path = router + conn = sqlite3.connect(db_path) + conn.execute("UPDATE models SET supports_json_mode = 0 WHERE model_id = ?", (DEAR,)) + conn.commit() + conn.close() + + resp = client.post( + "/v1/chat/completions", + json={ + "model": DEAR, + "messages": _messages(), + "response_format": {"type": "json_object"}, + }, + ) + + assert resp.status_code == 422 + assert "json" in resp.json()["detail"] + assert not calls, "no provider call should happen for an impossible json pin" diff --git a/tests/test_classifier_input.py b/tests/test_classifier_input.py new file mode 100644 index 0000000..d1cf2c7 --- /dev/null +++ b/tests/test_classifier_input.py @@ -0,0 +1,63 @@ +"""The classifier gets the instruction, not the document. + +It decides a category and a tier. Handing it a pasted document is harmful +rather than merely wasteful — measured on a ~20k-token prompt, both local +models failed and neither failed cleanly: + + qwen3.5 28.7s, finish_reason=length, empty content + mistral-nemo 41.8s, echoed the input back inside its JSON + +Both surface as unparseable output, i.e. 30-40s of local inference spent to +reach the same `source: "fallback"` an instant failure would have produced. +With the clamp, the same prompts classify correctly in ~2.2s. +""" + +from __future__ import annotations + +from dispatcher import clamp_for_classifier + + +def test_short_input_is_untouched(): + assert clamp_for_classifier("classify me", 8000) == "classify me" + + +def test_input_at_the_limit_is_untouched(): + text = "x" * 8000 + assert clamp_for_classifier(text, 8000) == text + + +def test_the_instruction_survives_at_either_end(): + # Which end carries the instruction depends on how the caller phrased it: + # "Translate this: " puts it first, " — translate this" last. + # The middle of a pasted document is the one part that never carries it. + text = "HEAD-INSTRUCTION " + ("filler " * 20000) + " TAIL-INSTRUCTION" + out = clamp_for_classifier(text, 8000) + assert "HEAD-INSTRUCTION" in out + assert "TAIL-INSTRUCTION" in out + + +def test_clamped_output_is_bounded(): + text = "y" * 500_000 + out = clamp_for_classifier(text, 8000) + # Head + tail + the elision marker, not the original. + assert len(out) < 8200 + assert len(out) < len(text) + + +def test_the_elision_is_visible_to_the_model(): + # The model should be able to tell it is seeing an excerpt, rather than + # silently reasoning about a document that appears to jump mid-sentence. + out = clamp_for_classifier("z" * 50_000, 8000) + assert "elided" in out + + +def test_zero_disables_clamping(): + # Escape hatch: a deployment with a large-context classifier and a reason + # to use it should not have to patch code. + text = "w" * 100_000 + assert clamp_for_classifier(text, 0) == text + + +def test_negative_is_treated_as_disabled_not_as_a_crash(): + text = "w" * 100_000 + assert clamp_for_classifier(text, -1) == text diff --git a/tests/test_config_endpoints.py b/tests/test_config_endpoints.py new file mode 100644 index 0000000..0458b7b --- /dev/null +++ b/tests/test_config_endpoints.py @@ -0,0 +1,232 @@ +"""The classifier and the local verifier are separate endpoints. + +They used to be one: the verifier derived its URL by stripping ``/v1`` off +``classifier.base_url``. That silently coupled two unrelated decisions, and +the coupling only becomes visible when the classifier is moved off-host — +pointing classification at a cloud provider would have sent every local +verification to ``/api/chat``, which does not exist. + +The verifier speaks Ollama's NATIVE API (``/api/chat`` with ``think=False``) +because that is the only way to disable the reasoning trace, so it cannot +follow the classifier anywhere the classifier can go. +""" + +from __future__ import annotations + +import copy + +import pytest +import yaml + +from config import RouterConfig + + +@pytest.fixture +def raw() -> dict: + with open("config.yaml") as fh: + return yaml.safe_load(fh) + + +def test_moving_the_classifier_to_a_cloud_provider_leaves_the_verifier_local(raw): + cfg = copy.deepcopy(raw) + cfg["classifier"]["base_url"] = "https://api.neuralwatt.com/v1" + cfg["classifier"]["api_key_env"] = "NEURALWATT_API_KEY" + cfg["classifier"]["model"] = "deepseek-v4-flash" + + loaded = RouterConfig(**cfg) + + assert loaded.classifier.base_url == "https://api.neuralwatt.com/v1" + # The whole point: this did NOT follow the line above. + assert "localhost" in loaded.verification.base_url + + +def test_the_verifier_defaults_to_a_local_ollama(raw): + cfg = copy.deepcopy(raw) + cfg["verification"].pop("base_url", None) + assert RouterConfig(**cfg).verification.base_url == "http://localhost:11434" + + +def test_an_absent_api_key_env_means_unauthenticated(raw): + # The local Ollama case. It ignores the key entirely, but the SDK requires + # one to be set, so the dispatcher substitutes a placeholder. Asserted on + # an explicit config rather than on whatever config.yaml currently says, + # which is a deployment choice and not a property of the code. + cfg = copy.deepcopy(raw) + cfg["classifier"]["base_url"] = "http://localhost:11434/v1" + cfg["classifier"].pop("api_key_env", None) + assert RouterConfig(**cfg).classifier.api_key_env is None + + cfg["classifier"]["api_key_env"] = "NEURALWATT_API_KEY" + assert RouterConfig(**cfg).classifier.api_key_env == "NEURALWATT_API_KEY" + + +def test_verification_model_may_be_unset_while_both_run_on_one_host(raw): + cfg = copy.deepcopy(raw) + cfg["classifier"]["base_url"] = "http://localhost:11434/v1" + cfg["verification"]["model"] = None + loaded = RouterConfig(**cfg) + # Resolved by the dispatcher as `verification.model or classifier.model`, + # which is right only because the hosts match. + assert loaded.verification.model is None + + +def test_an_unset_verifier_model_is_refused_once_the_hosts_differ(raw): + # The failure this prevents is SILENT, which is why it is a load-time + # error rather than a documented caveat. Observed directly: with the + # classifier on NeuralWatt and this left null, the verifier POSTed + # `deepseek-v4-flash` to localhost:11434, 404d, caught it, logged "local + # verification unavailable" and recorded no sample. Verification looked + # enabled while producing nothing. + cfg = copy.deepcopy(raw) + cfg["classifier"]["base_url"] = "https://api.neuralwatt.com/v1" + cfg["classifier"]["model"] = "deepseek-v4-flash" + cfg["verification"]["model"] = None + cfg["verification"]["local_llm_enabled"] = True + + with pytest.raises(ValueError, match="verification.model must be set"): + RouterConfig(**cfg) + + +def test_a_split_host_setup_loads_once_the_verifier_model_is_stated(raw): + cfg = copy.deepcopy(raw) + cfg["classifier"]["base_url"] = "https://api.neuralwatt.com/v1" + cfg["classifier"]["api_key_env"] = "NEURALWATT_API_KEY" + cfg["classifier"]["model"] = "deepseek-v4-flash" + cfg["verification"]["model"] = "qwen3.5:latest" + + loaded = RouterConfig(**cfg) + assert loaded.classifier.model == "deepseek-v4-flash" + assert loaded.verification.model == "qwen3.5:latest" + + +def test_disabling_the_local_check_lifts_the_requirement(raw): + # A host with no local inference at all: nothing to name, nothing to guard. + cfg = copy.deepcopy(raw) + cfg["classifier"]["base_url"] = "https://api.neuralwatt.com/v1" + cfg["verification"]["model"] = None + cfg["verification"]["local_llm_enabled"] = False + + loaded = RouterConfig(**cfg) + assert loaded.verification.local_llm_enabled is False + + +def test_a_host_with_no_local_inference_can_turn_the_local_check_off(raw): + # An RPi has no usable local model. Structural verification is pure Python + # and keeps running; only the LLM check goes away. + cfg = copy.deepcopy(raw) + cfg["verification"]["local_llm_enabled"] = False + loaded = RouterConfig(**cfg) + assert loaded.verification.local_llm_enabled is False + + +# --- an unknown key is an error, not a no-op ------------------------------ + +def test_a_key_in_the_wrong_section_is_rejected(raw): + # Exactly the mistake that shipped: max_input_chars belongs to the + # classifier and was written into verification, where pydantic's default + # extra="ignore" accepted it, dropped it, and left the code default in + # force. It carried the same value, so nothing looked wrong -- but editing + # it would have done nothing at all. + cfg = copy.deepcopy(raw) + cfg["verification"]["max_input_chars"] = 4000 + with pytest.raises(ValueError, match="max_input_chars"): + RouterConfig(**cfg) + + +def test_a_misspelled_key_is_rejected(raw): + cfg = copy.deepcopy(raw) + cfg["routing"]["min_tool_proficency"] = 0.5 # sic + with pytest.raises(ValueError, match="min_tool_proficency"): + RouterConfig(**cfg) + + +def test_the_shipped_config_has_no_unknown_keys(raw): + # Guards the whole file, not just the sections a test happens to name. + RouterConfig(**raw) + + +def test_the_tool_filter_can_be_turned_off_from_config(raw): + # It is a knob to experiment with, so null must be a legal value rather + # than something requiring a code change. + cfg = copy.deepcopy(raw) + cfg["routing"]["min_tool_proficiency"] = None + assert RouterConfig(**cfg).routing.min_tool_proficiency is None + + +def test_disabling_the_filter_lifts_the_category_name_check(raw): + # With no filter there is nothing to join, so an unused category name + # must not block startup. + cfg = copy.deepcopy(raw) + cfg["routing"]["min_tool_proficiency"] = None + cfg["routing"]["tool_use_category"] = "not_a_real_category" + assert RouterConfig(**cfg).routing.min_tool_proficiency is None + + +# --- the CLI sanity check ------------------------------------------------- + +def test_the_config_summary_names_fields_that_exist(raw): + """`python config.py` is the documented setup step, and it crashed. + + It printed cfg.weights, which had been replaced by cfg.objective, so the + one command whose job is to prove the config is fine reported "Config + loaded OK" and then died with AttributeError. A function plus this test + means the names cannot rot silently again. + """ + from config import summary_lines + + lines = summary_lines(RouterConfig(**raw)) + + assert lines[0] == "Config loaded OK" + body = "\n".join(lines[1:]) + assert "quality_tolerance=" in body + assert "classifier:" in body + assert "dispatch providers:" in body + + +def test_a_removed_key_is_rejected_rather_than_ignored(raw): + """log_path named a file nothing ever wrote; leaving it valid would lie.""" + import copy + + import pytest + + cfg = copy.deepcopy(raw) + cfg["logging"]["log_path"] = "router.log" + + with pytest.raises(Exception, match="log_path|Extra inputs"): + RouterConfig(**cfg) + + +# --- capability gates and the local vision fallback ------------------------ + +def test_the_shipped_config_loads_with_the_new_keys(raw): + loaded = RouterConfig(**raw) + assert loaded.routing.require_vision is True + assert loaded.routing.require_json_mode is True + assert loaded.local_vision.model == "qwen3-vl:4b" + assert loaded.local_vision.enabled is False + + +def test_a_typo_in_require_vision_is_rejected(raw): + cfg = copy.deepcopy(raw) + cfg["routing"]["require_visoin"] = True # sic + with pytest.raises(ValueError, match="require_visoin"): + RouterConfig(**cfg) + + +def test_local_vision_can_be_disabled_from_config(raw): + cfg = copy.deepcopy(raw) + cfg["local_vision"]["enabled"] = False + assert RouterConfig(**cfg).local_vision.enabled is False + + +def test_local_vision_defaults_when_absent(raw): + cfg = copy.deepcopy(raw) + cfg.pop("local_vision") + assert RouterConfig(**cfg).local_vision.enabled is False + + +def test_nonpositive_local_timeout_is_rejected(raw): + cfg = copy.deepcopy(raw) + cfg["local_vision"]["timeout_seconds"] = 0 + with pytest.raises(ValueError, match="timeout_seconds"): + RouterConfig(**cfg) diff --git a/tests/test_eval_scoring.py b/tests/test_eval_scoring.py index 0768e5e..7bac321 100644 --- a/tests/test_eval_scoring.py +++ b/tests/test_eval_scoring.py @@ -13,6 +13,8 @@ point of testing it. import pytest from eval_proficiency import ( + judge_for, + parse_check_results, extract_judge_json, judge_for, normalize_answer, @@ -207,3 +209,92 @@ def test_a_single_leading_space_does_not_destroy_a_correct_answer(): def test_indented_fenced_block_is_still_recovered(): code = ' ```python\ndef f(x):\n return x * 2\n```' assert score_code(code, ["f(2)==4"])[0] == 1.0 + + +# --- harness integrity ---------------------------------------------------- +# +# The harness runs the model's code as `__main__`, so anything the model prints +# lands in the same stdout the verdicts are read from. Counting the substring +# "PASS" therefore let a model vote on its own work. + +def test_a_models_own_demo_output_cannot_award_it_marks(): + """Measured at 1.50 on a two-check task before the verdicts carried a nonce.""" + code = ( + "def add(a, b):\n" + " return a + b\n" + "\n" + 'if __name__ == "__main__":\n' + ' print("self-test:", add(1, 2) == 3 and "PASS" or "FAIL")\n' + ) + + score, detail = score_code(code, ["add(1,2)==3", "add(-1,1)==0"]) + + assert score == 1.0 + assert detail == "2/2 checks" + + +def test_printing_PASS_cannot_rescue_a_wrong_answer(): + """The failure that matters: noise must not manufacture credit.""" + code = ( + "def add(a, b):\n" + " return 0\n" + "\n" + 'if __name__ == "__main__":\n' + ' print("PASS PASS PASS")\n' + ) + + # Both checks genuinely fail for a function that always returns 0. + assert score_code(code, ["add(1,2)==3", "add(2,2)==4"])[0] == 0.0 + + +def test_a_score_can_never_exceed_one(): + """Whatever a model prints, the fraction stays a fraction.""" + code = "def f(x):\n return x\n" + 'print("CHECK 0 PASS\\n" * 50)\n' + + assert score_code(code, ["f(1)==1"])[0] <= 1.0 + + +def test_check_results_are_read_by_index_not_counted(): + marker = "CHECK-deadbeef" + stdout = ( + "PASS\n" # the model's noise + f"{marker} 0 PASS\n" + f"{marker} 1 FAIL\n" + "CHECK 2 PASS\n" # right shape, wrong marker + f"{marker} 0 PASS\n" # a repeat cannot double-count + ) + + assert parse_check_results(stdout, marker) == {0: True, 1: False} + + +# --- judge selection ------------------------------------------------------ + +def test_the_configured_judge_is_used_when_it_is_a_different_model(): + assert judge_for("gemma-4-31b", "kimi-k3") == "kimi-k3" + + +def test_a_model_is_never_judged_by_its_own_family(): + """Including the case that broke it: the judge and the alternate agreeing.""" + assert judge_for("kimi-k3", "kimi-k3") != "kimi-k3" + # `--judge-model qwen3.6-35b` used to hand qwen3.6-35b back to itself. + assert judge_for("qwen3.6-35b", "qwen3.6-35b") != "qwen3.6-35b" + + +def test_a_fast_row_is_not_judged_by_its_reasoning_on_sibling(): + """Same weights, so the conflict is the same one.""" + assert judge_for("qwen3.6-35b-fast", "qwen3.6-35b") not in ( + "qwen3.6-35b", "qwen3.6-35b-fast" + ) + + +def test_no_judge_outside_the_family_means_no_sample(): + """Skipping beats self-grading, the same rule an unusable judge reply follows.""" + assert judge_for("kimi-k3", "kimi-k3", ) is not None # alternates exist + import eval_proficiency + + original = eval_proficiency.ALTERNATE_JUDGES + try: + eval_proficiency.ALTERNATE_JUDGES = ("kimi-k3-fast",) + assert eval_proficiency.judge_for("kimi-k3", "kimi-k3") is None + finally: + eval_proficiency.ALTERNATE_JUDGES = original diff --git a/tests/test_load_candidates.py b/tests/test_load_candidates.py index cbbe99c..0c6ca7f 100644 --- a/tests/test_load_candidates.py +++ b/tests/test_load_candidates.py @@ -170,3 +170,22 @@ def test_context_estimate_reads_multimodal_text_parts(): {"type": "image_url", "image_url": {"url": "data:..."}}, ]}] assert estimate_prompt_tokens(msgs) == 100 + + +def test_context_estimate_counts_tool_definitions(): + # upstream_body = {**body, "model": target} forwards `tools` to the + # provider verbatim on every call, so it counts against the same context + # window and has to count here too. Omitting it undercounts every + # tool-carrying request -- nearly all agent traffic -- which is exactly + # how qwen3.6-35b got a live 400 for "too long ... even after + # compaction" while the router's own estimate said it fit. + from dispatcher import estimate_prompt_tokens + + msgs = [{"role": "user", "content": "a" * 300}] + tools = [{"type": "function", "function": { + "name": "read_file", + "parameters": {"type": "object", "properties": {"path": {"type": "string"}}}, + }}] + without_tools = estimate_prompt_tokens(msgs) + with_tools = estimate_prompt_tokens(msgs, tools=tools) + assert with_tools > without_tools diff --git a/tests/test_logs.py b/tests/test_logs.py new file mode 100644 index 0000000..bf40b90 --- /dev/null +++ b/tests/test_logs.py @@ -0,0 +1,238 @@ +"""Tests for the router's structured logging. + +Written against the real handler and formatter rather than pytest's caplog, +because the thing worth pinning is what actually lands in the journal -- +including the priority prefix, which caplog would never see. +""" + +import io +import logging +import os + +import pytest + +import logs + + +@pytest.fixture(autouse=True) +def isolated_logger(monkeypatch): + """Every test gets a fresh logger and a clean environment.""" + monkeypatch.delenv(logs.LEVEL_ENV, raising=False) + monkeypatch.delenv("JOURNAL_STREAM", raising=False) + logs.set_trace(logs.NO_TRACE) + yield + for handler in list(logs.log.handlers): + logs.log.removeHandler(handler) + + +def _capture(level="debug", **kwargs): + buf = io.StringIO() + logs.configure(level, stream=buf, **kwargs) + return buf + + +# --- rendering ------------------------------------------------------------ + +def test_a_line_is_logfmt_with_the_trace_id_first(): + buf = _capture() + logs.set_trace("r123456") + + logs.info("route", cat="coding_general", tier=2) + + assert buf.getvalue() == "route id=r123456 cat=coding_general tier=2\n" + + +def test_values_with_spaces_are_quoted(): + buf = _capture() + + logs.info("filter", detail="held during peak") + + assert 'detail="held during peak"' in buf.getvalue() + + +def test_a_value_never_breaks_the_line(): + """One event, one line -- a newline in a detail string would forge a record.""" + buf = _capture() + + logs.info("verify", detail="line one\nline two") + + assert buf.getvalue().count("\n") == 1 + + +def test_none_is_a_dash_and_bools_are_binary(): + buf = _capture() + + logs.info("route", pick=None, tools=True, stream=False) + + assert "pick=- tools=1 stream=0" in buf.getvalue() + + +def test_floats_are_readable_not_repr(): + """Microdollars and kWh, without 0.0007830000000000001.""" + buf = _capture() + + logs.info("dispatch", usd=0.000783123456, kwh=4.75e-05) + + assert "usd=0.000783123" in buf.getvalue() + assert "kwh=4.75e-05" in buf.getvalue() + + +def test_an_empty_string_is_visible(): + buf = _capture() + + logs.info("classify", cat="") + + assert 'cat=""' in buf.getvalue() + + +# --- levels --------------------------------------------------------------- + +def test_debug_is_silent_at_info_level(): + buf = _capture("info") + + logs.debug("filter", model="m") + logs.info("route", pick="m") + + assert "filter" not in buf.getvalue() + assert "route" in buf.getvalue() + + +def test_enabled_for_debug_reports_the_level(): + _capture("info") + assert logs.enabled_for_debug() is False + _capture("debug") + assert logs.enabled_for_debug() is True + + +def test_the_env_var_overrides_the_configured_level(monkeypatch): + """So a live service can be turned up without editing a tracked file.""" + monkeypatch.setenv(logs.LEVEL_ENV, "debug") + buf = _capture("warning") + + logs.debug("filter", model="m") + + assert "filter" in buf.getvalue() + + +def test_a_nonsense_env_level_falls_back_rather_than_crashing(monkeypatch): + monkeypatch.setenv(logs.LEVEL_ENV, "loud") + + assert logs.resolve_level("warning") == logging.WARNING + + +# --- journald ------------------------------------------------------------- + +def test_no_priority_prefix_in_a_terminal(): + """A foreground uvicorn should not print a literal <6> on every line.""" + buf = _capture(journald=False) + + logs.info("route", pick="m") + + assert buf.getvalue().startswith("route ") + + +def test_priority_prefix_under_journald(): + """This is what makes `journalctl -p warning` mean anything.""" + buf = _capture(journald=True) + + logs.warning("retry", verdict="malformed") + logs.error("upstream", status=500) + logs.info("route", pick="m") + + lines = buf.getvalue().splitlines() + assert lines[0].startswith("<4>") # warning + assert lines[1].startswith("<3>") # error + assert lines[2].startswith("<6>") # info + + +def test_journald_is_detected_by_matching_the_actual_stream(tmp_path, monkeypatch): + """The env var alone is not enough -- it is inherited by every child. + + A foreground uvicorn started from a systemd-managed session, with output + redirected to a file, inherited JOURNAL_STREAM and put a literal <7> on + every line. systemd documents comparing the value against the fd. + """ + path = tmp_path / "out.log" + with open(path, "w") as handle: + stat = os.fstat(handle.fileno()) + + monkeypatch.setenv("JOURNAL_STREAM", f"{stat.st_dev}:{stat.st_ino}") + assert logs.under_journald(handle) is True + + # Same variable, a different stream: this is the inherited case. + monkeypatch.setenv("JOURNAL_STREAM", f"{stat.st_dev}:{stat.st_ino + 1}") + assert logs.under_journald(handle) is False + + +def test_no_journald_variable_means_no_prefix(): + assert logs.under_journald() is False + + +def test_a_stream_with_no_fileno_is_not_journald(): + """StringIO under test, and anything else without an fd.""" + assert logs.under_journald(io.StringIO()) is False + + +def test_a_malformed_journald_variable_is_ignored(monkeypatch): + monkeypatch.setenv("JOURNAL_STREAM", "not-a-device") + assert logs.under_journald() is False + + +# --- trace ids ------------------------------------------------------------ + +def test_a_trace_id_is_stable_until_a_new_one_is_started(): + first = logs.new_trace() + + assert logs.current_trace() == first + assert logs.new_trace() != first + + +def test_trace_ids_do_not_collide(): + assert len({logs.new_trace() for _ in range(200)}) == 200 + + +def test_configure_is_idempotent(): + """Re-importing or re-configuring must not double every line.""" + buf = io.StringIO() + logs.configure("info", stream=buf, journald=False) + logs.configure("info", stream=buf, journald=False) + + logs.info("route", pick="m") + + assert buf.getvalue().count("route") == 1 + + +def test_the_logger_does_not_propagate(): + """uvicorn configures its own logging; a root handler would double-print.""" + _capture() + assert logs.log.propagate is False + + +# --- bound emitters ------------------------------------------------------- + +def test_a_bound_emitter_ignores_the_ambient_trace(): + """What the streaming path needs: an id that survives a context copy.""" + buf = _capture() + bound = logs.bind("rfixed1") + logs.set_trace("rother2") + + bound.info("dispatch", model="m") + + assert "id=rfixed1" in buf.getvalue() + + +def test_bind_defaults_to_the_current_trace(): + buf = _capture() + logs.set_trace("rcurrent") + + logs.bind().warning("retry", verdict="malformed") + + assert "id=rcurrent" in buf.getvalue() + + +def test_a_bound_emitter_respects_the_level(): + buf = _capture("info") + + logs.bind("r1").debug("filter", model="m") + + assert buf.getvalue() == "" diff --git a/tests/test_outcome_attribution.py b/tests/test_outcome_attribution.py new file mode 100644 index 0000000..599dfd1 --- /dev/null +++ b/tests/test_outcome_attribution.py @@ -0,0 +1,156 @@ +"""Tests for which completion an outcome report attaches to. + +`POST /outcome` is the only ground truth the router gets, so attributing one +to the wrong conversation is worse than losing it: a model gets penalized for +work it never did. The guard is a short window plus a refusal -- if more than +one conversation was served inside it, the report is refused rather than +guessed at. + +The window was not a window. `observed_at` is written by +`datetime.now(timezone.utc).isoformat()`, which separates date from time with +'T', while `datetime('now', ...)` returns a space. Compared as strings, 'T' +sorts after ' ', so the time of day never participated once the dates matched +and a 120-second window admitted everything served that day. Measured on the +live DB: 14 rows across 2 sessions where the correct comparison matched 0. + +These tests write timestamps through the same call the dispatcher uses, so +they stay honest if that format ever changes. +""" + +import sqlite3 +from datetime import datetime, timedelta, timezone +from pathlib import Path + +import pytest + +import dispatcher +from dispatcher import AMBIGUOUS, SEED_CATEGORY, _most_recent_if_unambiguous + +ROOT = Path(__file__).resolve().parent.parent +SCHEMA_SQL = (ROOT / "schema.sql").read_text() + +# Short, so "earlier today" is unambiguously outside it. The old row below sits +# at the first instant of the current UTC day -- the earliest time that still +# shares today's date, which is what the string comparison needed to go wrong. +WINDOW_SECONDS = 5 + + +def _now() -> datetime: + return datetime.now(timezone.utc) + + +def _start_of_today() -> datetime: + return _now().replace(hour=0, minute=0, second=0, microsecond=1) + + +@pytest.fixture(autouse=True) +def short_window(monkeypatch): + monkeypatch.setattr( + dispatcher.cfg.verification, + "outcome_attribution_window_seconds", + WINDOW_SECONDS, + ) + + +@pytest.fixture +def db(tmp_path): + conn = sqlite3.connect(tmp_path / "test.db") + conn.row_factory = sqlite3.Row + conn.executescript(SCHEMA_SQL) + yield conn + conn.close() + + +def _observe(conn, *, when: datetime, session_key: str, request_id: str, + category: str = "coding_general"): + conn.execute( + """ + INSERT INTO energy_observations ( + model_id, provider, request_id, session_key, task_category, observed_at + ) VALUES ('m', 'neuralwatt', ?, ?, ?, ?) + """, + # Written exactly the way log_observation writes it. + (request_id, session_key, category, when.isoformat()), + ) + conn.commit() + + +def test_a_session_from_earlier_today_is_outside_the_window(db): + """The regression: same calendar date is not the same as inside 5 seconds.""" + _observe(db, when=_start_of_today(), session_key="morning", request_id="old") + _observe(db, when=_now(), session_key="now", request_id="new") + + row = _most_recent_if_unambiguous(db) + + assert row is not AMBIGUOUS, "an 8-hours-stale session must not create ambiguity" + assert row["request_id"] == "new" + + +def test_an_old_session_alone_is_not_recent_enough_to_attribute(db): + """With nothing inside the window there is nothing to attach to -- 404, not a guess.""" + _observe(db, when=_start_of_today(), session_key="morning", request_id="old") + + assert _most_recent_if_unambiguous(db) is None + + +def test_two_live_conversations_are_refused(db): + """The guard the window exists to serve, still working.""" + _observe(db, when=_now(), session_key="alice", request_id="a") + _observe(db, when=_now(), session_key="bob", request_id="b") + + assert _most_recent_if_unambiguous(db) is AMBIGUOUS + + +def test_one_conversation_across_several_turns_is_not_ambiguous(db): + """Several completions, one session -- the ordinary case must still attribute.""" + for i in range(3): + _observe(db, when=_now(), session_key="alice", request_id=f"a{i}") + + row = _most_recent_if_unambiguous(db) + + assert row is not AMBIGUOUS + assert row["request_id"] == "a2" + + +def test_the_reference_sweep_is_not_a_conversation(db): + """seed_energy's traffic must never make a real report look ambiguous.""" + _observe(db, when=_now(), session_key=None, request_id="seed", + category=SEED_CATEGORY) + _observe(db, when=_now(), session_key="alice", request_id="a") + + row = _most_recent_if_unambiguous(db) + + assert row is not AMBIGUOUS + assert row["request_id"] == "a" + + +def test_quota_burn_counts_only_the_last_thirty_days(db, tmp_path, monkeypatch): + """Same comparison, same fix -- an old row must not inflate the burn figure. + + The row sits at the START of the day 30 days ago: the cutoff falls on that + same date but later in it, which is exactly the case a string comparison + gets wrong. A row from 45 days ago would be excluded either way and would + pin nothing. + """ + monkeypatch.setattr(dispatcher.cfg.database, "path", str(tmp_path / "test.db")) + monkeypatch.setattr(dispatcher.cfg.objective, "plan_kwh_per_period", 6.25) + just_outside = (_now() - timedelta(days=30)).replace( + hour=0, minute=0, second=0, microsecond=1 + ) + db.execute( + """ + INSERT INTO energy_observations (model_id, provider, energy_kwh, observed_at) + VALUES ('m', 'neuralwatt', 1.0, ?) + """, + (just_outside.isoformat(),), + ) + db.execute( + """ + INSERT INTO energy_observations (model_id, provider, energy_kwh, observed_at) + VALUES ('m', 'neuralwatt', 0.25, ?) + """, + (_now().isoformat(),), + ) + db.commit() + + assert dispatcher.quota_burn()["metered_kwh_30d"] == pytest.approx(0.25) diff --git a/tests/test_poller_parsing.py b/tests/test_poller_parsing.py index bcffd61..05ea985 100644 --- a/tests/test_poller_parsing.py +++ b/tests/test_poller_parsing.py @@ -105,3 +105,87 @@ def test_family_never_collapses_to_empty(): # A degenerate id made only of suffix tokens must keep a segment, or every # such row would share one meaningless family assert parse_base_model_id("flex") == "flex" + + +# --- per-model context overrides ------------------------------------------ +# +# The knob validated and was never read, while config.yaml shipped a worked +# example for it. Anyone who followed that example got silence. + +def _row(**overrides): + from poller import ModelRow + + fields = dict( + model_id="qwen3.6-35b", provider="neuralwatt", base_model_id="qwen3.6-35b", + display_name=None, cost_per_1m_prompt=None, cost_per_1m_completion=None, + cost_per_1m_prompt_cached=None, context_window=200_000, + max_output_tokens=None, supports_tools=True, supports_json_mode=True, + supports_vision=False, supports_reasoning=True, + reasoning_default_enabled=True, latency_class="standard", + reasoning_mode="default", context_variant="full", access_level="public", + pricing_tbd=False, deprecated=False, + ) + fields.update(overrides) + return ModelRow(**fields) + + +def _cfg(overrides=None): + import copy + + import yaml + + from config import RouterConfig + + with open("config.yaml") as fh: + raw = copy.deepcopy(yaml.safe_load(fh)) + raw["context"]["per_model_overrides"] = overrides or {} + return RouterConfig(**raw) + + +def test_the_global_safety_factor_applies_without_an_override(): + # 200000 * 0.75 - 4096 + assert _row().effective_context_window(_cfg()) == 145_904 + + +def test_a_per_model_safety_factor_is_actually_applied(): + cfg = _cfg({"qwen3.6-35b": {"safety_factor": 0.85}}) + + # 200000 * 0.85 - 4096 + assert _row().effective_context_window(cfg) == 165_904 + + +def test_an_override_for_another_model_does_not_leak(): + cfg = _cfg({"kimi-k3": {"safety_factor": 0.85}}) + + assert _row().effective_context_window(cfg) == 145_904 + + +def test_a_zero_output_reserve_means_zero_not_unset(): + """Present-but-falsy: `or` would silently fall through to 4096.""" + cfg = _cfg({"qwen3.6-35b": {"output_reserve_tokens": 0}}) + + assert _row().effective_context_window(cfg) == 150_000 + + +def test_an_override_reserve_beats_the_models_advertised_ceiling(): + cfg = _cfg({"qwen3.6-35b": {"output_reserve_tokens": 8192}}) + + assert _row(max_output_tokens=16384).effective_context_window(cfg) == 141_808 + + +def test_each_key_falls_back_independently(): + cfg = _cfg({"qwen3.6-35b": {"safety_factor": 0.85}}) + + # factor overridden, reserve still the global default + assert _row().effective_context_window(cfg) == 165_904 + + +def test_a_typo_inside_an_override_is_an_error(): + """The point of typing it: StrictModel has to reach inside the block.""" + with pytest.raises(Exception, match="safety_factr|Extra inputs"): + _cfg({"qwen3.6-35b": {"safety_factr": 0.85}}) + + +def test_an_out_of_range_override_is_an_error(): + with pytest.raises(Exception, match="safety_factor"): + _cfg({"qwen3.6-35b": {"safety_factor": 1.5}}) diff --git a/tests/test_proficiency.py b/tests/test_proficiency.py index 413b2ae..7c416fb 100644 --- a/tests/test_proficiency.py +++ b/tests/test_proficiency.py @@ -177,3 +177,175 @@ def test_a_directly_measured_variant_is_never_overwritten(tmp_path): "SELECT self_eval_score FROM proficiency WHERE model_id='kimi-k3-flex'" ).fetchone() assert got["self_eval_score"] == 0.2 + + +def test_a_score_above_one_never_reaches_the_table(tmp_path): + """The single write path is where a bad scorer gets stopped. + + A blended_score above 1.0 does not just misreport one row: it raises + `best` in rank_candidates and shifts every other candidate's quality band. + """ + from config import load_config + from proficiency_store import add_self_eval + + cfg = load_config("config.yaml") + conn = _models_db(tmp_path, [("kimi-k3", "kimi-k3", "standard", "default", "full")]) + + add_self_eval(conn, cfg, "kimi-k3", "nw", "coding_general", [1.5, 1.0]) + + row = conn.execute( + "SELECT self_eval_score, blended_score FROM proficiency WHERE model_id = 'kimi-k3'" + ).fetchone() + assert row["self_eval_score"] == 1.0 + assert row["blended_score"] <= 1.0 + + +def test_a_negative_score_is_floored_at_zero(tmp_path): + from config import load_config + from proficiency_store import add_self_eval + + cfg = load_config("config.yaml") + conn = _models_db(tmp_path, [("kimi-k3", "kimi-k3", "standard", "default", "full")]) + + add_self_eval(conn, cfg, "kimi-k3", "nw", "coding_general", [-0.5, 0.5]) + + assert conn.execute( + "SELECT self_eval_score FROM proficiency WHERE model_id = 'kimi-k3'" + ).fetchone()["self_eval_score"] == 0.25 + + +def test_an_inherited_variant_refreshes_when_its_family_is_remeasured(tmp_path): + """Inheritance was a one-shot: a variant froze at its first copy, forever. + + The old guard skipped any row with self_eval_samples > 0, and an inherited + row has samples > 0 because inheritance copies them -- so it could never + tell "measured here" from "copied here" and never refreshed. Observed live: + kimi-k3 reached 0.957 while kimi-k3-flex sat at 0.85 with a week-old + timestamp, and the eval reported "propagated 0 inherited rows". Flex rows + serve auto:batch, so those requests ranked on stale scores. + """ + from config import load_config + from proficiency_store import add_self_eval, propagate_to_variants + + cfg = load_config("config.yaml") + conn = _models_db(tmp_path, [ + ("kimi-k3", "kimi-k3", "standard", "default", "full"), + ("kimi-k3-flex", "kimi-k3", "flex", "default", "full"), + ]) + + add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [0.85, 0.85]) + assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 1 + inherited = conn.execute( + "SELECT blended_score, inherited_from FROM proficiency WHERE model_id='kimi-k3-flex'" + ).fetchone() + assert inherited["blended_score"] == pytest.approx(0.85) + assert inherited["inherited_from"] == "kimi-k3" + + # The family learns more; the variant must follow it. + add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [1.0, 1.0, 1.0, 1.0]) + + assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 1 + refreshed = conn.execute( + "SELECT blended_score, self_eval_samples FROM proficiency WHERE model_id='kimi-k3-flex'" + ).fetchone() + assert refreshed["blended_score"] == pytest.approx(0.95) + assert refreshed["self_eval_samples"] == 6 + + +def test_a_measured_variant_still_outranks_its_family(tmp_path): + """The protection the old guard was reaching for, kept intact.""" + from config import load_config + from proficiency_store import add_self_eval, propagate_to_variants + + cfg = load_config("config.yaml") + conn = _models_db(tmp_path, [ + ("kimi-k3", "kimi-k3", "standard", "default", "full"), + ("kimi-k3-flex", "kimi-k3", "flex", "default", "full"), + ]) + + add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [1.0]) + add_self_eval(conn, cfg, "kimi-k3-flex", "nw", "docs_writing", [0.2]) + + assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 0 + row = conn.execute( + "SELECT blended_score, inherited_from FROM proficiency WHERE model_id='kimi-k3-flex'" + ).fetchone() + assert row["blended_score"] == pytest.approx(0.2) + assert row["inherited_from"] is None + + +def test_a_database_without_the_column_is_migrated(tmp_path): + """schema.sql only defines a NEW database; existing ones need the ALTER.""" + import sqlite3 + + from config import load_config + from proficiency_store import add_self_eval, ensure_columns + + cfg = load_config("config.yaml") + conn = _models_db(tmp_path, [("kimi-k3", "kimi-k3", "standard", "default", "full")]) + conn.execute("ALTER TABLE proficiency DROP COLUMN inherited_from") + assert "inherited_from" not in { + r[1] for r in conn.execute("PRAGMA table_info(proficiency)") + } + + ensure_columns(conn) + ensure_columns(conn) # idempotent + + add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [1.0]) + assert conn.execute( + "SELECT inherited_from FROM proficiency WHERE model_id='kimi-k3'" + ).fetchone()["inherited_from"] is None + + +def test_migration_unfreezes_variants_that_predate_the_column(tmp_path): + """The migration must ship the repair, not just the fix. + + ADD COLUMN gives every existing row NULL, and NULL means "measured here" -- + so without the backfill, the rows this whole change exists for would stay + frozen and the live flex rows would never catch up. + """ + from config import load_config + from proficiency_store import add_self_eval, ensure_columns, propagate_to_variants + + cfg = load_config("config.yaml") + conn = _models_db(tmp_path, [ + ("kimi-k3", "kimi-k3", "standard", "default", "full"), + ("kimi-k3-flex", "kimi-k3", "flex", "default", "full"), + ]) + # An old database: both rows carry samples, neither records provenance. + add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [0.957]) + add_self_eval(conn, cfg, "kimi-k3-flex", "nw", "docs_writing", [0.85]) + conn.execute("ALTER TABLE proficiency DROP COLUMN inherited_from") + + ensure_columns(conn) + + assert conn.execute( + "SELECT inherited_from FROM proficiency WHERE model_id='kimi-k3-flex'" + ).fetchone()["inherited_from"] == "kimi-k3" + # ...so the next run actually moves it. + assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 1 + assert conn.execute( + "SELECT blended_score FROM proficiency WHERE model_id='kimi-k3-flex'" + ).fetchone()["blended_score"] == pytest.approx(0.957) + + +def test_the_backfill_never_touches_a_standard_row(tmp_path): + """Only flex rows with a standard equivalent are provably inherited.""" + from config import load_config + from proficiency_store import add_self_eval, ensure_columns + + cfg = load_config("config.yaml") + conn = _models_db(tmp_path, [ + ("kimi-k3", "kimi-k3", "standard", "default", "full"), + ("kimi-k3-fast", "kimi-k3", "standard", "reduced", "full"), + ]) + add_self_eval(conn, cfg, "kimi-k3", "nw", "docs_writing", [1.0]) + add_self_eval(conn, cfg, "kimi-k3-fast", "nw", "docs_writing", [0.888]) + conn.execute("ALTER TABLE proficiency DROP COLUMN inherited_from") + + ensure_columns(conn) + + for model_id in ("kimi-k3", "kimi-k3-fast"): + assert conn.execute( + "SELECT inherited_from FROM proficiency WHERE model_id=?", (model_id,) + ).fetchone()["inherited_from"] is None diff --git a/tests/test_routing.py b/tests/test_routing.py index 43398ff..9773083 100644 --- a/tests/test_routing.py +++ b/tests/test_routing.py @@ -6,7 +6,12 @@ proving it rejects and a case proving it does not over-reject. import pytest -from routing import is_eligible, rank_candidates, select_candidates +from routing import ( + is_eligible, + rank_candidates, + rejection_reason, + select_candidates, +) @@ -25,6 +30,8 @@ def _row(**overrides) -> dict: "latency_class": "standard", "reasoning_mode": "default", "context_variant": "full", + "supports_vision": 1, + "supports_json_mode": 1, } row.update(overrides) return row @@ -233,3 +240,211 @@ def test_unmeasured_cost_is_admitted_under_a_ceiling(): def test_ceiling_at_exactly_the_cost_admits(): rows = [_row(model_id="borderline", energy=1e-6, proficiency=0.5)] assert len(rank_candidates(rows, max_energy_per_request=1e-6)) == 1 + + +# --- tool competence is read from the request, not guessed at ------------- + +def test_tools_in_the_request_exclude_a_model_that_overreaches_for_them(): + # deepseek-v4-flash measures 0.33 here. The recorded failure is a + # NON-agentic prompt ("it is 1:20pm, my meeting is at 3pm, how many + # minutes?") where it called two tools instead of subtracting -- so the + # hazard is tools being available, not the task being agentic. + weak = _row(model_id="overreacher", proficiency=1.0, tool_proficiency=0.33) + strong = _row(model_id="reliable", proficiency=1.0, tool_proficiency=1.0) + assert _eligible(weak, min_tool_proficiency=0.5) is False + assert _eligible(strong, min_tool_proficiency=0.5) is True + + +def test_without_tools_the_same_model_is_fine(): + # The filter applies only when the request carries tool definitions; + # otherwise a cheap tool-clumsy model is a perfectly good choice. + weak = _row(model_id="overreacher", tool_proficiency=0.33) + assert _eligible(weak, min_tool_proficiency=None) is True + + +def test_an_unmeasured_model_is_unproven_not_disqualified(): + # Same principle as the tier-1 context gate: absent evidence must not + # decide anything. A model nobody has evaluated for tool use yet is + # unproven, not proven bad. + unknown = _row(model_id="unevaluated", tool_proficiency=None) + assert _eligible(unknown, min_tool_proficiency=0.5) is True + + +def test_the_threshold_is_exclusive_at_the_boundary(): + assert _eligible(_row(tool_proficiency=0.5), min_tool_proficiency=0.5) is True + assert _eligible(_row(tool_proficiency=0.49), min_tool_proficiency=0.5) is False + + +def test_the_tool_filter_can_change_the_winner(): + # The whole point: it is a hard filter, so it removes a candidate that + # would otherwise win on cost rather than merely penalizing it. + rows = [ + _row(model_id="cheap-clumsy", proficiency=1.0, tool_proficiency=0.33, + cost_per_1m_prompt=0.1, cost_per_1m_completion=0.2), + _row(model_id="dearer-reliable", proficiency=1.0, tool_proficiency=1.0, + cost_per_1m_prompt=1.0, cost_per_1m_completion=2.0), + ] + no_tools = select_candidates( + rows, required_context_tokens=1000, required_tier=1, + latency_tolerance="interactive", allowed_access_levels=["public"], + exclude_stale=True, exclude_deprecated=True, min_tool_proficiency=None) + with_tools = select_candidates( + rows, required_context_tokens=1000, required_tier=1, + latency_tolerance="interactive", allowed_access_levels=["public"], + exclude_stale=True, exclude_deprecated=True, min_tool_proficiency=0.5) + + assert rank_candidates(no_tools, prompt_tokens=1000)[0]["model_id"] == "cheap-clumsy" + assert rank_candidates(with_tools, prompt_tokens=1000)[0]["model_id"] == "dearer-reliable" + + +# --- capability filters ---------------------------------------------------- + +def test_vision_required_excludes_a_model_without_vision(): + blind = _row(model_id="blind", supports_vision=0) + sighted = _row(model_id="sighted", supports_vision=1) + assert _eligible(blind, require_vision=True) is False + assert _eligible(sighted, require_vision=True) is True + + +def test_vision_not_required_admits_everyone(): + # The gate is off by default; not asking for vision must not exclude + blind = _row(model_id="blind", supports_vision=0) + assert _eligible(blind, require_vision=False) is True + + +def test_unknown_vision_capability_fails_closed(): + # A capability FLAG that is absent means "cannot confirm". Unlike a + # proficiency MEASUREMENT, admit-on-None would route a vision request to a + # model that might lack it -- a guaranteed provider 400. + unknown = _row(model_id="unknown", supports_vision=None) + assert _eligible(unknown, require_vision=True) is False + + +def test_json_mode_required_excludes_without_support(): + plain = _row(model_id="plain", supports_json_mode=0) + json = _row(model_id="json", supports_json_mode=1) + assert _eligible(plain, require_json_mode=True) is False + assert _eligible(json, require_json_mode=True) is True + + +def test_unknown_json_mode_fails_closed(): + unknown = _row(model_id="unknown", supports_json_mode=None) + assert _eligible(unknown, require_json_mode=True) is False + + +def test_json_mode_gate_admits_supporting_model(): + assert _eligible(_row(supports_json_mode=1), require_json_mode=True) is True + + +def test_the_vision_filter_can_change_the_winner(): + # Same shape as the tool-filter winner test: the gate is a hard filter, so + # it removes a model that would otherwise win on cost rather than merely + # penalizing it. + rows = [ + _row(model_id="cheap-blind", proficiency=1.0, supports_vision=0, + cost_per_1m_prompt=0.1, cost_per_1m_completion=0.2), + _row(model_id="dearer-sighted", proficiency=1.0, supports_vision=1, + cost_per_1m_prompt=1.0, cost_per_1m_completion=2.0), + ] + no_images = select_candidates( + rows, required_context_tokens=1000, required_tier=1, + latency_tolerance="interactive", allowed_access_levels=["public"], + exclude_stale=True, exclude_deprecated=True, require_vision=False) + with_images = select_candidates( + rows, required_context_tokens=1000, required_tier=1, + latency_tolerance="interactive", allowed_access_levels=["public"], + exclude_stale=True, exclude_deprecated=True, require_vision=True) + + assert rank_candidates(no_images, prompt_tokens=1000)[0]["model_id"] == "cheap-blind" + assert rank_candidates(with_images, prompt_tokens=1000)[0]["model_id"] == "dearer-sighted" + + +def test_rejection_reason_names_the_missing_capability(): + assert _reason(_row(supports_vision=0), require_vision=True) == "vision(unsupported)" + assert _reason(_row(supports_vision=None), require_vision=True) == "vision(unknown)" + assert _reason(_row(supports_json_mode=0), require_json_mode=True) == "json_mode(unsupported)" + assert _reason(_row(supports_json_mode=None), require_json_mode=True) == "json_mode(unknown)" + + +def test_zero_tolerance_ranks_strictly_on_quality(): + """"Never trade quality for cost" is a legitimate setting. + + The validator has always accepted 0; `band()` divided by it, so every + request raised ZeroDivisionError on a config that had loaded cleanly. + """ + rows = [_row(model_id="cheap", cost=1e-9, proficiency=0.98), + _row(model_id="dear", cost=1e-3, proficiency=1.00)] + + assert rank_candidates(rows, quality_tolerance=0)[0]["model_id"] == "dear" + + +def test_zero_tolerance_still_breaks_exact_ties_on_cost(): + rows = [_row(model_id="dear", cost=1e-3, proficiency=1.00), + _row(model_id="cheap", cost=1e-9, proficiency=1.00)] + + assert rank_candidates(rows, quality_tolerance=0)[0]["model_id"] == "cheap" + + +# --- rejection reasons ---------------------------------------------------- +# +# is_eligible returns a bool, so a dropped model used to vanish without +# explanation and "no model satisfies the hard filters" was a dead end. The +# reason string is what the debug log prints, so it has to be exact rather than +# re-derived somewhere else. + +def _reason(row, **overrides): + kwargs = { + "required_context_tokens": 10_000, + "required_tier": 2, + "latency_tolerance": "interactive", + "allowed_access_levels": ["public"], + "exclude_stale": True, + "exclude_deprecated": True, + } + kwargs.update(overrides) + return rejection_reason(row, **kwargs) + + +def test_an_eligible_row_has_no_reason(): + assert _reason(_row()) is None + + +def test_the_reason_names_the_filter_and_its_numbers(): + assert _reason(_row(effective_context_window=5_000)) == "context(5000<10000)" + assert _reason(_row(tier=1)) == "tier(1<2)" + assert _reason(_row(availability="stale")) == "stale" + assert _reason(_row(availability="deprecated")) == "deprecated" + assert _reason(_row(access_level="canary")) == "access_level(canary)" + assert _reason(_row(latency_class="flex")) == "latency_class(flex)" + + +def test_unknown_values_say_unknown_rather_than_comparing(): + assert _reason(_row(effective_context_window=None)) == "context(unknown)" + assert _reason(_row(tier=None)) == "tier(unknown)" + + +def test_the_tool_filter_reports_the_measured_score(): + reason = _reason(_row(tool_proficiency=0.33), min_tool_proficiency=0.5) + + assert reason == "tool_proficiency(0.33<0.5)" + + +def test_reasons_are_single_tokens(): + """They go straight into a logfmt value; a space would force quoting.""" + for row in (_row(tier=1), _row(availability="stale"), + _row(access_level="canary"), _row(latency_class="flex"), + _row(supports_vision=0), _row(supports_vision=None), + _row(supports_json_mode=0), _row(supports_json_mode=None)): + reason = _reason(row, require_vision=True, require_json_mode=True) + assert reason is not None + assert " " not in reason + + +def test_is_eligible_still_agrees_with_the_reason(): + """One copy of the rules, two views of it.""" + for row in (_row(), _row(tier=1), _row(latency_class="flex")): + assert is_eligible(row, **{ + "required_context_tokens": 10_000, "required_tier": 2, + "latency_tolerance": "interactive", "allowed_access_levels": ["public"], + "exclude_stale": True, "exclude_deprecated": True, + }) == (_reason(row) is None) diff --git a/tests/test_seed_sweep.py b/tests/test_seed_sweep.py new file mode 100644 index 0000000..e0fd34e --- /dev/null +++ b/tests/test_seed_sweep.py @@ -0,0 +1,142 @@ +"""End-to-end test of the reference sweep, with the provider stubbed. + +The sweep is what makes `eco` and the per-request energy ceiling real numbers +instead of the neutral 0.5, and `llm-router-seed.timer` runs it every six +hours. It had been dead for weeks: `log_observation` grew `request_id`, +`session_key` and `session_dir` in the middle of its signature and made the +trailing three keyword-only, while this call still passed six positionals. The +result was a TypeError -- not a RequestException, so the per-sample `except` +never caught it -- raised AFTER the first billed completion. Every scheduled +run spent money and wrote nothing. + +So this is a smoke test of `main()` rather than an assertion about a +signature: it catches this class of drift wherever it next appears, and it +costs nothing to run because `sample_once` never leaves the process. +""" + +import sqlite3 +import sys +from pathlib import Path + +import pytest + +import dispatcher +import seed_energy + +ROOT = Path(__file__).resolve().parent.parent +SCHEMA_SQL = (ROOT / "schema.sql").read_text() + +MODEL = "gemma-4-31b" + + +def _payload(model_id): + return { + "id": "chatcmpl-seed-1", + "model": model_id, + "choices": [{"message": {"content": "A B-tree is..."}, + "finish_reason": "length"}], + "usage": {"prompt_tokens": 31, "completion_tokens": 400}, + "energy": {"energy_kwh": 4.75e-05, "avg_power_watts": 420.0, + "duration_seconds": 1.6, "attribution_ratio": 0.25, + "carbon_g_co2eq": 2.32e-04, "carbon_source": "agent_cache", + "grid_id": "FI"}, + "cost": {"request_cost_usd": 3.8e-04, "allowance_remaining_usd": 49.5}, + } + + +@pytest.fixture +def sweep(tmp_path, monkeypatch): + db_path = tmp_path / "test.db" + conn = sqlite3.connect(db_path) + conn.executescript(SCHEMA_SQL) + conn.execute( + """ + INSERT INTO models ( + model_id, provider, tier, latency_class, access_level, + availability, cost_per_1m_completion, last_updated + ) VALUES (?, 'neuralwatt', 1, 'standard', 'public', 'active', 0.42, + '2026-08-22T00:00:00+00:00') + """, + (MODEL,), + ) + conn.commit() + conn.close() + + # main() loads its own config; log_observation reads dispatcher's. They must + # agree on the database or the sweep writes somewhere the test cannot see. + monkeypatch.setattr(dispatcher.cfg.database, "path", str(db_path)) + monkeypatch.setattr(seed_energy, "load_config", lambda path: dispatcher.cfg) + monkeypatch.setenv("NEURALWATT_API_KEY", "test-key") + monkeypatch.setattr(seed_energy.time, "sleep", lambda _: None) + + calls = [] + + def fake_sample(base_url, api_key, model_id, max_tokens=400, timeout=300): + calls.append({"model_id": model_id, "max_tokens": max_tokens}) + return _payload(model_id) + + monkeypatch.setattr(seed_energy, "sample_once", fake_sample) + yield calls, db_path + + +def _run(monkeypatch, *argv): + monkeypatch.setattr(sys, "argv", ["seed_energy.py", *argv]) + return seed_energy.main() + + +def test_a_sweep_writes_the_observation_it_paid_for(sweep, monkeypatch): + """The regression: this raised TypeError after the first billed call.""" + calls, db_path = sweep + + assert _run(monkeypatch, "--samples", "2") == 0 + assert len(calls) == 2 + + conn = sqlite3.connect(db_path) + conn.row_factory = sqlite3.Row + rows = conn.execute( + "SELECT * FROM energy_observations WHERE task_category = ?", + (seed_energy.SEED_CATEGORY,), + ).fetchall() + conn.close() + + assert len(rows) == 2 + row = rows[0] + assert row["model_id"] == MODEL + assert row["prompt_tokens"] == 31 + assert row["completion_tokens"] == 400 + assert row["energy_kwh"] == pytest.approx(4.75e-05) + assert row["carbon_g_co2eq"] == pytest.approx(2.32e-04) + assert row["cost_usd"] == pytest.approx(3.8e-04) + + +def test_the_sweep_records_the_completion_id(sweep, monkeypatch): + """Positional drift silently dropped request_id; it is the join key for /outcome.""" + _, db_path = sweep + _run(monkeypatch, "--samples", "1") + + conn = sqlite3.connect(db_path) + request_id = conn.execute( + "SELECT request_id FROM energy_observations" + ).fetchone()[0] + conn.close() + + assert request_id == "chatcmpl-seed-1" + + +def test_max_tokens_reaches_the_request(sweep, monkeypatch): + """The flag was parsed, printed in the banner, and never sent.""" + calls, _ = sweep + _run(monkeypatch, "--samples", "1", "--max-tokens", "800") + + assert calls[0]["max_tokens"] == 800 + + +def test_a_dry_run_spends_nothing(sweep, monkeypatch): + calls, db_path = sweep + + assert _run(monkeypatch, "--dry-run") == 0 + assert not calls + + conn = sqlite3.connect(db_path) + assert conn.execute("SELECT COUNT(*) FROM energy_observations").fetchone()[0] == 0 + conn.close()