From c3484f05e52853027e0b9c34ec5c2d5d55ec09d6 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Wed, 12 Aug 2026 00:03:43 -0400 Subject: [PATCH 01/20] feat: retarget router to NeuralWatt-only, add serving-class routing and an OpenAI-compatible dispatcher MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drops OpenRouter entirely. The provider column and (model_id, provider) key stay so a second provider needs no migration. Verified against the live API — the poller's field mappings were previously unconfirmed guesses and turned out correct. Routing correctness: - Tier on metadata.reasoning.default_enabled, not capabilities.reasoning. The latter only means "the endpoint accepts a reasoning param" and is true for 17 of 19 rows, which put 17 models in tier 3 and left tier 1 empty. Cost is now checked before the reasoning rule so $0.28/1M models can reach tier 1. Distribution goes from 2/17 to 4/6/9. - Capture serving class. NeuralWatt ships ~6 base models as 19 rows whose id suffixes are three orthogonal dimensions (hence glm-5.2-short-fast-flex): -flex is discounted async held during peak, -fast is reasoning disabled or capped, -short is a 200K pool. They carry identical catalog pricing, so without these columns all 7 GLM rows tie exactly and an interactive request could land on a preemptible row. Latency tolerance is a hard filter, not a weight. Suffixes match whole segments so deepseek-v4-flash is not read as a -fast row. - Exclude access-gated models. 6 of 19 rows are grant-gated or canary, marked only in prose, and would 403 at dispatch. - Log the provider's real billed cost and carbon rather than a tokens x list-price estimate, and score eco on carbon per design doc §4. New dispatcher.py exposes /health, /route (dry run, no spend), /dispatch, plus an OpenAI-compatible /v1/models and /v1/chat/completions so any normal client can use it. Streaming is proxied chunk by chunk; NeuralWatt emits energy and cost as SSE comment lines, which clients ignore and the router reads on the way past — otherwise streamed calls would log no energy at all. Classifier now gets the allowed category list injected from config (it was returning invented labels that join against nothing) and runs at temperature 0, because the same prompt was classifying tier 2 then tier 1 and routing to different models. Ships systemd user units. The poller timer is load-bearing, not housekeeping: stale_after_days is 3 with exclude_stale true, so an unpolled catalog eventually marks every row stale and the router returns no candidates at all. Documents the finding that most affects this project: NeuralWatt bills a flat $8.00/kWh, not per token. List price ranks models backwards — on the same prompt kimi-k2.7-code-fast ($4/1M) cost 10x more than kimi-k3-fast ($15/1M). scoring.cost_score still reads list price; re-basing it is the open call. Tests 28 -> 74. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- .env.example | 1 - CLAUDE.md | 314 ++++++++++--- config.py | 38 ++ config.yaml | 47 +- deploy/README.md | 81 ++++ deploy/llm-router-poller.service | 21 + deploy/llm-router-poller.timer | 15 + deploy/llm-router.service | 34 ++ design/local-llm-model-router.md | 106 ++++- dispatcher.py | 730 +++++++++++++++++++++++++++++++ opencode.json | 30 ++ poller.py | 199 +++++---- pyproject.toml | 7 + requirements.txt | 20 +- routing.py | 164 +++++++ schema.sql | 48 +- tests/test_apply_tiering.py | 56 ++- tests/test_poller_parsing.py | 78 ++++ tests/test_routing.py | 228 ++++++++++ tests/test_scoring.py | 2 +- tests/test_tiering.py | 106 ++++- tier.py | 32 +- tiering.py | 71 ++- 23 files changed, 2166 insertions(+), 262 deletions(-) create mode 100644 deploy/README.md create mode 100644 deploy/llm-router-poller.service create mode 100644 deploy/llm-router-poller.timer create mode 100644 deploy/llm-router.service create mode 100644 dispatcher.py create mode 100644 opencode.json create mode 100644 pyproject.toml create mode 100644 routing.py create mode 100644 tests/test_poller_parsing.py create mode 100644 tests/test_routing.py diff --git a/.env.example b/.env.example index e36502d..21c4fd4 100644 --- a/.env.example +++ b/.env.example @@ -1,3 +1,2 @@ # Copy to .env and fill in. Never commit the real .env file. -OPENROUTER_API_KEY= NEURALWATT_API_KEY= diff --git a/CLAUDE.md b/CLAUDE.md index c13d642..9ff7a16 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,10 +8,14 @@ This file is the working state + immediate next steps. A router that uses a local model (served via Ollama on an RTX 6000, 24GB) to classify incoming coding/documentation tasks — category, tier, required context size — and dispatch each task to the cheapest/best-fit open-weight -model across OpenRouter and Neuralwatt Cloud, weighted by cost, ecological -impact (Neuralwatt exposes real energy-per-request data), and per-category +model on **Neuralwatt Cloud**, weighted by cost, ecological impact +(Neuralwatt exposes real energy-per-request data), and per-category proficiency. +Neuralwatt is the only provider. OpenRouter was removed — the `provider` +column and the `(model_id, provider)` primary key stay so a second provider +can be added later without a migration. + ## Stack - Python (chosen over Rust — this is I/O-bound against provider APIs, not @@ -20,90 +24,268 @@ proficiency. - SQLite for the decision table - Ollama (OpenAI-compatible endpoint at `localhost:11434/v1`) for local classification -- FastAPI planned for the dispatcher service (not yet built — see below) +- FastAPI for the dispatcher service + +## Billing is per-kWh, not per-token — this invalidates `cost_score` + +**Measured against the live API, 2026-08-11.** Neuralwatt bills a flat +**$8.00 per kWh** and the catalog's `input_per_million` / +`output_per_million` prices are not what this account is charged. Confirmed +across five models; `cost_usd / energy_kwh` came back 8.00 every time: + +| model | list $/1M out | completion tokens | billed USD | kWh | $/kWh | +|---|---|---|---|---|---| +| deepseek-v4-flash | 0.28 | 600 | 5.00e-06 | 5.88e-07 | 8.50\* | +| gemma-4-31b | 0.42 | 540 | 3.80e-04 | 4.75e-05 | 8.00 | +| qwen3.6-35b-fast | 1.15 | 600 | 2.53e-04 | 3.16e-05 | 8.01 | +| kimi-k2.7-code-fast | 4.00 | 600 | 2.25e-03 | 2.81e-04 | 8.00 | +| kimi-k3-fast | 15.00 | 539 | 2.17e-04 | 2.71e-05 | 8.00 | + +\* rounding — billed cost is quantized to ~1e-06. + +Two consequences, both load-bearing: + +1. **List price ranks models wrong.** `kimi-k2.7-code-fast` lists at $4/1M + and `kimi-k3-fast` at $15/1M, but on the same prompt the "cheap" one cost + **10x more** ($2.25e-03 vs $2.17e-04) because it burned 10x the energy. + `scoring.cost_score` currently reads `cost_per_1m_completion`, so it is + ranking on a number nobody is billed. +2. **`cost` and `eco` are the same axis.** `cost_usd = 8.00 x energy_kwh` + exactly, so `w_cost` (0.4) and `w_eco` (0.2) put 0.6 of the weight on one + underlying signal and leave proficiency at 0.4. The balance in + `config.yaml` is not the balance in effect. They separate only via grid + carbon intensity (`grid_carbon_intensity_gco2perkwhr`, currently 37.0 on + grid `FI`), which matters only when routing across regions. + +Also: energy for the *same* token count varies up to 4x run-to-run (same +18->900 tokens billed 1.8e-05 through 8.8e-05), so a single observation is +not a reliable per-model estimate — this needs several samples per model +before it can drive routing. + +**Not yet acted on.** Deciding what replaces `cost_score` is the open design +call, see "Pick up here" below. ## What's built and working -- `schema.sql` — `models`, `proficiency`, `energy_observations` tables. - Applies cleanly (`sqlite3 db/router.db < schema.sql`). -- `poller.py` — fetches OpenRouter's and Neuralwatt's `/models` endpoints - (both public/unauthenticated), normalizes pricing/context/capabilities, - upserts into `models`, marks stale rows. Syntax-validated but **not yet - run against live APIs** — this sandbox couldn't reach openrouter.ai or - api.neuralwatt.com, so the field-mapping assumptions (especially - OpenRouter's `supported_parameters` list and Neuralwatt's - `metadata.pricing`/`metadata.capabilities` shape) need to be confirmed - against real payloads on first run. Fix field paths if the actual JSON - differs. -- `config.yaml` / `config.py` — all weights, thresholds, provider settings. - Pydantic-validated (weights must sum to 1.0, etc.), confirmed loading - correctly. +- `schema.sql` — `models`, `proficiency`, `energy_observations`. Applies + cleanly (`sqlite3 router.db < schema.sql`). `models` carries the serving + class columns (below); `energy_observations` carries real carbon/cost. +- `poller.py` — fetches Neuralwatt's `/models` endpoint (public, + unauthenticated), normalizes, upserts, marks stale rows. **Verified + against the live API**: 19 models, and the `metadata.pricing` / + `metadata.capabilities` / `metadata.limits` field mappings are confirmed + correct. +- `config.yaml` / `config.py` — weights, thresholds, provider settings, + Pydantic-validated. +- `scoring.py` — pure min-max normalize-and-invert for cost/eco, plus the + weighted composite. (Correct as written; the *input* is the problem — see + the billing section.) +- `tiering.py` / `tier.py` — pure tier resolver + the DB pass that applies it. +- `routing.py` — pure hard filters and ranking. +- `dispatcher.py` — FastAPI service. `GET /health`, `POST /route` (classify + and pick, no provider call), `POST /dispatch` (route, call, log). +- `tests/` — 74 tests, all passing. + +### Serving class: one base model, many rows + +Neuralwatt ships ~6 base models as 19 catalog rows. The id suffixes are three +**orthogonal** dimensions (hence `glm-5.2-short-fast-flex`), parsed by +`poller.parse_serving_class` into columns: + +| suffix | column | meaning | +|---|---|---| +| `-flex` | `latency_class` | discounted async; held server-side during peak until a capacity gap opens | +| `-fast` | `reasoning_mode` | thinking disabled or capped to a short budget — *not* queue priority | +| `-short` | `context_variant` | 200K pool with a bounded reasoning budget | + +These rows carry **identical catalog pricing**, so without these columns all +7 GLM rows tie exactly and the router picks arbitrarily — which could send an +interactive request to a preemptible flex row. `latency_tolerance` +(`interactive` | `batch`) is therefore a hard filter in `routing.py`, not a +weight. Suffixes are matched as whole `-`-delimited segments so +`deepseek-v4-flash` is not misread as a `-fast` row. + +### Access gating is prose-only + +6 of 19 rows are restricted, and the catalog says so only in free text +("Private preview (grant-gated)", "(Canary)") — there is no structured field. +`poller.parse_access_level` parses it into `access_level`, and +`routing.allowed_access_levels` (default `[public]`) excludes them, so the +dispatcher doesn't select a model that 403s. 13 of 19 rows are routable. + +### Tiering + +Tier on `reasoning_default_enabled` (from `metadata.reasoning.default_enabled`, +falling back to `capabilities.reasoning`), **not** `supports_reasoning`. +`capabilities.reasoning` only means "the endpoint accepts a reasoning +param" — it is true for 17 of 19 rows, and tiering on it put 17 models in +tier 3 and left tier 1 empty. A `-fast` row does not inherit its sibling's +tier 3. Cost is checked before the reasoning rule so $0.28/1M models can +reach tier 1. Current distribution: **4 / 6 / 9**. + +## The classifier is the latency floor + +Every routed request pays a full local classification round-trip before a +single upstream token is requested. Measured on `qwen3.5:latest`: + +| | latency | +|---|---| +| cold (model not resident) | **43s** — exceeded the old 30s ceiling and returned 503 | +| warm | **5–10s** | + +`timeout_seconds` is now 120 so a cold load no longer 503s, and +`temperature` is 0 because at the default the same prompt classified tier 2 +then tier 1 on consecutive calls and routed to two different models — +routing that moves under an identical prompt can't be reasoned about. + +But ~10s of local overhead on every message is a real tax for an interactive +agent, where the upstream answer itself may take 2s. Unaddressed options: +keep Ollama resident (`OLLAMA_KEEP_ALIVE`), classify once per session rather +than per message, cache by prompt hash, use a smaller classifier, or skip +classification for short prompts. This is a design call, not a tuning one. + +Note also that opencode sends **~32K prompt tokens** of system prompt and +tool definitions on a trivial request, so the measured-size floor in +`estimate_prompt_tokens` does real work — the classifier's own estimate for +that request was two orders of magnitude low. ## What's NOT built yet — pick up here -1. **FastAPI dispatcher** (`dispatcher.py`, doesn't exist yet). Should: - - Accept a task (text + optional context/file references) - - Call Ollama with `config.classifier.system_prompt`, parse the JSON - response (`task_category`, `task_tier`, `required_context_tokens`, - `confidence`) - - If `confidence < config.escalation.min_confidence_before_bump`, bump - required tier by one (per config) - - Query `models` + `proficiency` tables: hard-filter on - `effective_context_window >= required_context_tokens`, - `tier >= required_tier`, `availability == 'active'` (respecting - `freshness.exclude_stale`/`exclude_deprecated`) - - Score remaining candidates: `w_cost * cost_score + w_eco * eco_score + - w_prof * proficiency_score[task_category]` — cost_score and eco_score - need a normalization function (e.g. min-max across candidate set, - inverted since lower cost/carbon = better) that doesn't exist yet - - Dispatch to the winning (model, provider) via that provider's - OpenAI-compatible endpoint - - Log the real response's energy/cost into `energy_observations` - (`energy_btu` = `energy_kwh * 3412.14`, yes really, see design doc history) +1. **Replace `cost_score`'s input** (see the billing section). It currently + reads catalog list price, which is not billed. Options: score on observed + mean energy per model (needs seeding, and several samples each given the + 4x variance), or drop `cost` as a separate axis and let `eco` carry it, + re-weighting `config.yaml` accordingly. This is a design call, not a + mechanical fix. -2. **Energy/cost normalization functions** — cost_score and eco_score are - referenced in the design doc's scoring formula but the actual - min-max-normalize-and-invert logic isn't written anywhere yet. +2. **Benchmark/proficiency poller** — nothing populates `proficiency` yet, so + `proficiency_score` returns the neutral 0.5 for every candidate and 0.4 of + the weight is currently inert. Needs a leaderboard-scraping path (LMSYS + Arena, LiveBench, Aider polyglot — no unified API) and a self-eval harness + writing `source='self_eval'`. Blending rule is already in `config.yaml`. -3. **Benchmark/proficiency poller** — nothing populates the `proficiency` - table yet. Needs both a leaderboard-scraping path (LMSYS Arena, LiveBench, - Aider polyglot — no unified API, likely manual/periodic scripted pulls) - and a self-eval harness (runs a fixed task set against candidate models, - scores results, writes to `proficiency` with `source='self_eval'`). - Blending rule is in `config.yaml` (`proficiency.leaderboard_weight` / - `self_eval_weight`, gated on `self_eval_min_samples`). +3. **Energy seeding sweep** — `energy_observations` starts empty, so `eco` is + also neutral 0.5 until real traffic accrues. A fixed prompt run N times + against each routable model would seed it; the ad-hoc version of this is + what produced the $8/kWh table above. -4. **Escalation feedback loop** — a way for a human or downstream agent to - flag "this was routed wrong" and have it retry one tier up. Not designed - in detail yet beyond the config toggle. +4. **Escalation feedback loop** — low-confidence tier bumping works + (`apply_escalation`), but there's no way for a human or downstream agent to + flag "this was routed wrong" after the fact and retry one tier up. -5. **Scheduling** — poller.py needs a cron/systemd timer, not yet set up. - No script for that exists here. +5. **Scheduling** — `poller.py` needs a cron/systemd timer. No unit file here. -## Known open questions (from design doc, still unresolved) +## Known open questions -- How much context-assembly (RAG-style doc/code retrieval) should live in - the classifier step vs. a separate pre-step? Leaning toward decoupled but - not decided. -- Self-run eval set: what's the minimum viable task set to start proficiency - scoring without it becoming its own maintenance burden? -- Whether eco_score should use Neuralwatt's real-time grid carbon intensity - per-request or a stable per-model average — real-time is more accurate but - adds volatility to routing decisions. +- Given billing is per-kWh, is `cost` still a meaningful separate axis from + `eco`, or should the weights collapse to energy + proficiency? +- Energy per request varies up to 4x for identical token counts. How many + samples per model before an energy estimate is trustworthy enough to route + on? +- How much context-assembly (RAG-style retrieval) belongs in the classifier + step vs. a separate pre-step? Leaning decoupled, undecided. +- Self-run eval set: minimum viable task set to start proficiency scoring + without it becoming its own maintenance burden? +- Should `eco_score` use real-time grid carbon intensity per request or a + stable per-model average? `grid_carbon_intensity` and `grid_id` are logged + per observation, so this stays answerable from data either way. ## Setup ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt -sqlite3 db/router.db < schema.sql -cp .env.example .env # fill in OPENROUTER_API_KEY / NEURALWATT_API_KEY -python poller.py # first live run — verify field mappings against real API responses +sqlite3 router.db < schema.sql +cp .env.example .env # fill in NEURALWATT_API_KEY +python poller.py # populate the catalog +python tier.py # resolve tiers python config.py # sanity-check config loads +python -m uvicorn dispatcher:app --reload ``` -Ollama must be running locally with a classifier model pulled, e.g.: +Ollama must be running locally with the classifier model pulled — the name +must match `classifier.model` in `config.yaml`: ```bash -ollama pull qwen2.5:32b +ollama pull qwen3.5:latest ``` -(model name must match `classifier.model` in `config.yaml`) + +`requirements.txt` is pinned. Recreating the venv with the old `>=` ranges +jumped openai 2.53 -> 3.0 and httpx -> httpx2 without warning; a service that +restarts on boot shouldn't change its dependency tree underneath itself. +Bump deliberately. + +`pyproject.toml` puts the repo root on `sys.path` for pytest — the modules +live at the root rather than in a package, so `pytest` (console script) and +`python -m pytest` would otherwise disagree about whether `import config` +resolves. + +## Run as a service + +`deploy/` holds three systemd **user** units — see `deploy/README.md` for +install and operation. In short: + +```bash +echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env +cp deploy/llm-router*.{service,timer} ~/.config/systemd/user/ +systemctl --user daemon-reload +systemctl --user enable --now llm-router.service llm-router-poller.timer +``` + +The dispatcher binds `127.0.0.1:8080`. **The poller timer is load-bearing, +not housekeeping**: `stale_after_days` is 3 with `exclude_stale: true`, so an +unpolled catalog eventually marks every row stale and the router returns zero +candidates for everything. + +The service holds a billable API key and has **no auth of its own**. Loopback +bind is the only thing standing between the open internet and your allowance; +add auth before widening `--host`. + +## Pointing a coding agent at it + +The `/v1` endpoints are OpenAI-compatible, so any normal client works — +opencode, an SDK, plain curl. Repo-local `opencode.json` is already wired up, +so `cd ~/Sources/6krrt && opencode` routes by default. For global use, merge +`provider.llm-router` into `~/.config/opencode/opencode.json`. + +| model name | behavior | +|---|---| +| `auto` | router picks; flex rows excluded so nothing is held during peak | +| `auto:batch` | router picks; flex rows admitted, for overnight/async work | +| any real model id | dispatched as asked, still logged | + +Streaming is proxied chunk by chunk rather than buffered, so tokens still +render as they arrive. NeuralWatt emits its energy and cost blocks as SSE +**comment** lines (`: energy {...}`) before `data: [DONE]` — ordinary clients +ignore comments, so the stream passes through untouched while the router +reads the telemetry on the way past. Without that, streamed calls would log +no energy at all, which is most of the point of this project. + +## Try it + +```bash +# Pick a model without spending anything +curl -s -X POST localhost:8080/route -H 'content-type: application/json' \ + -d '{"task":"Refactor this 800-line Django view into service objects."}' + +# Same, but admit flex rows (held during peak, fine for overnight work) +curl -s -X POST localhost:8080/route -H 'content-type: application/json' \ + -d '{"task":"nightly code review","latency_tolerance":"batch"}' + +# Actually call the winner and log energy/cost +curl -s -X POST localhost:8080/dispatch -H 'content-type: application/json' \ + -d '{"task":"What is a Python context manager?"}' + +# The OpenAI-compatible surface any client can use +curl -s localhost:8080/v1/models +curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \ + -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}' + +# What did routing actually cost and burn? +sqlite3 -header -column router.db \ + "SELECT model_id, prompt_tokens, completion_tokens, energy_kwh, carbon_g_co2eq, cost_usd + FROM energy_observations ORDER BY id DESC LIMIT 10;" +``` + +`/route` takes `task_category`, `task_tier`, and `required_context_tokens` +overrides, which skip the classifier — useful for testing routing changes +deterministically. diff --git a/config.py b/config.py index fe8d54f..c0ace7a 100644 --- a/config.py +++ b/config.py @@ -85,6 +85,42 @@ class TieringConfig(BaseModel): return v +class RoutingConfig(BaseModel): + allowed_access_levels: list[str] + default_latency_tolerance: str + flex_cost_multiplier: float + + @field_validator("allowed_access_levels") + @classmethod + def levels_known(cls, v: list[str]) -> list[str]: + known = {"public", "preview", "canary"} + unknown = set(v) - known + if unknown: + raise ValueError( + f"routing.allowed_access_levels contains unknown levels {sorted(unknown)}; " + f"must be a subset of {sorted(known)}" + ) + if not v: + raise ValueError("routing.allowed_access_levels must not be empty") + return v + + @field_validator("default_latency_tolerance") + @classmethod + def tolerance_known(cls, v: str) -> str: + if v not in ("interactive", "batch"): + raise ValueError( + f"routing.default_latency_tolerance must be 'interactive' or 'batch', got {v!r}" + ) + return v + + @field_validator("flex_cost_multiplier") + @classmethod + def multiplier_in_range(cls, v: float) -> float: + if not (0.0 < v <= 1.0): + raise ValueError("routing.flex_cost_multiplier must be in (0, 1]") + return v + + class EscalationConfig(BaseModel): enabled: bool max_tier: int @@ -106,6 +142,7 @@ class ClassifierConfig(BaseModel): base_url: str model: str timeout_seconds: int + temperature: float = 0.0 response_format: str system_prompt: str @@ -126,6 +163,7 @@ class RouterConfig(BaseModel): tiers: dict[int, str] tiering: TieringConfig proficiency: ProficiencyConfig + routing: RoutingConfig escalation: EscalationConfig freshness: FreshnessConfig database: DatabaseConfig diff --git a/config.yaml b/config.yaml index 02de43e..4369cfd 100644 --- a/config.yaml +++ b/config.yaml @@ -55,6 +55,26 @@ escalation: # required_tier by one as a precaution rather than trusting a shaky call. min_confidence_before_bump: 0.6 +routing: + # Access gating is prose-only in the NeuralWatt catalog ("Private preview + # (grant-gated)", "(Canary)"), so the poller parses it into access_level and + # routing excludes anything not listed here. Add 'preview'/'canary' only if + # the account actually holds the grant — otherwise dispatch earns a 403. + allowed_access_levels: + - public + + # '-flex' rows are held server-side during peak until a capacity gap opens. + # That's correct for overnight/batch agent work and wrong for anything + # interactive, so a request has to opt in via latency_tolerance. + default_latency_tolerance: interactive # 'interactive' | 'batch' + + # The catalog advertises flex at the SAME sticker price as standard serving, + # even though flex describes itself as "billed at a reduced rate". Until that + # discount is confirmed against a real invoice this stays 1.0 rather than + # inventing a discount the scorer would then act on. Set it below 1.0 once + # you have billing data. + flex_cost_multiplier: 1.0 + freshness: stale_after_days: 3 # Router refuses to route to a model whose row is stale/deprecated, @@ -69,23 +89,36 @@ classifier: # Local Ollama instance doing task classification / context sizing. provider: "ollama" base_url: "http://localhost:11434/v1" - model: "qwen2.5:32b" - timeout_seconds: 30 + model: "qwen3.5:latest" # must match a model `ollama list` reports + # A cold Ollama took 43s to answer the first classification, which blew the + # old 30s ceiling and returned 503 to the client. Warm it is ~5s. The + # ceiling is for a cold model load, not the steady state. + timeout_seconds: 120 + # Classification must be reproducible: at the default temperature the same + # task was classified tier 2 then tier 1 on consecutive calls, which routed + # it to two different models. Routing that changes under an identical + # prompt is untraceable. + temperature: 0 response_format: "json" # ask Ollama to constrain output to valid JSON + # The dispatcher appends the authoritative category list from + # proficiency.categories to this prompt at call time. Do not enumerate the + # categories here as well — a hand-copied list drifts, and a category the + # model invents joins against nothing in the proficiency table. system_prompt: | You are a task router. Given a task description and any attached context, respond with ONLY a JSON object with these fields: { - "task_category": one of the configured categories, - "task_tier": integer 1-3, + "task_category": one of the allowed categories listed below, + "task_tier": integer 1-3, where 1 is cheap/simple, 2 is mid/general, + and 3 is frontier/high-stakes, "required_context_tokens": integer estimate of prompt+context token count, "confidence": float 0-1 } dispatch_providers: - openrouter: - base_url: "https://openrouter.ai/api/v1" - api_key_env: "OPENROUTER_API_KEY" + # NeuralWatt is the only provider. The dict shape and the models table's + # (model_id, provider) key are kept so a second one can be added without a + # migration. neuralwatt: base_url: "https://api.neuralwatt.com/v1" api_key_env: "NEURALWATT_API_KEY" diff --git a/deploy/README.md b/deploy/README.md new file mode 100644 index 0000000..185892c --- /dev/null +++ b/deploy/README.md @@ -0,0 +1,81 @@ +# Deploying the router + +Three units. The dispatcher runs continuously; the poller runs on a timer and +is **not** optional — `freshness.stale_after_days` is 3 and +`freshness.exclude_stale` is true, so a catalog that goes unpolled for three +days marks every row stale and the router stops returning any candidate at +all. + +| file | what it does | +|---|---| +| `llm-router.service` | the FastAPI dispatcher, on `127.0.0.1:8080` | +| `llm-router-poller.service` | one-shot: `poller.py` then `tier.py` | +| `llm-router-poller.timer` | fires the poller 2 min after boot, then every 2 h | + +These are **user** units — no root, and they run as you with your own +`$HOME`. The tradeoff is that a user service does not inherit your shell +environment, so the API key has to come from a file. + +## Install + +```bash +# 1. The key. User units don't see your shell env, so .env is required. +cd ~/Sources/6krrt +echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env + +# 2. Install and start +cp deploy/llm-router*.{service,timer} ~/.config/systemd/user/ +systemctl --user daemon-reload +systemctl --user enable --now llm-router.service llm-router-poller.timer + +# 3. Check +curl -s localhost:8080/health | python -m json.tool +systemctl --user list-timers llm-router-poller.timer +``` + +To survive logout without an active session, enable lingering: + +```bash +loginctl enable-linger "$USER" +``` + +## Operating it + +```bash +systemctl --user status llm-router.service +journalctl --user -u llm-router.service -f # request log +journalctl --user -u llm-router-poller.service # catalog refreshes +systemctl --user restart llm-router.service # after editing config.yaml +systemctl --user start llm-router-poller.service # force a refresh now +``` + +`config.yaml` is read once at startup, so weight and threshold changes need a +restart. The catalog is read per-request, so a poller run takes effect +immediately. + +## A note on the bind address + +`--host 127.0.0.1` is deliberate. The service holds a billable API key and +has **no authentication of its own** — anything that reaches it can spend +your allowance. `ProtectHome=read-only` plus a `ReadWritePaths` exception for +the repo limits the blast radius on the filesystem, but nothing limits spend. +Putting this on a LAN address needs an auth layer first. + +## Pointing opencode at it + +The repo-local `opencode.json` sets this up already, so running `opencode` +from inside `~/Sources/6krrt` uses the router by default. To use it from +anywhere, merge the `provider.llm-router` block into +`~/.config/opencode/opencode.json` and set `"model": "llm-router/auto"`. + +Two model names: + +- `llm-router/auto` — normal routing; flex rows excluded, so nothing gets + held server-side during peak +- `llm-router/auto:batch` — admits flex rows, for overnight/async work + +`limit.context` is declared as 782324, the largest effective window in the +routable catalog. The router hard-filters on the measured conversation size, +so a prompt too big for the smaller models simply won't be routed to them; +if it fits nothing, `/v1/chat/completions` returns a 422 naming the +constraint rather than truncating. diff --git a/deploy/llm-router-poller.service b/deploy/llm-router-poller.service new file mode 100644 index 0000000..b8a0887 --- /dev/null +++ b/deploy/llm-router-poller.service @@ -0,0 +1,21 @@ +[Unit] +Description=Refresh the NeuralWatt model catalog and re-resolve tiers +Documentation=file:%h/Sources/6krrt/CLAUDE.md +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +WorkingDirectory=%h/Sources/6krrt +EnvironmentFile=%h/Sources/6krrt/.env +# poller.py refreshes the catalog; tier.py re-resolves tiers from it. Tiers +# are derived from cost and reasoning fields the poll may have changed, so +# they always run as a pair. +ExecStart=%h/Sources/6krrt/.venv/bin/python poller.py +ExecStart=%h/Sources/6krrt/.venv/bin/python tier.py + +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ProtectHome=read-only +ReadWritePaths=%h/Sources/6krrt diff --git a/deploy/llm-router-poller.timer b/deploy/llm-router-poller.timer new file mode 100644 index 0000000..668ce20 --- /dev/null +++ b/deploy/llm-router-poller.timer @@ -0,0 +1,15 @@ +[Unit] +Description=Refresh the NeuralWatt model catalog periodically + +[Timer] +OnBootSec=2min +OnUnitActiveSec=2h +# Catch up after the machine has been asleep rather than waiting a full +# interval — this is not optional maintenance. freshness.stale_after_days is +# 3 and freshness.exclude_stale is true, so a catalog that goes unpolled for +# three days marks every row stale and the router stops returning ANY +# candidate. The timer is what keeps the service able to route at all. +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/deploy/llm-router.service b/deploy/llm-router.service new file mode 100644 index 0000000..3e27d01 --- /dev/null +++ b/deploy/llm-router.service @@ -0,0 +1,34 @@ +[Unit] +Description=Local LLM model router (FastAPI dispatcher) +Documentation=file:%h/Sources/6krrt/CLAUDE.md +# The classifier talks to Ollama on localhost and the dispatcher talks to +# NeuralWatt, so both loopback and real network need to be up. Ollama is a +# system service and cannot be ordered against from a user unit, so a failed +# classifier call is handled by Restart= below rather than by ordering. +After=network-online.target +Wants=network-online.target + +[Service] +Type=exec +# config.yaml, router.db and router.log are all referenced as relative paths, +# so this has to be the repo root. +WorkingDirectory=%h/Sources/6krrt +# Holds NEURALWATT_API_KEY. Create it with: +# echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env +EnvironmentFile=%h/Sources/6krrt/.env +ExecStart=%h/Sources/6krrt/.venv/bin/uvicorn dispatcher:app --host 127.0.0.1 --port 8080 + +Restart=on-failure +RestartSec=5s + +# Loopback only by default — this service holds a billable API key and has no +# auth of its own. Anything that widens --host should add auth first. +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ProtectHome=read-only +# ...except the repo, which needs to be writable for router.db and router.log. +ReadWritePaths=%h/Sources/6krrt + +[Install] +WantedBy=default.target diff --git a/design/local-llm-model-router.md b/design/local-llm-model-router.md index 684d344..9bd2532 100644 --- a/design/local-llm-model-router.md +++ b/design/local-llm-model-router.md @@ -2,7 +2,13 @@ **Status:** design draft **Owner:** Aaron Lee -**Hardware:** RTX 6000 24GB (local classifier/context-assembly), routes to OpenRouter + Neuralwatt Cloud +**Hardware:** RTX 6000 24GB (local classifier/context-assembly), routes to Neuralwatt Cloud + +> **Revised 2026-08-11 after first live run.** Neuralwatt is now the only provider (OpenRouter +> dropped). Two findings from real API traffic change this document materially: billing is +> **per-kWh, not per-token** (§4.1), and one base model appears as several catalog rows that +> differ only by serving class (§3.4). Sections below are annotated where the original design +> assumed otherwise. ## 1. Goal @@ -32,9 +38,9 @@ the marginal cost of a routing decision is near zero. └──────────┬───────────┘ ▼ ┌─────────────────────┐ - │ Dispatcher │──▶ OpenRouter - │ (OpenAI-compatible) │──▶ Neuralwatt Cloud - └──────────┬───────────┘──▶ (future providers) + │ Dispatcher │──▶ Neuralwatt Cloud + │ (OpenAI-compatible) │──▶ (future providers) + └──────────┬───────────┘ ▼ ┌─────────────────────┐ │ Verify / Escalate │ (local model spot-checks output) @@ -54,7 +60,7 @@ per model+category), joined at query time. | column | type | notes | |---|---|---| | `model_id` | text | canonical model name | -| `provider` | text | `openrouter`, `neuralwatt`, future providers | +| `provider` | text | `neuralwatt` today; column kept for future providers | | `cost_per_1m_prompt` | real | listed token price | | `cost_per_1m_completion` | real | listed token price | | `cost_per_1m_prompt_cached` | real | e.g. Neuralwatt bills cached prefix at 25% | @@ -130,13 +136,36 @@ primary category (and optionally a secondary one for hybrid tasks, e.g. "refacto category — not an average across categories — so a model that's great at `docs_writing` but mediocre at `coding_refactor` doesn't get miscredited on refactor tasks. +### 3.4 Serving class (added after first live run) + +Neuralwatt ships ~6 base models as 19 catalog rows. The id suffixes are three **orthogonal** +dimensions — hence ids like `glm-5.2-short-fast-flex` — and they are *not* visible anywhere +except the id string and the prose description: + +| suffix | column | meaning | +|---|---|---| +| `-flex` | `latency_class` | discounted async tier; requests held server-side during peak until a capacity gap opens | +| `-fast` | `reasoning_mode` | thinking disabled or capped to a short budget. **Not** queue priority | +| `-short` | `context_variant` | 200K pool with a bounded reasoning budget | + +These rows carry **identical catalog pricing**, so they tie exactly on every scored dimension. +Left to the weighted score, a coin flip decides whether an interactive request lands on a +preemptible flex row. So latency tolerance is a **third hard filter**, not a weight: +`interactive` (default) excludes `-flex`; `batch` admits it. + +Access gating is likewise prose-only — 6 of 19 rows say "Private preview (grant-gated)" or +"(Canary)" in free text with no structured field. Those are parsed into `access_level` and +excluded by default, since routing to one earns a 403 at dispatch rather than a bad answer. + ## 4. Weighted scoring -Router computes a composite score per candidate model and picks the max, after two hard +Router computes a composite score per candidate model and picks the max, after three hard filters are applied (not weighted — these disqualify a candidate outright): 1. `effective_context_window >= required_context` (§3.2) 2. `tier >= required_tier` (from classifier) +3. serving class compatible with the request's latency tolerance, and `access_level` + actually reachable by this account (§3.4) ``` score = (w_cost × cost_score) @@ -145,12 +174,10 @@ score = (w_cost × cost_score) ``` - **cost_score**: normalized inverse cost (cheapest candidate = 1.0, scaled down from there). - For Neuralwatt, compute both token-price and energy-price cost and take whichever the account - is actually billed under. -- **eco_score**: normalized inverse `carbon_g_co2eq_per_req` where available (Neuralwatt only - today); models without a carbon figure get a neutral/default score rather than being - penalized, so the OpenRouter side of the table isn't unfairly downweighted until more - providers expose this. + ⚠️ **See §4.1 — the input this currently reads is not the price being billed.** +- **eco_score**: normalized inverse `carbon_g_co2eq_per_req`, which Neuralwatt reports + per-completion; models without a carbon figure get a neutral/default score rather than being + penalized, so a model new to the table isn't downweighted just for lacking history. - **proficiency_score**: from the benchmark refresh — general leaderboard score blended with your own eval set (see §5), weighted toward the latter over time as it accumulates. @@ -162,13 +189,48 @@ cost don't matter much, crank `w_prof` up for anything touching production code. let a great cost/eco score smuggle a task into a model that's actually too weak for it) — the weights decide *within* the eligible set, not whether a bad-fit model can win purely on price. +### 4.1 Billing is per-kWh — the cost axis needs rethinking + +The line above ("take whichever the account is actually billed under") turned out to be the +whole ballgame. Measured across five models on 2026-08-11, `cost_usd / energy_kwh` came back +**8.00 every time**: Neuralwatt bills a flat **$8.00/kWh** and the catalog's +`input_per_million` / `output_per_million` figures are not what this account is charged. + +The ranking consequence is not subtle. On the same prompt: + +| model | list $/1M out | billed USD | kWh | +|---|---|---|---| +| `kimi-k2.7-code-fast` | 4.00 | 2.25e-03 | 2.81e-04 | +| `kimi-k3-fast` | 15.00 | 2.17e-04 | 2.71e-05 | + +The model listing at **a quarter the price cost 10x more to run.** Any `cost_score` computed +from list price ranks these backwards. + +This collapses two of the three scoring axes into one. `cost_usd = 8.00 x energy_kwh` exactly, +so `w_cost` and `w_eco` are weighting the same underlying quantity — 0.6 of the composite on +one signal, with proficiency left holding 0.4. Cost and eco separate *only* through grid carbon +intensity (`grid_carbon_intensity_gco2perkwhr`, 37.0 on grid `FI` at time of writing), which is +a real distinction only when routing across regions. + +Two further wrinkles before this can be fixed: + +- **Energy is noisy.** Identical token counts (18 -> 900) billed anywhere from 1.8e-05 to + 8.8e-05 — a 4x spread run to run. A single observation is not a per-model estimate. +- **Cold start.** `energy_observations` is empty until real traffic accrues, so energy-based + scoring needs a seeding sweep (fixed prompt, N runs per routable model) before it can drive + anything. + +**Open decision:** either re-base `cost_score` on observed mean energy, or drop cost as a +separate axis entirely and let `eco` carry it with re-tuned weights. Not yet actioned. + ## 5. Freshness / scheduled jobs Two independent pollers, different cadences: -- **Pricing poller** (hourly–daily): hits `OpenRouter /models` and `Neuralwatt /v1/models` - (both unauthenticated for public listing), writes cost/context/energy columns, flags rows - unchanged > N days as `stale`. +- **Pricing poller** (hourly–daily): hits `Neuralwatt /v1/models` (unauthenticated for public + listing), writes cost/context/serving-class/access columns, flags rows unchanged > N days as + `stale`. Note the catalog carries **no energy fields at all** — energy arrives only + per-completion, so the dispatcher populates `energy_observations`, not this poller. - **Benchmark poller** (weekly, or on-demand): blends general leaderboard data (LMSYS Arena, LiveBench, Aider polyglot) with a small self-run eval set built from your actual task patterns (docstring quality, refactor correctness, doc-formatting) — the self-run set is more @@ -211,7 +273,13 @@ reported together. ## 9. Next build steps -1. Poller: OpenRouter + Neuralwatt → SQLite decision table (start here, it's the load-bearing piece) -2. Minimal FastAPI dispatcher: reads table, applies weighted score, calls provider, returns -3. Local classifier prompt/schema: `{task_tier, required_context, confidence, escalation_flag}` -4. Wire into one real workflow (coding agent subagent spin-off) end to end before generalizing +1. ~~Poller: Neuralwatt → SQLite decision table~~ **done**, verified against the live API +2. ~~Minimal FastAPI dispatcher: reads table, applies weighted score, calls provider, returns~~ + **done** — `/health`, `/route` (dry run), `/dispatch` +3. ~~Local classifier prompt/schema~~ **done** — the allowed category list is injected from + config at call time, since the model otherwise invents labels that join against nothing +4. **Re-base the cost axis** (§4.1) — the open design call, and it gates any claim that routing + decisions are actually cost-optimal +5. Seed `energy_observations` with a fixed-prompt sweep so `eco` stops returning neutral 0.5 +6. Populate `proficiency` — currently empty, so 0.4 of the composite weight is inert +7. Wire into one real workflow (coding agent subagent spin-off) end to end before generalizing diff --git a/dispatcher.py b/dispatcher.py new file mode 100644 index 0000000..9735345 --- /dev/null +++ b/dispatcher.py @@ -0,0 +1,730 @@ +#!/usr/bin/env python3 +"""FastAPI dispatcher for the local LLM model router. + +Pipeline per request: + + task text + -> local classifier (Ollama) category / tier / context estimate + -> escalation low confidence bumps the tier + -> hard filters (routing.py) context, tier, freshness, access, latency + -> weighted score (scoring.py) cost / eco / proficiency + -> dispatch to NeuralWatt OpenAI-compatible chat completion + -> energy_observations real cost + energy for the call + +Run: + uvicorn dispatcher:app --reload + +Endpoints: + GET /health provider/classifier reachability, catalog counts + POST /route classify and pick a model, but do NOT call it + POST /dispatch /route, then call the winning model and log it + + GET /v1/models OpenAI-compatible catalog + POST /v1/chat/completions OpenAI-compatible, routes then proxies + +/route exists because every part of this is testable without spending a +token, and a routing bug is much easier to see in a ranked candidate list +than in a completion. + +The /v1 pair is what a normal OpenAI client (opencode, an SDK, curl) talks +to. Ask for the model `auto` and the router chooses; ask for `auto:batch` +and it also admits flex rows; ask for a real model id and it goes straight +there, still logged. +""" + +from __future__ import annotations + +import json +import os +import sqlite3 +from datetime import datetime, timezone +from typing import Any, Literal, Optional + +import requests +from dotenv import load_dotenv +from fastapi import FastAPI, HTTPException +from fastapi.responses import StreamingResponse +from openai import OpenAI, OpenAIError +from pydantic import BaseModel, Field + +from config import RouterConfig, load_config +from routing import BATCH, INTERACTIVE, rank_candidates, select_candidates + +# Virtual model names that mean "you pick". Anything else is taken as a real +# model id and dispatched as asked. +ROUTER_MODEL = "auto" +ROUTER_MODEL_BATCH = "auto:batch" + +# Rough chars-per-token. Only used as a FLOOR on the classifier's context +# estimate: a real coding session sends the whole conversation, and the +# classifier — which sees that conversation and is asked to estimate its own +# input — routinely underestimates it by orders of magnitude (10 tokens for a +# 600-token exchange in testing). Underestimating here silently admits models +# that cannot hold the prompt, so the measured size wins when it is larger. +CHARS_PER_TOKEN = 4 + +load_dotenv() + +# kWh -> BTU. Serves no routing purpose; the design doc wants it on the +# dashboard, so it is stored alongside the kWh figure rather than derived. +BTU_PER_KWH = 3412.14 + +app = FastAPI(title="Local LLM Model Router", version="0.1.0") +cfg: RouterConfig = load_config("config.yaml") + + +# --- request/response models --------------------------------------------- + +class TaskRequest(BaseModel): + task: str = Field(..., description="The task to route.") + context: Optional[str] = Field( + None, description="Assembled context (docs/code) to send with the task." + ) + latency_tolerance: Optional[Literal["interactive", "batch"]] = Field( + None, + description=( + "'batch' admits flex rows, which are held server-side during peak. " + "Defaults to routing.default_latency_tolerance." + ), + ) + # Overrides, mostly for testing the router without the classifier in the loop. + task_category: Optional[str] = None + task_tier: Optional[int] = Field(None, ge=1, le=3) + required_context_tokens: Optional[int] = Field(None, ge=0) + + +class Classification(BaseModel): + task_category: str + task_tier: int + required_context_tokens: int + confidence: float + escalated: bool = False + source: Literal["classifier", "override"] = "classifier" + + +class Candidate(BaseModel): + model_id: str + provider: str + tier: int + cost_per_1m_completion: Optional[float] + latency_class: str + reasoning_mode: str + context_variant: str + effective_context_window: int + composite: float + cost_score: float + eco_score: float + proficiency_score: float + + +class RouteResponse(BaseModel): + classification: Classification + latency_tolerance: str + candidates_considered: int + selected: Optional[Candidate] + runners_up: list[Candidate] = [] + + +class DispatchResponse(BaseModel): + route: RouteResponse + content: str + prompt_tokens: Optional[int] + completion_tokens: Optional[int] + telemetry: "Telemetry" + + +# --- infrastructure ------------------------------------------------------- + +def _db() -> sqlite3.Connection: + conn = sqlite3.connect(cfg.database.path) + conn.row_factory = sqlite3.Row + return conn + + +def _classifier_client() -> OpenAI: + # Ollama ignores the key but the SDK requires one to be set. + return OpenAI(base_url=cfg.classifier.base_url, api_key="ollama") + + +def _provider_client(provider: str) -> OpenAI: + try: + settings = cfg.dispatch_providers[provider] + except KeyError: + raise HTTPException(500, f"No dispatch config for provider {provider!r}") + api_key = os.environ.get(settings.api_key_env) + if not api_key: + raise HTTPException( + 503, + f"{settings.api_key_env} is not set — copy .env.example to .env and fill it in.", + ) + return OpenAI(base_url=settings.base_url, api_key=api_key) + + +# --- classification ------------------------------------------------------- + +def classify(task: str, context: Optional[str]) -> Classification: + """Ask the local model to categorize and size the task. + + The allowed categories are appended from config rather than duplicated in + the prompt text, so the classifier cannot return a label that joins + against nothing in the proficiency table. + """ + categories = cfg.proficiency.categories + system_prompt = ( + f"{cfg.classifier.system_prompt}\n" + f"Allowed values for task_category (use EXACTLY one of these strings):\n" + + "\n".join(f" - {c}" for c in categories) + ) + user_content = task if not context else f"{task}\n\n--- context ---\n{context}" + + client = _classifier_client() + try: + resp = client.chat.completions.create( + model=cfg.classifier.model, + messages=[ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_content}, + ], + response_format={"type": "json_object"} + if cfg.classifier.response_format == "json" + else None, + temperature=cfg.classifier.temperature, + timeout=cfg.classifier.timeout_seconds, + ) + except OpenAIError as e: + raise HTTPException( + 503, f"Classifier ({cfg.classifier.model} @ {cfg.classifier.base_url}) failed: {e}" + ) + + raw = resp.choices[0].message.content or "" + try: + parsed = json.loads(raw) + except json.JSONDecodeError: + raise HTTPException(502, f"Classifier returned non-JSON output: {raw[:300]!r}") + + category = parsed.get("task_category") + if category not in categories: + # A label outside the configured set would silently miss every + # proficiency row, so fall back rather than routing on a phantom join. + category = "general_chat" if "general_chat" in categories else categories[0] + + try: + return Classification( + task_category=category, + task_tier=int(parsed["task_tier"]), + required_context_tokens=int(parsed["required_context_tokens"]), + confidence=float(parsed["confidence"]), + ) + except (KeyError, TypeError, ValueError) as e: + raise HTTPException(502, f"Classifier JSON missing/invalid fields: {e}; got {raw[:300]!r}") + + +def apply_escalation(c: Classification) -> Classification: + """Bump the tier when the classifier isn't confident in its own call.""" + if not cfg.escalation.enabled: + return c + if c.confidence >= cfg.escalation.min_confidence_before_bump: + return c + bumped = min(c.task_tier + 1, cfg.escalation.max_tier) + if bumped == c.task_tier: + return c + return c.model_copy(update={"task_tier": bumped, "escalated": True}) + + +# --- candidate lookup ----------------------------------------------------- + +def load_candidates(conn: sqlite3.Connection, category: str) -> list[dict]: + """All model rows, joined to this category's proficiency and mean carbon. + + Filtering happens in ``routing.py`` rather than in SQL so the hard filters + stay in one testable place; the catalog is 19 rows. + + ``eco`` is mean observed carbon, not energy: design doc §4 scores eco on + gCO2eq, and the same kWh in a different region is a different carbon + figure. Averaging over observations is the "stable per-model average" + answer to the doc's open question — grid intensity and region are logged + per observation, so switching to a request-time figure later needs no + re-run. Models with no observations yet come back None and take the + neutral 0.5 from ``scoring``, so a cold table never penalizes anyone. + """ + rows = conn.execute( + """ + SELECT m.*, + p.blended_score AS proficiency, + (SELECT AVG(e.carbon_g_co2eq) + FROM energy_observations e + WHERE e.model_id = m.model_id + AND e.provider = m.provider + AND e.carbon_g_co2eq IS NOT NULL) AS eco + FROM models m + LEFT JOIN proficiency p + ON p.model_id = m.model_id + AND p.provider = m.provider + AND p.category = ? + """, + (category,), + ).fetchall() + return [dict(r) for r in rows] + + +def _to_candidate(row: dict) -> Candidate: + return Candidate(**{k: row[k] for k in Candidate.model_fields if k in row}) + + +def route(req: TaskRequest) -> RouteResponse: + latency_tolerance = req.latency_tolerance or cfg.routing.default_latency_tolerance + + if req.task_category and req.task_tier and req.required_context_tokens is not None: + classification = Classification( + task_category=req.task_category, + task_tier=req.task_tier, + required_context_tokens=req.required_context_tokens, + confidence=1.0, + source="override", + ) + else: + classification = apply_escalation(classify(req.task, req.context)) + if req.task_category: + classification = classification.model_copy( + update={"task_category": req.task_category} + ) + if req.task_tier: + classification = classification.model_copy(update={"task_tier": req.task_tier}) + if req.required_context_tokens is not None: + classification = classification.model_copy( + update={"required_context_tokens": req.required_context_tokens} + ) + + conn = _db() + try: + rows = load_candidates(conn, classification.task_category) + finally: + conn.close() + + eligible = select_candidates( + rows, + required_context_tokens=classification.required_context_tokens, + required_tier=classification.task_tier, + latency_tolerance=latency_tolerance, + allowed_access_levels=cfg.routing.allowed_access_levels, + exclude_stale=cfg.freshness.exclude_stale, + exclude_deprecated=cfg.freshness.exclude_deprecated, + ) + ranked = rank_candidates( + eligible, + weights=(cfg.weights.cost, cfg.weights.eco, cfg.weights.proficiency), + flex_cost_multiplier=cfg.routing.flex_cost_multiplier, + ) + + return RouteResponse( + classification=classification, + latency_tolerance=latency_tolerance, + candidates_considered=len(eligible), + selected=_to_candidate(ranked[0]) if ranked else None, + runners_up=[_to_candidate(r) for r in ranked[1:4]], + ) + + +# --- energy / cost accounting -------------------------------------------- + +class Telemetry(BaseModel): + """What the provider reported about a completion it just served.""" + + energy_kwh: Optional[float] = None + energy_btu: Optional[float] = None + carbon_g_co2eq: Optional[float] = None + grid_carbon_intensity: Optional[float] = None + grid_id: Optional[str] = None + cost_usd: Optional[float] = None + allowance_remaining_usd: Optional[float] = None + service_tier: Optional[str] = None + + +def extract_telemetry(payload: dict) -> Telemetry: + """Read NeuralWatt's per-request energy and cost blocks. + + Both sit at the top level of the completion response, outside anything + the OpenAI schema models, so this reads the raw JSON rather than the + parsed object. Every field is optional: a provider that reports nothing + yields an all-None Telemetry rather than an error, and the neutral-0.5 + path in scoring covers the gap. + """ + energy = payload.get("energy") or {} + cost = payload.get("cost") or {} + kwh = energy.get("energy_kwh") + + return Telemetry( + energy_kwh=kwh, + energy_btu=kwh * BTU_PER_KWH if kwh is not None else None, + carbon_g_co2eq=energy.get("carbon_g_co2eq"), + grid_carbon_intensity=energy.get("grid_carbon_intensity_gco2perkwhr"), + grid_id=energy.get("grid_id"), + # The provider's billed figure, not a tokens x list-price estimate. + # Flex rows bill under their standard sibling despite identical + # advertised pricing, so the estimate is wrong for every flex call. + cost_usd=cost.get("request_cost_usd"), + allowance_remaining_usd=cost.get("allowance_remaining_usd"), + service_tier=payload.get("service_tier"), + ) + + +def log_observation( + model_id: str, + provider: str, + task_category: str, + prompt_tokens: Optional[int], + completion_tokens: Optional[int], + telemetry: Telemetry, +) -> None: + if not cfg.logging.log_energy_observations: + return + conn = _db() + try: + conn.execute( + """ + INSERT INTO energy_observations ( + model_id, provider, task_category, prompt_tokens, completion_tokens, + energy_kwh, energy_btu, carbon_g_co2eq, grid_carbon_intensity, grid_id, + cost_usd, allowance_remaining_usd, service_tier, observed_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + model_id, + provider, + task_category, + prompt_tokens, + completion_tokens, + telemetry.energy_kwh, + telemetry.energy_btu, + telemetry.carbon_g_co2eq, + telemetry.grid_carbon_intensity, + telemetry.grid_id, + telemetry.cost_usd, + telemetry.allowance_remaining_usd, + telemetry.service_tier, + datetime.now(timezone.utc).isoformat(), + ), + ) + conn.commit() + finally: + conn.close() + + +# --- endpoints ------------------------------------------------------------ + +@app.get("/health") +def health(): + conn = _db() + try: + counts = dict( + conn.execute( + """ + SELECT 'models', COUNT(*) FROM models + UNION ALL SELECT 'routable', COUNT(*) FROM models + WHERE access_level = 'public' AND availability = 'active' + UNION ALL SELECT 'proficiency', COUNT(*) FROM proficiency + UNION ALL SELECT 'energy_observations', COUNT(*) FROM energy_observations + """ + ).fetchall() + ) + finally: + conn.close() + + try: + _classifier_client().models.list() + classifier_ok = True + except OpenAIError: + classifier_ok = False + + return { + "status": "ok", + "counts": counts, + "classifier_reachable": classifier_ok, + "classifier_model": cfg.classifier.model, + "providers": list(cfg.dispatch_providers), + "api_keys_present": { + name: bool(os.environ.get(p.api_key_env)) + for name, p in cfg.dispatch_providers.items() + }, + } + + +@app.post("/route", response_model=RouteResponse) +def route_endpoint(req: TaskRequest): + """Classify and pick a model without calling it.""" + return route(req) + + +# --- OpenAI-compatible surface ------------------------------------------- + +def estimate_prompt_tokens(messages: list[dict]) -> int: + """Floor estimate of a conversation's token count, from its characters. + + Deliberately crude. Its only job is to stop a long conversation being + routed to a model that cannot hold it when the classifier lowballs its + own input size. + """ + chars = 0 + for m in messages: + content = m.get("content") + if isinstance(content, str): + chars += len(content) + elif isinstance(content, list): + # Multimodal content parts; count the text ones. + for part in content: + if isinstance(part, dict) and isinstance(part.get("text"), str): + chars += len(part["text"]) + return chars // CHARS_PER_TOKEN + + +def _last_user_text(messages: list[dict]) -> str: + for m in reversed(messages): + if m.get("role") == "user": + content = m.get("content") + if isinstance(content, str): + return content + if isinstance(content, list): + return " ".join( + p["text"] + for p in content + if isinstance(p, dict) and isinstance(p.get("text"), str) + ) + return "" + + +@app.get("/v1/models") +def list_models(): + """The routable catalog, in OpenAI's list shape. + + The virtual router entries are listed first so a client that just picks + the head of the list gets routing rather than an arbitrary model. + """ + conn = _db() + try: + rows = conn.execute( + """ + SELECT model_id, latency_class FROM models + WHERE access_level IN (%s) AND availability = 'active' + ORDER BY model_id + """ + % ",".join("?" * len(cfg.routing.allowed_access_levels)), + tuple(cfg.routing.allowed_access_levels), + ).fetchall() + finally: + conn.close() + + data = [ + {"id": ROUTER_MODEL, "object": "model", "owned_by": "router"}, + {"id": ROUTER_MODEL_BATCH, "object": "model", "owned_by": "router"}, + ] + data += [ + {"id": r["model_id"], "object": "model", "owned_by": "neuralwatt"} + for r in rows + ] + return {"object": "list", "data": data} + + +def _sniff_telemetry_line(line: str) -> Optional[tuple[str, dict]]: + """Parse NeuralWatt's SSE *comment* lines carrying energy/cost. + + A streaming response ends with `: energy {...}` and `: cost {...}` before + `data: [DONE]`. They are SSE comments, so every ordinary client ignores + them — which is exactly why the stream can be proxied through untouched + while still being read on the way past. + """ + if not line.startswith(":"): + return None + body = line[1:].strip() + for key in ("energy", "cost"): + prefix = f"{key} " + if body.startswith(prefix): + try: + return key, json.loads(body[len(prefix):]) + except json.JSONDecodeError: + return None + return None + + +@app.post("/v1/chat/completions") +def chat_completions(body: dict[str, Any]): + """OpenAI-compatible completions, routed then proxied. + + Streaming is passed through chunk by chunk rather than buffered, so a + coding agent still renders tokens as they arrive; the telemetry is + scraped from the trailing SSE comments on the way past. + """ + messages = body.get("messages") or [] + if not messages: + raise HTTPException(400, "messages is required") + + requested = body.get("model") or ROUTER_MODEL + # Some clients (opencode among them) send the model as `provider/model`. + # Real ids can contain slashes too (`deepseek-ai/DeepSeek-V4-Flash`), so + # only the virtual names are matched against a stripped suffix. + bare = requested.rsplit("/", 1)[-1] + wants_routing = bare in (ROUTER_MODEL, ROUTER_MODEL_BATCH) + if wants_routing: + requested = bare + + if wants_routing: + latency = BATCH if requested == ROUTER_MODEL_BATCH else INTERACTIVE + decision = route( + TaskRequest( + task=_last_user_text(messages), + latency_tolerance=latency, + # Take whichever is larger: what the classifier thinks it + # needs, or what the conversation actually measures. + required_context_tokens=None, + ) + ) + measured = estimate_prompt_tokens(messages) + if measured > decision.classification.required_context_tokens: + decision = route( + TaskRequest( + task=_last_user_text(messages), + latency_tolerance=latency, + task_category=decision.classification.task_category, + task_tier=decision.classification.task_tier, + required_context_tokens=measured, + ) + ) + if decision.selected is None: + raise HTTPException( + 422, + "No model satisfies the hard filters for this request " + f"(tier >= {decision.classification.task_tier}, context >= " + f"{decision.classification.required_context_tokens} tokens, " + f"{decision.latency_tolerance}).", + ) + target = decision.selected.model_id + provider = decision.selected.provider + category = decision.classification.task_category + else: + target, provider, category = requested, "neuralwatt", "general_chat" + + settings = cfg.dispatch_providers[provider] + api_key = os.environ.get(settings.api_key_env) + if not api_key: + raise HTTPException(503, f"{settings.api_key_env} is not set.") + + upstream_body = {**body, "model": target} + streaming = bool(body.get("stream")) + if streaming: + # Without this the final chunk carries no usage and the observation + # would be logged with null token counts. + upstream_body.setdefault("stream_options", {"include_usage": True}) + + url = f"{settings.base_url}/chat/completions" + headers = {"authorization": f"Bearer {api_key}"} + + if not streaming: + resp = requests.post(url, headers=headers, json=upstream_body, timeout=600) + if resp.status_code >= 400: + raise HTTPException(resp.status_code, resp.text[:500]) + payload = resp.json() + usage = payload.get("usage") or {} + log_observation( + target, + provider, + category, + usage.get("prompt_tokens"), + usage.get("completion_tokens"), + extract_telemetry(payload), + ) + # Report the model actually used, so the client isn't told 'auto'. + payload["model"] = target + return payload + + def proxy(): + collected: dict[str, dict] = {} + usage: dict = {} + upstream = requests.post( + url, headers=headers, json=upstream_body, stream=True, timeout=600 + ) + if upstream.status_code >= 400: + detail = upstream.text[:500] + yield f"data: {json.dumps({'error': {'message': detail}})}\n\n".encode() + return + try: + for raw in upstream.iter_lines(decode_unicode=True): + if raw is None: + continue + sniffed = _sniff_telemetry_line(raw) + if sniffed: + collected[sniffed[0]] = sniffed[1] + elif raw.startswith("data: ") and raw.strip() != "data: [DONE]": + try: + chunk = json.loads(raw[6:]) + if chunk.get("usage"): + usage = chunk["usage"] + except json.JSONDecodeError: + pass + yield f"{raw}\n".encode() + finally: + upstream.close() + # Logged even on a client disconnect — the energy was spent. + log_observation( + target, + provider, + category, + usage.get("prompt_tokens"), + usage.get("completion_tokens"), + extract_telemetry(collected), + ) + + return StreamingResponse(proxy(), media_type="text/event-stream") + + +@app.post("/dispatch", response_model=DispatchResponse) +def dispatch_endpoint(req: TaskRequest): + decision = route(req) + if decision.selected is None: + raise HTTPException( + 422, + "No model satisfies the hard filters for this task " + f"(tier >= {decision.classification.task_tier}, context >= " + f"{decision.classification.required_context_tokens} tokens, " + f"{decision.latency_tolerance}). Trim the context or widen " + "routing.allowed_access_levels.", + ) + + selected = decision.selected + client = _provider_client(selected.provider) + messages = [{"role": "user", "content": req.task}] + if req.context: + messages.insert(0, {"role": "system", "content": req.context}) + + try: + # with_raw_response because the energy and cost blocks sit outside the + # OpenAI schema and the parsed object drops them. + raw = client.chat.completions.with_raw_response.create( + model=selected.model_id, messages=messages + ) + except OpenAIError as e: + raise HTTPException(502, f"Dispatch to {selected.model_id} failed: {e}") + + payload = json.loads(raw.text) + usage = payload.get("usage") or {} + prompt_tokens = usage.get("prompt_tokens") + completion_tokens = usage.get("completion_tokens") + telemetry = extract_telemetry(payload) + + log_observation( + selected.model_id, + selected.provider, + decision.classification.task_category, + prompt_tokens, + completion_tokens, + telemetry, + ) + + choices = payload.get("choices") or [{}] + content = (choices[0].get("message") or {}).get("content") or "" + + return DispatchResponse( + route=decision, + content=content, + prompt_tokens=prompt_tokens, + completion_tokens=completion_tokens, + telemetry=telemetry, + ) diff --git a/opencode.json b/opencode.json new file mode 100644 index 0000000..300f2a5 --- /dev/null +++ b/opencode.json @@ -0,0 +1,30 @@ +{ + "$schema": "https://opencode.ai/config.json", + "model": "llm-router/auto", + "provider": { + "llm-router": { + "name": "Local LLM Router", + "npm": "@ai-sdk/openai-compatible", + "options": { + "baseURL": "http://127.0.0.1:8080/v1", + "apiKey": "unused" + }, + "models": { + "auto": { + "name": "auto (router picks, interactive)", + "limit": { + "context": 782324, + "output": 16384 + } + }, + "auto:batch": { + "name": "auto (router picks, admits flex/async)", + "limit": { + "context": 782324, + "output": 16384 + } + } + } + } + } +} diff --git a/poller.py b/poller.py index 54b58ed..a33f517 100644 --- a/poller.py +++ b/poller.py @@ -2,9 +2,9 @@ """ Pricing/catalog poller for the local LLM router. -Fetches model catalogs from OpenRouter and Neuralwatt (both unauthenticated -public endpoints), normalizes into a common shape, and upserts into the -`models` table in router.db. +Fetches the NeuralWatt model catalog (unauthenticated public endpoint), +normalizes it into the `models` table in router.db, and flags rows that have +gone stale. Run manually: python poller.py @@ -13,7 +13,11 @@ Run on a schedule (cron example, every 2 hours): 0 */2 * * * /usr/bin/python3 /path/to/poller.py >> /var/log/router-poller.log 2>&1 Does NOT touch energy data — energy is only available per-completion, not from -a models list. See energy_logger.py (dispatcher-side) for that. +a models list, so the dispatcher writes `energy_observations` instead. + +NeuralWatt is the only provider. The `provider` column and the (model_id, +provider) primary key are kept so a second provider can be added without a +migration. """ from __future__ import annotations @@ -26,20 +30,55 @@ from typing import Optional import requests -DB_PATH = "router.db" +from config import RouterConfig, load_config -OPENROUTER_MODELS_URL = "https://openrouter.ai/api/v1/models" NEURALWATT_MODELS_URL = "https://api.neuralwatt.com/v1/models" REQUEST_TIMEOUT = 20 # seconds -# Fraction of advertised context to treat as usable, before subtracting the -# output reserve. Tune per-provider/model once you have real data; this is a -# deliberately conservative global default to start. -CONTEXT_SAFETY_FACTOR = 0.75 -DEFAULT_OUTPUT_RESERVE_TOKENS = 4096 +# Serving-class suffixes. NeuralWatt ships one base model as several catalog +# rows that differ only by these tokens, and they combine freely — hence ids +# like 'glm-5.2-short-fast-flex'. They are stripped from the end of the id one +# segment at a time so a base name that merely *contains* a lookalike token is +# never misread (e.g. 'deepseek-v4-flash' is not a '-fast' row). +SUFFIX_FLEX = "flex" +SUFFIX_FAST = "fast" +SUFFIX_SHORT = "short" +SERVING_SUFFIXES = frozenset({SUFFIX_FLEX, SUFFIX_FAST, SUFFIX_SHORT}) -STALE_AFTER_DAYS = 3 + +def parse_serving_class(model_id: str) -> tuple[str, str, str]: + """Derive (latency_class, reasoning_mode, context_variant) from a model id. + + Returns the schema defaults ('standard', 'default', 'full') for a base + model. Suffixes are matched as whole '-'-delimited segments only. + """ + segments = model_id.lower().split("-") + found = set() + while len(segments) > 1 and segments[-1] in SERVING_SUFFIXES: + found.add(segments.pop()) + + return ( + "flex" if SUFFIX_FLEX in found else "standard", + "reduced" if SUFFIX_FAST in found else "default", + "short" if SUFFIX_SHORT in found else "full", + ) + + +def parse_access_level(display_name: Optional[str], description: Optional[str]) -> str: + """Derive an access level from the catalog's prose. + + NeuralWatt exposes no structured gating field — restricted models are only + marked in free text ("Private preview (grant-gated)", "(Canary)"). Routing + to one earns a 403 at dispatch, so this is parsed defensively: anything + that looks gated is treated as gated. + """ + blob = f"{display_name or ''} {description or ''}".lower() + if "grant-gated" in blob or "private preview" in blob: + return "preview" + if "canary" in blob: + return "canary" + return "public" @dataclass @@ -56,62 +95,24 @@ class ModelRow: supports_json_mode: bool supports_vision: bool supports_reasoning: bool + reasoning_default_enabled: bool + latency_class: str + reasoning_mode: str + context_variant: str + access_level: str pricing_tbd: bool deprecated: bool - def effective_context_window(self) -> Optional[int]: + def effective_context_window(self, cfg: RouterConfig) -> Optional[int]: if not self.context_window: return None - reserve = self.max_output_tokens or DEFAULT_OUTPUT_RESERVE_TOKENS - usable = int(self.context_window * CONTEXT_SAFETY_FACTOR) - reserve + # 11 of 19 catalog rows report no max_output_tokens, so the configured + # reserve carries most of the catalog. + reserve = self.max_output_tokens or cfg.context.default_output_reserve_tokens + usable = int(self.context_window * cfg.context.safety_factor) - reserve return max(usable, 0) -def fetch_openrouter() -> list[ModelRow]: - resp = requests.get(OPENROUTER_MODELS_URL, timeout=REQUEST_TIMEOUT) - resp.raise_for_status() - payload = resp.json() - - rows = [] - for m in payload.get("data", []): - pricing = m.get("pricing", {}) or {} - - def _price(key: str) -> Optional[float]: - # OpenRouter returns per-token price as a string, e.g. "0.0000007" - val = pricing.get(key) - if val is None: - return None - try: - return float(val) * 1_000_000 # normalize to per-1M - except (TypeError, ValueError): - return None - - top_provider = m.get("top_provider", {}) or {} - rows.append( - ModelRow( - model_id=m.get("id"), - provider="openrouter", - display_name=m.get("name"), - cost_per_1m_prompt=_price("prompt"), - cost_per_1m_completion=_price("completion"), - cost_per_1m_prompt_cached=_price("input_cache_read"), - context_window=m.get("context_length") - or top_provider.get("context_length"), - max_output_tokens=top_provider.get("max_completion_tokens"), - supports_tools="tools" in (m.get("supported_parameters") or []), - supports_json_mode="response_format" - in (m.get("supported_parameters") or []), - supports_vision="image" in (m.get("architecture", {}) or {}).get( - "input_modalities", [] - ), - supports_reasoning="reasoning" in (m.get("supported_parameters") or []), - pricing_tbd=False, - deprecated=False, - ) - ) - return rows - - def fetch_neuralwatt() -> list[ModelRow]: resp = requests.get(NEURALWATT_MODELS_URL, timeout=REQUEST_TIMEOUT) resp.raise_for_status() @@ -119,14 +120,27 @@ def fetch_neuralwatt() -> list[ModelRow]: rows = [] for m in payload.get("data", []): + model_id = m.get("id") meta = m.get("metadata", {}) or {} pricing = meta.get("pricing", {}) or {} caps = meta.get("capabilities", {}) or {} limits = meta.get("limits", {}) or {} + reasoning = meta.get("reasoning") or {} + + supports_reasoning = bool(caps.get("reasoning")) + # capabilities.reasoning only means "the API accepts a reasoning + # param" and is true for nearly the whole catalog. default_enabled is + # the discriminating signal; a few models (the kimi-k2.7-code family) + # expose no reasoning block at all, so fall back to the capability. + default_enabled = reasoning.get("default_enabled") + if default_enabled is None: + default_enabled = supports_reasoning + + latency_class, reasoning_mode, context_variant = parse_serving_class(model_id) rows.append( ModelRow( - model_id=m.get("id"), + model_id=model_id, provider="neuralwatt", display_name=meta.get("display_name"), cost_per_1m_prompt=pricing.get("input_per_million"), @@ -138,7 +152,14 @@ def fetch_neuralwatt() -> list[ModelRow]: supports_tools=bool(caps.get("tools")), supports_json_mode=bool(caps.get("json_mode")), supports_vision=bool(caps.get("vision")), - supports_reasoning=bool(caps.get("reasoning")), + supports_reasoning=supports_reasoning, + reasoning_default_enabled=bool(default_enabled), + latency_class=latency_class, + reasoning_mode=reasoning_mode, + context_variant=context_variant, + access_level=parse_access_level( + meta.get("display_name"), meta.get("description") + ), pricing_tbd=bool(pricing.get("pricing_tbd")), deprecated=bool(meta.get("deprecated")), ) @@ -146,7 +167,7 @@ def fetch_neuralwatt() -> list[ModelRow]: return rows -def upsert(conn: sqlite3.Connection, rows: list[ModelRow]) -> None: +def upsert(conn: sqlite3.Connection, rows: list[ModelRow], cfg: RouterConfig) -> None: now = datetime.now(timezone.utc).isoformat() for r in rows: availability = "deprecated" if r.deprecated else "active" @@ -157,8 +178,10 @@ def upsert(conn: sqlite3.Connection, rows: list[ModelRow]) -> None: cost_per_1m_prompt, cost_per_1m_completion, cost_per_1m_prompt_cached, context_window, effective_context_window, max_output_tokens, supports_tools, supports_json_mode, supports_vision, supports_reasoning, + reasoning_default_enabled, latency_class, reasoning_mode, + context_variant, access_level, pricing_tbd, deprecated, availability, last_updated - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(model_id, provider) DO UPDATE SET display_name = excluded.display_name, cost_per_1m_prompt = excluded.cost_per_1m_prompt, @@ -171,6 +194,11 @@ def upsert(conn: sqlite3.Connection, rows: list[ModelRow]) -> None: supports_json_mode = excluded.supports_json_mode, supports_vision = excluded.supports_vision, supports_reasoning = excluded.supports_reasoning, + reasoning_default_enabled = excluded.reasoning_default_enabled, + latency_class = excluded.latency_class, + reasoning_mode = excluded.reasoning_mode, + context_variant = excluded.context_variant, + access_level = excluded.access_level, pricing_tbd = excluded.pricing_tbd, deprecated = excluded.deprecated, availability = excluded.availability, @@ -184,12 +212,17 @@ def upsert(conn: sqlite3.Connection, rows: list[ModelRow]) -> None: r.cost_per_1m_completion, r.cost_per_1m_prompt_cached, r.context_window, - r.effective_context_window(), + r.effective_context_window(cfg), r.max_output_tokens, int(r.supports_tools), int(r.supports_json_mode), int(r.supports_vision), int(r.supports_reasoning), + int(r.reasoning_default_enabled), + r.latency_class, + r.reasoning_mode, + r.context_variant, + r.access_level, int(r.pricing_tbd), int(r.deprecated), availability, @@ -199,39 +232,39 @@ def upsert(conn: sqlite3.Connection, rows: list[ModelRow]) -> None: conn.commit() -def mark_stale(conn: sqlite3.Connection) -> None: +def mark_stale(conn: sqlite3.Connection, cfg: RouterConfig) -> None: """Flag rows that weren't touched by this poll run as stale, rather than silently leaving old data looking current.""" conn.execute( - f""" + """ UPDATE models SET availability = 'stale' WHERE availability = 'active' - AND julianday('now') - julianday(last_updated) > {STALE_AFTER_DAYS} - """ + AND julianday('now') - julianday(last_updated) > ? + """, + (cfg.freshness.stale_after_days,), ) conn.commit() def main() -> int: - conn = sqlite3.connect(DB_PATH) + cfg = load_config("config.yaml") + conn = sqlite3.connect(cfg.database.path) conn.execute("PRAGMA foreign_keys = ON") - total = 0 - for name, fetch_fn in (("openrouter", fetch_openrouter), ("neuralwatt", fetch_neuralwatt)): - try: - rows = fetch_fn() - upsert(conn, rows) - print(f"[{name}] upserted {len(rows)} models") - total += len(rows) - except requests.RequestException as e: - print(f"[{name}] FAILED: {e}", file=sys.stderr) - # Don't let one provider's outage nuke the whole poll run. - continue + try: + rows = fetch_neuralwatt() + except requests.RequestException as e: + print(f"[neuralwatt] FAILED: {e}", file=sys.stderr) + conn.close() + return 1 - mark_stale(conn) + upsert(conn, rows, cfg) + print(f"[neuralwatt] upserted {len(rows)} models") + + mark_stale(conn, cfg) conn.close() - print(f"done, {total} rows upserted total") + print(f"done, {len(rows)} rows upserted total") return 0 diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..61dc700 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,7 @@ +[tool.pytest.ini_options] +# The router modules sit at the repo root rather than in a package, so the +# root has to be on sys.path for `from config import ...` to resolve. Running +# as `python -m pytest` happens to add the cwd and hides this; the `pytest` +# console script does not. Setting it here makes both invocations work. +pythonpath = ["."] +testpaths = ["tests"] diff --git a/requirements.txt b/requirements.txt index 3a9240d..ef766fd 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,8 +1,12 @@ -pyyaml>=6.0 -pydantic>=2.0 -requests>=2.31 -fastapi>=0.110 -uvicorn[standard]>=0.29 -openai>=1.30 -pytest>=8.0 -pytest-cov>=5.0 +# Pinned. Recreating the venv with >= constraints silently jumped openai +# 2.53 -> 3.0 and httpx -> httpx2; a service that restarts on boot should not +# change its dependency tree underneath itself. Bump deliberately. +pyyaml==6.0.3 +pydantic==2.13.4 +requests==2.34.2 +fastapi==0.141.1 +uvicorn[standard]==0.52.1 +openai==3.0.0 +python-dotenv==1.2.2 +pytest==9.1.1 +pytest-cov==7.1.0 diff --git a/routing.py b/routing.py new file mode 100644 index 0000000..259bb64 --- /dev/null +++ b/routing.py @@ -0,0 +1,164 @@ +"""Pure candidate selection and ranking for the local LLM model router. + +Like ``scoring.py`` and ``tiering.py``, this module is free of I/O: rows come +in as dicts, thresholds come in as arguments. ``dispatcher.py`` owns the DB +reads and the provider call. + +Two stages, in order: + +1. **Hard filters** (``select_candidates``) — disqualify outright. A model + that cannot hold the context, is below the required tier, is stale, or is + not actually reachable by this account is not a cost tradeoff to be + weighed; it is not a candidate at all (design doc §3.2, §4). +2. **Weighted scoring** (``rank_candidates``) — order what survives by + ``w_cost*cost + w_eco*eco + w_prof*proficiency``. + +The flex filter is the one that is easy to get wrong. NeuralWatt's ``-flex`` +rows are the same weights at the same advertised price, so on every scored +dimension they tie exactly with their standard sibling — but they are "held +server-side during peak until a capacity gap opens". Left to the scorer, a +coin flip decides whether an interactive request waits out a capacity gap. +Latency tolerance is therefore a filter, not a weight. +""" + +from __future__ import annotations + +from collections.abc import Sequence + +from scoring import composite_score, cost_score, eco_score, proficiency_score + +INTERACTIVE = "interactive" +BATCH = "batch" + + +def is_eligible( + row: dict, + *, + required_context_tokens: int, + required_tier: int, + latency_tolerance: str, + allowed_access_levels: Sequence[str], + exclude_stale: bool, + exclude_deprecated: bool, +) -> bool: + """Whether a model row survives every hard filter. + + A NULL ``effective_context_window`` fails the context filter: an unknown + window cannot be shown to be large enough, and silently truncating + mid-task is worse than routing elsewhere. + """ + eff_ctx = row.get("effective_context_window") + if eff_ctx is None or eff_ctx < required_context_tokens: + return False + + tier = row.get("tier") + if tier is None or tier < required_tier: + return False + + availability = row.get("availability") + if exclude_stale and availability == "stale": + return False + if exclude_deprecated and (availability == "deprecated" or row.get("deprecated")): + return False + + if row.get("access_level", "public") not in allowed_access_levels: + return False + + # Flex rows may be queued behind a capacity gap; only batch work opts in. + if latency_tolerance == INTERACTIVE and row.get("latency_class") == "flex": + return False + + return True + + +def select_candidates( + rows: Sequence[dict], + *, + required_context_tokens: int, + required_tier: int, + latency_tolerance: str, + allowed_access_levels: Sequence[str], + exclude_stale: bool, + exclude_deprecated: bool, +) -> list[dict]: + """Apply every hard filter, preserving input order.""" + return [ + row + for row in rows + if is_eligible( + row, + required_context_tokens=required_context_tokens, + required_tier=required_tier, + latency_tolerance=latency_tolerance, + allowed_access_levels=allowed_access_levels, + exclude_stale=exclude_stale, + exclude_deprecated=exclude_deprecated, + ) + ] + + +def effective_cost(row: dict, flex_cost_multiplier: float) -> float | None: + """Completion cost, discounted if this row bills at the flex rate. + + The catalog lists flex at the same sticker price as standard serving even + though it advertises itself as reduced-rate, so the discount is a config + knob (default 1.0 — no assumed discount) rather than a number invented + here. See ``routing.flex_cost_multiplier`` in config.yaml. + """ + cost = row.get("cost_per_1m_completion") + if cost is None: + return None + if row.get("latency_class") == "flex": + return cost * flex_cost_multiplier + return cost + + +def rank_candidates( + rows: Sequence[dict], + *, + weights: tuple[float, float, float], + flex_cost_multiplier: float = 1.0, +) -> list[dict]: + """Score and sort candidates best-first. + + Each row is expected to carry ``cost_per_1m_completion``, an optional + ``eco`` (mean energy per request, from ``energy_observations``), and an + optional ``proficiency`` (blended score for the task's category). Missing + eco/proficiency data yields the neutral 0.5 from ``scoring``, so a model + is never penalized merely for being new to the table. + + Returns a list of dicts: the original row plus ``composite`` and the + three sub-scores, so a caller can log exactly why a model won. + """ + costs = [effective_cost(r, flex_cost_multiplier) for r in rows] + ecos = [r.get("eco") for r in rows] + + cost_scores = cost_score(costs) + eco_scores = eco_score(ecos) + + ranked = [] + for row, c_s, e_s in zip(rows, cost_scores, eco_scores): + p_s = proficiency_score(row.get("proficiency")) + ranked.append( + { + **row, + "cost_score": c_s, + "eco_score": e_s, + "proficiency_score": p_s, + "composite": composite_score(c_s, e_s, p_s, weights), + } + ) + + # Sort by composite, then by cost ascending, then model_id, so that rows + # tying on every scored dimension still resolve deterministically instead + # of returning whatever order the DB happened to hand back. + ranked.sort( + key=lambda r: ( + -r["composite"], + r["cost_per_1m_completion"] + if r["cost_per_1m_completion"] is not None + else float("inf"), + r["model_id"], + ) + ) + return ranked diff --git a/schema.sql b/schema.sql index cbc5d5f..54b77ef 100644 --- a/schema.sql +++ b/schema.sql @@ -6,7 +6,7 @@ PRAGMA foreign_keys = ON; -- One row per (model_id, provider). Refreshed by the pricing poller. CREATE TABLE IF NOT EXISTS models ( model_id TEXT NOT NULL, - provider TEXT NOT NULL, -- 'openrouter' | 'neuralwatt' + provider TEXT NOT NULL, -- 'neuralwatt' (only provider today) display_name TEXT, cost_per_1m_prompt REAL, -- USD, null if pricing_tbd cost_per_1m_completion REAL, @@ -18,7 +18,29 @@ CREATE TABLE IF NOT EXISTS models ( supports_tools INTEGER DEFAULT 0, -- boolean 0/1 supports_json_mode INTEGER DEFAULT 0, supports_vision INTEGER DEFAULT 0, - supports_reasoning INTEGER DEFAULT 0, + supports_reasoning INTEGER DEFAULT 0, -- capabilities.reasoning: "API accepts a + -- reasoning param", NOT a quality signal. + -- True for ~90% of the catalog; do not tier on it. + -- Whether reasoning is ON by default (metadata.reasoning.default_enabled), falling back to + -- supports_reasoning when the model exposes no reasoning block. This is the tier-bearing signal. + reasoning_default_enabled INTEGER DEFAULT 0, + + -- Serving class. NeuralWatt ships one base model as several rows that differ only by suffix; + -- these three dimensions are orthogonal, hence ids like 'glm-5.2-short-fast-flex'. They carry + -- no price difference in the catalog, so without these columns such rows tie exactly and the + -- router picks between them arbitrarily. + latency_class TEXT DEFAULT 'standard', -- 'standard' | 'flex' (-flex: discounted + -- async, held server-side during peak) + reasoning_mode TEXT DEFAULT 'default', -- 'default' | 'reduced' (-fast: thinking + -- disabled or capped to a short budget) + context_variant TEXT DEFAULT 'full', -- 'full' | 'short' (-short: 200K pool with + -- a bounded reasoning budget) + + -- Access gating is prose-only in the catalog ("Private preview (grant-gated)", "canary"), so + -- it is parsed from the description. Non-public rows are excluded from routing by default, + -- otherwise the dispatcher selects them and takes a 403. + access_level TEXT DEFAULT 'public', -- 'public' | 'preview' | 'canary' + pricing_tbd INTEGER DEFAULT 0, deprecated INTEGER DEFAULT 0, availability TEXT DEFAULT 'active', -- 'active' | 'deprecated' | 'stale' @@ -54,13 +76,31 @@ CREATE TABLE IF NOT EXISTS energy_observations ( task_category TEXT, prompt_tokens INTEGER, completion_tokens INTEGER, - energy_kwh REAL, -- from response.energy.energy_kwh, when present + energy_kwh REAL, -- response.energy.energy_kwh energy_btu REAL, -- energy_kwh * 3412.14, purely for comedic dashboard value - cost_usd REAL, -- computed from tokens x pricing at call time + + -- Carbon is what design doc §4 actually scores eco on, and NeuralWatt + -- reports it per-request rather than making us derive it. Grid intensity + -- and region are stored alongside because the same energy in a different + -- region is a different carbon figure -- keeping them makes the open + -- question (real-time intensity vs per-model average) answerable later + -- from logged data instead of a re-run. + carbon_g_co2eq REAL, -- response.energy.carbon_g_co2eq + grid_carbon_intensity REAL, -- gCO2/kWh at call time + grid_id TEXT, -- e.g. 'FI' + + -- The provider's own billed figure (response.cost.request_cost_usd), NOT + -- a tokens x list-price estimate. These disagree: flex rows bill roughly + -- 40% under their standard sibling while the catalog advertises both at + -- the same price, so the estimate would be wrong for every flex call. + cost_usd REAL, + allowance_remaining_usd REAL, -- response.cost.allowance_remaining_usd + service_tier TEXT, -- response.service_tier, as billed observed_at TEXT NOT NULL ); CREATE INDEX IF NOT EXISTS idx_models_provider ON models (provider); CREATE INDEX IF NOT EXISTS idx_models_availability ON models (availability); +CREATE INDEX IF NOT EXISTS idx_models_routing ON models (access_level, latency_class, tier); CREATE INDEX IF NOT EXISTS idx_proficiency_category ON proficiency (category); CREATE INDEX IF NOT EXISTS idx_energy_model ON energy_observations (model_id, provider); diff --git a/tests/test_apply_tiering.py b/tests/test_apply_tiering.py index d26ef8f..c3bd9d2 100644 --- a/tests/test_apply_tiering.py +++ b/tests/test_apply_tiering.py @@ -1,9 +1,9 @@ -"""Integration tests for router-package/tier.py — apply_tiering DB upsert. +"""Integration tests for tier.py — apply_tiering DB upsert. Seeds a temp sqlite DB from schema.sql, inserts synthetic model rows with known expected tiers, and asserts the tiering pass writes the right tier for every row (no NULLs), honors the override map, matches the heuristic, -is idempotent, and warns when supports_reasoning is uniformly False. +is idempotent, and warns when reasoning_default_enabled is uniformly False. """ import sqlite3 @@ -41,19 +41,27 @@ def db(tmp_path): def _insert(conn: sqlite3.Connection, rows: list[dict]) -> None: - """Insert synthetic model rows (only the columns the tiering pass reads).""" + """Insert synthetic model rows (only the columns the tiering pass reads). + + ``supports_reasoning`` defaults to mirroring ``reasoning_default_enabled`` + so callers only specify the signal under test; the fallback path is + exercised by setting them independently. + """ for r in rows: + default_enabled = r["reasoning_default_enabled"] conn.execute( """ INSERT INTO models ( - model_id, provider, supports_reasoning, - cost_per_1m_completion, pricing_tbd, last_updated - ) VALUES (?, ?, ?, ?, ?, ?) + model_id, provider, supports_reasoning, reasoning_default_enabled, + reasoning_mode, cost_per_1m_completion, pricing_tbd, last_updated + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) """, ( r["model_id"], r["provider"], - int(r["supports_reasoning"]), + int(r.get("supports_reasoning", default_enabled)), + int(default_enabled), + r.get("reasoning_mode", "default"), r["cost_per_1m_completion"], int(r["pricing_tbd"]), "2026-08-07T00:00:00+00:00", @@ -76,15 +84,15 @@ def _tiers(conn: sqlite3.Connection) -> dict[tuple[str, str], int | None]: def test_every_row_gets_a_tier_in_1_2_3(db): # Given: a mixed catalog covering every heuristic branch _insert(db, [ - {"model_id": "reasoner", "provider": "p", "supports_reasoning": True, + {"model_id": "reasoner", "provider": "p", "reasoning_default_enabled": True, "cost_per_1m_completion": 5.0, "pricing_tbd": False}, - {"model_id": "cheap", "provider": "p", "supports_reasoning": False, + {"model_id": "cheap", "provider": "p", "reasoning_default_enabled": False, "cost_per_1m_completion": 0.25, "pricing_tbd": False}, - {"model_id": "mid", "provider": "p", "supports_reasoning": False, + {"model_id": "mid", "provider": "p", "reasoning_default_enabled": False, "cost_per_1m_completion": 5.0, "pricing_tbd": False}, - {"model_id": "null-cost", "provider": "p", "supports_reasoning": False, + {"model_id": "null-cost", "provider": "p", "reasoning_default_enabled": False, "cost_per_1m_completion": None, "pricing_tbd": False}, - {"model_id": "tbd", "provider": "p", "supports_reasoning": False, + {"model_id": "tbd", "provider": "p", "reasoning_default_enabled": False, "cost_per_1m_completion": 0.10, "pricing_tbd": True}, ]) # When: applying the tiering pass @@ -100,7 +108,7 @@ def test_every_row_gets_a_tier_in_1_2_3(db): def test_override_map_row_matches_override(db): # Given: a reasoning model (heuristic would say 3) overridden to tier 1 _insert(db, [ - {"model_id": "deep-reasoner", "provider": "p", "supports_reasoning": True, + {"model_id": "deep-reasoner", "provider": "p", "reasoning_default_enabled": True, "cost_per_1m_completion": 5.0, "pricing_tbd": False}, ]) # When: applying tiering with an override map @@ -114,11 +122,11 @@ def test_override_map_row_matches_override(db): def test_heuristic_rows_match_expected_tiers(db): # Given: reasoning -> 3, cheap non-reasoning -> 1, everything else -> 2 _insert(db, [ - {"model_id": "reasoner", "provider": "p", "supports_reasoning": True, + {"model_id": "reasoner", "provider": "p", "reasoning_default_enabled": True, "cost_per_1m_completion": 5.0, "pricing_tbd": False}, - {"model_id": "cheap", "provider": "p", "supports_reasoning": False, + {"model_id": "cheap", "provider": "p", "reasoning_default_enabled": False, "cost_per_1m_completion": 0.25, "pricing_tbd": False}, - {"model_id": "mid", "provider": "p", "supports_reasoning": False, + {"model_id": "mid", "provider": "p", "reasoning_default_enabled": False, "cost_per_1m_completion": 5.0, "pricing_tbd": False}, ]) # When: applying the tiering pass @@ -135,11 +143,11 @@ def test_heuristic_rows_match_expected_tiers(db): def test_rerun_is_idempotent(db): # Given: a mixed catalog _insert(db, [ - {"model_id": "reasoner", "provider": "p", "supports_reasoning": True, + {"model_id": "reasoner", "provider": "p", "reasoning_default_enabled": True, "cost_per_1m_completion": 5.0, "pricing_tbd": False}, - {"model_id": "cheap", "provider": "p", "supports_reasoning": False, + {"model_id": "cheap", "provider": "p", "reasoning_default_enabled": False, "cost_per_1m_completion": 0.25, "pricing_tbd": False}, - {"model_id": "mid", "provider": "p", "supports_reasoning": False, + {"model_id": "mid", "provider": "p", "reasoning_default_enabled": False, "cost_per_1m_completion": 5.0, "pricing_tbd": False}, ]) # When: applying the tiering pass twice @@ -153,16 +161,16 @@ def test_rerun_is_idempotent(db): # --- required test 5: sanity-guard warning path --------------------------- -def test_warns_when_supports_reasoning_uniformly_false(db, capsys): - # Given: a catalog where no row supports reasoning +def test_warns_when_reasoning_default_enabled_uniformly_false(db, capsys): + # Given: a catalog where no row reasons by default _insert(db, [ - {"model_id": "cheap", "provider": "p", "supports_reasoning": False, + {"model_id": "cheap", "provider": "p", "reasoning_default_enabled": False, "cost_per_1m_completion": 0.25, "pricing_tbd": False}, - {"model_id": "mid", "provider": "p", "supports_reasoning": False, + {"model_id": "mid", "provider": "p", "reasoning_default_enabled": False, "cost_per_1m_completion": 5.0, "pricing_tbd": False}, ]) # When: applying the tiering pass apply_tiering(db, _make_config()) # Then: a clear warning is emitted (not a hard failure) captured = capsys.readouterr() - assert "supports_reasoning is uniformly False" in captured.err + assert "reasoning_default_enabled is uniformly False" in captured.err diff --git a/tests/test_poller_parsing.py b/tests/test_poller_parsing.py new file mode 100644 index 0000000..0a1dfc3 --- /dev/null +++ b/tests/test_poller_parsing.py @@ -0,0 +1,78 @@ +"""Tests for poller.py's pure catalog-parsing helpers. + +The NeuralWatt catalog encodes serving class in the model id and access +gating only in prose, so both are parsed rather than read from a field. +Cases below are taken from the live catalog. +""" + +import pytest + +from poller import parse_access_level, parse_serving_class + + +# --- serving class -------------------------------------------------------- + +@pytest.mark.parametrize( + "model_id,expected", + [ + # base rows take the schema defaults + ("gemma-4-31b", ("standard", "default", "full")), + ("kimi-k3", ("standard", "default", "full")), + # one dimension at a time + ("kimi-k3-fast", ("standard", "reduced", "full")), + ("kimi-k3-flex", ("flex", "default", "full")), + ("glm-5.2-short", ("standard", "default", "short")), + # the dimensions are orthogonal and combine freely + ("glm-5.2-short-fast", ("standard", "reduced", "short")), + ("glm-5.2-short-flex", ("flex", "default", "short")), + ("glm-5.2-short-fast-flex", ("flex", "reduced", "short")), + ], +) +def test_parse_serving_class(model_id, expected): + assert parse_serving_class(model_id) == expected + + +def test_flash_is_not_read_as_fast(): + # Given: a base model whose name ends in 'flash' — a near-miss for the + # '-fast' suffix that a substring match would misclassify, wrongly + # demoting the cheapest model in the catalog out of tier 1 + # Then: whole-segment matching leaves it a plain standard row + assert parse_serving_class("deepseek-v4-flash") == ("standard", "default", "full") + assert parse_serving_class("deepseek-v4-flash-flex") == ("flex", "default", "full") + + +def test_namespaced_and_mixed_case_ids(): + # Given: the HF-style duplicate row, which carries a namespace and caps + assert parse_serving_class("deepseek-ai/DeepSeek-V4-Flash") == ( + "standard", + "default", + "full", + ) + + +def test_suffix_only_id_is_not_stripped_to_nothing(): + # Given: a degenerate id consisting solely of a suffix token, the loop + # must leave at least one segment rather than consuming the whole id + assert parse_serving_class("fast") == ("standard", "default", "full") + + +# --- access level --------------------------------------------------------- + +@pytest.mark.parametrize( + "display_name,description,expected", + [ + ("GLM-5.2 (short)", "... Private preview (grant-gated).", "preview"), + ("GLM-5.2", "Private GLM-5.2 test canary", "canary"), + # gating can appear in the display name rather than the description + ("DeepSeek V4 Flash 0731 (Canary)", "1M context window.", "canary"), + ("Gemma 4 31B", "Google Gemma 4 31B — multimodal with tool calling.", "public"), + (None, None, "public"), + ], +) +def test_parse_access_level(display_name, description, expected): + assert parse_access_level(display_name, description) == expected + + +def test_preview_takes_precedence_over_canary(): + # Given: prose mentioning both, the more restrictive label wins + assert parse_access_level("x", "Private preview (grant-gated) canary") == "preview" diff --git a/tests/test_routing.py b/tests/test_routing.py new file mode 100644 index 0000000..2a63dc0 --- /dev/null +++ b/tests/test_routing.py @@ -0,0 +1,228 @@ +"""Tests for routing.py — hard filters and weighted ranking. + +The filters are the part that disqualifies outright, so each one gets a case +proving it rejects and a case proving it does not over-reject. +""" + +import pytest + +from routing import ( + effective_cost, + is_eligible, + rank_candidates, + select_candidates, +) + +WEIGHTS = (0.4, 0.2, 0.4) + + +def _row(**overrides) -> dict: + """A routable model row; override one field per test.""" + row = { + "model_id": "m", + "provider": "neuralwatt", + "tier": 2, + "cost_per_1m_completion": 1.0, + "effective_context_window": 100_000, + "availability": "active", + "deprecated": 0, + "access_level": "public", + "latency_class": "standard", + "reasoning_mode": "default", + "context_variant": "full", + } + row.update(overrides) + return row + + +def _eligible(row, **overrides) -> bool: + kwargs = { + "required_context_tokens": 10_000, + "required_tier": 2, + "latency_tolerance": "interactive", + "allowed_access_levels": ["public"], + "exclude_stale": True, + "exclude_deprecated": True, + } + kwargs.update(overrides) + return is_eligible(row, **kwargs) + + +# --- context window ------------------------------------------------------- + +def test_context_window_too_small_is_rejected(): + assert _eligible(_row(effective_context_window=5_000)) is False + + +def test_context_window_exactly_equal_is_accepted(): + # The filter is >=, so a model that exactly fits is still a candidate + assert _eligible(_row(effective_context_window=10_000)) is True + + +def test_null_context_window_is_rejected(): + # Given: a row whose window never got derived. An unknown window cannot be + # shown to fit, and truncating mid-task is worse than routing elsewhere. + assert _eligible(_row(effective_context_window=None)) is False + + +# --- tier floor ----------------------------------------------------------- + +def test_tier_below_required_is_rejected(): + assert _eligible(_row(tier=1), required_tier=2) is False + + +def test_tier_above_required_is_accepted(): + # tier is a floor, not an equality match + assert _eligible(_row(tier=3), required_tier=2) is True + + +def test_null_tier_is_rejected(): + assert _eligible(_row(tier=None)) is False + + +# --- freshness ------------------------------------------------------------ + +def test_stale_row_is_rejected_when_configured(): + assert _eligible(_row(availability="stale")) is False + + +def test_stale_row_is_kept_when_not_excluded(): + assert _eligible(_row(availability="stale"), exclude_stale=False) is True + + +def test_deprecated_row_is_rejected(): + assert _eligible(_row(availability="deprecated")) is False + assert _eligible(_row(deprecated=1)) is False + + +# --- access gating -------------------------------------------------------- + +def test_grant_gated_rows_are_rejected_by_default(): + # Given: the glm-5.2-short* rows, which are private preview. Routing to + # one earns a 403 at dispatch, so they are excluded before scoring. + assert _eligible(_row(access_level="preview")) is False + assert _eligible(_row(access_level="canary")) is False + + +def test_gated_rows_are_admitted_when_the_account_holds_the_grant(): + assert ( + _eligible(_row(access_level="preview"), allowed_access_levels=["public", "preview"]) + is True + ) + + +# --- latency tolerance ---------------------------------------------------- + +def test_flex_row_is_rejected_for_interactive_work(): + # Given: a flex row, held server-side during peak until capacity frees up + assert _eligible(_row(latency_class="flex"), latency_tolerance="interactive") is False + + +def test_flex_row_is_admitted_for_batch_work(): + assert _eligible(_row(latency_class="flex"), latency_tolerance="batch") is True + + +def test_standard_row_is_admitted_for_batch_work(): + # Batch tolerates flex; it does not require it + assert _eligible(_row(latency_class="standard"), latency_tolerance="batch") is True + + +# --- select_candidates ---------------------------------------------------- + +def test_select_candidates_filters_and_preserves_order(): + rows = [ + _row(model_id="keep-1"), + _row(model_id="drop-flex", latency_class="flex"), + _row(model_id="keep-2", tier=3), + _row(model_id="drop-gated", access_level="preview"), + ] + selected = select_candidates( + rows, + required_context_tokens=10_000, + required_tier=2, + latency_tolerance="interactive", + allowed_access_levels=["public"], + exclude_stale=True, + exclude_deprecated=True, + ) + assert [r["model_id"] for r in selected] == ["keep-1", "keep-2"] + + +# --- flex pricing --------------------------------------------------------- + +def test_flex_cost_is_unchanged_at_the_default_multiplier(): + # The catalog lists flex at the same sticker price, so the default asserts + # no discount rather than inventing one + assert effective_cost(_row(latency_class="flex"), 1.0) == 1.0 + + +def test_flex_cost_multiplier_discounts_only_flex_rows(): + assert effective_cost(_row(latency_class="flex"), 0.5) == 0.5 + assert effective_cost(_row(latency_class="standard"), 0.5) == 1.0 + + +def test_effective_cost_of_unknown_price_is_none(): + assert effective_cost(_row(cost_per_1m_completion=None), 0.5) is None + + +# --- ranking -------------------------------------------------------------- + +def test_cheaper_model_wins_when_only_cost_differs(): + rows = [_row(model_id="pricey", cost_per_1m_completion=15.0), + _row(model_id="cheap", cost_per_1m_completion=0.28)] + ranked = rank_candidates(rows, weights=WEIGHTS) + assert ranked[0]["model_id"] == "cheap" + assert ranked[0]["composite"] > ranked[1]["composite"] + + +def test_opposed_full_spreads_tie_at_equal_weight(): + # Given: two candidates at opposite extremes of both cost and proficiency. + # Min-max normalization puts each at 0.0/1.0, and cost and proficiency + # carry the same 0.4 weight, so the composites land exactly equal and the + # cost tiebreak decides. Worth pinning: with only two candidates the + # normalization always produces a full spread, which makes the default + # weights much tie-ier than they look. + rows = [ + _row(model_id="cheap-bad", cost_per_1m_completion=0.1, proficiency=0.0), + _row(model_id="dear-good", cost_per_1m_completion=0.2, proficiency=1.0), + ] + ranked = rank_candidates(rows, weights=WEIGHTS) + assert ranked[0]["composite"] == ranked[1]["composite"] + assert ranked[0]["model_id"] == "cheap-bad" + + +def test_proficiency_wins_when_weighted_above_cost(): + rows = [ + _row(model_id="cheap-bad", cost_per_1m_completion=0.1, proficiency=0.0), + _row(model_id="dear-good", cost_per_1m_completion=0.2, proficiency=1.0), + ] + ranked = rank_candidates(rows, weights=(0.2, 0.2, 0.6)) + assert ranked[0]["model_id"] == "dear-good" + + +def test_missing_proficiency_and_eco_are_neutral_not_penalized(): + ranked = rank_candidates([_row()], weights=WEIGHTS) + assert ranked[0]["proficiency_score"] == 0.5 + assert ranked[0]["eco_score"] == 0.5 + + +def test_ties_break_deterministically_by_cost_then_model_id(): + # Given: the real problem — sibling rows identical on every scored + # dimension. Order must not depend on what the DB happened to return. + rows = [_row(model_id="glm-b"), _row(model_id="glm-a"), _row(model_id="glm-c")] + first = rank_candidates(rows, weights=WEIGHTS) + second = rank_candidates(list(reversed(rows)), weights=WEIGHTS) + assert [r["model_id"] for r in first] == ["glm-a", "glm-b", "glm-c"] + assert [r["model_id"] for r in first] == [r["model_id"] for r in second] + + +def test_ranking_reports_the_sub_scores_that_produced_the_winner(): + ranked = rank_candidates([_row(proficiency=0.8)], weights=WEIGHTS) + top = ranked[0] + assert top["composite"] == pytest.approx( + 0.4 * top["cost_score"] + 0.2 * top["eco_score"] + 0.4 * top["proficiency_score"] + ) + + +def test_empty_candidate_set_ranks_to_empty(): + assert rank_candidates([], weights=WEIGHTS) == [] diff --git a/tests/test_scoring.py b/tests/test_scoring.py index 059b775..1752b99 100644 --- a/tests/test_scoring.py +++ b/tests/test_scoring.py @@ -1,4 +1,4 @@ -"""Tests for router-package/scoring.py — pure scoring functions. +"""Tests for scoring.py — pure scoring functions. Covers the required edge cases from the router-scoring-tiering plan: cost min-max inversion (incl. $0-cheapest and None handling), eco diff --git a/tests/test_tiering.py b/tests/test_tiering.py index 50d09d6..1ce9a26 100644 --- a/tests/test_tiering.py +++ b/tests/test_tiering.py @@ -1,35 +1,43 @@ -"""Tests for router-package/tiering.py — pure tier resolver. +"""Tests for tiering.py — pure tier resolver. -Covers the required cases from the router-scoring-tiering plan: -override-wins, reasoning -> 3, cheap non-reasoning -> 1, and every -fallback path -> 2 (missing/NULL cost, pricing_tbd, cost >= threshold, -including the strict-< boundary at exactly the threshold). +Covers override-wins, effective-reasoning -> 3, cheap non-reasoning -> 1, +and every fallback path -> 2 (missing/NULL cost, pricing_tbd, cost >= the +threshold, including the strict-< boundary at exactly the threshold). + +Also covers the two signals that replaced ``supports_reasoning``: a +``-fast`` row (``reasoning_mode='reduced'``) does not earn tier 3, and +``supports_reasoning`` is consulted only when ``reasoning_default_enabled`` +is absent. """ -from tiering import resolve_tier +from tiering import reasoning_effectively_on, resolve_tier def _row( *, model_id: str = "m", - supports_reasoning: bool = False, + reasoning_default_enabled: bool = False, + reasoning_mode: str = "default", cost_per_1m_completion: float | None = 1.0, pricing_tbd: bool = False, + supports_reasoning: bool = False, ) -> dict: """Build a minimal models-table row dict (ModelRow-shaped).""" return { "model_id": model_id, - "supports_reasoning": supports_reasoning, + "reasoning_default_enabled": reasoning_default_enabled, + "reasoning_mode": reasoning_mode, "cost_per_1m_completion": cost_per_1m_completion, "pricing_tbd": pricing_tbd, + "supports_reasoning": supports_reasoning, } # --- rule 1: override wins ------------------------------------------------ def test_override_wins_over_heuristic(): - # Given: a reasoning-capable model (heuristic would say 3) with an override to 1 - row = _row(model_id="deep-reasoner", supports_reasoning=True) + # Given: a reasoning model (heuristic would say 3) with an override to 1 + row = _row(model_id="deep-reasoner", reasoning_default_enabled=True) override_map = {"deep-reasoner": 1} # When: resolving the tier tier = resolve_tier(row, cheap_completion_max=1.00, override_map=override_map) @@ -40,8 +48,8 @@ def test_override_wins_over_heuristic(): def test_override_applies_to_all_providers_for_same_model_id(): # Given: the same model_id on two providers, override keyed by model_id only override_map = {"shared-model": 2} - row_a = _row(model_id="shared-model", supports_reasoning=True) - row_b = _row(model_id="shared-model", supports_reasoning=True) + row_a = _row(model_id="shared-model", reasoning_default_enabled=True) + row_b = _row(model_id="shared-model", reasoning_default_enabled=True) # When: resolving both rows tier_a = resolve_tier(row_a, cheap_completion_max=1.00, override_map=override_map) tier_b = resolve_tier(row_b, cheap_completion_max=1.00, override_map=override_map) @@ -50,17 +58,67 @@ def test_override_applies_to_all_providers_for_same_model_id(): assert tier_b == 2 -# --- rule 2: reasoning -> 3 ---------------------------------------------- +# --- rule 2: effective reasoning -> 3 ------------------------------------- -def test_reasoning_capable_tiers_to_3(): - # Given: a reasoning-capable model with any cost - row = _row(model_id="deep-reasoner", supports_reasoning=True, cost_per_1m_completion=5.0) +def test_reasoning_default_enabled_tiers_to_3(): + # Given: a model that reasons by default, with any cost + row = _row( + model_id="deep-reasoner", + reasoning_default_enabled=True, + cost_per_1m_completion=5.0, + ) # When: resolving the tier tier = resolve_tier(row, cheap_completion_max=1.00, override_map={}) # Then: it is tier 3 (frontier / high-stakes) assert tier == 3 +def test_supports_reasoning_alone_does_not_tier_to_3(): + # Given: a model that merely ACCEPTS a reasoning param but does not use it + # by default — true for all but two rows of the live catalog + row = _row( + model_id="accepts-param", + supports_reasoning=True, + reasoning_default_enabled=False, + cost_per_1m_completion=5.0, + ) + # When: resolving the tier + tier = resolve_tier(row, cheap_completion_max=1.00, override_map={}) + # Then: it is not promoted to tier 3 on capability alone + assert tier == 2 + + +def test_fast_variant_does_not_inherit_tier_3(): + # Given: a '-fast' row — same weights, thinking disabled or capped + row = _row( + model_id="glm-5.2-fast", + reasoning_default_enabled=True, + reasoning_mode="reduced", + cost_per_1m_completion=4.5, + ) + # When: resolving the tier + tier = resolve_tier(row, cheap_completion_max=1.00, override_map={}) + # Then: reduced reasoning drops it out of tier 3 + assert tier == 2 + + +def test_missing_default_enabled_falls_back_to_supports_reasoning(): + # Given: a model exposing no reasoning block (the kimi-k2.7-code family), + # so reasoning_default_enabled is absent entirely + row = { + "model_id": "kimi-k2.7-code", + "reasoning_mode": "default", + "cost_per_1m_completion": 4.0, + "pricing_tbd": False, + "supports_reasoning": True, + } + # When: resolving the tier + tier = resolve_tier(row, cheap_completion_max=1.00, override_map={}) + # Then: the capability flag is used as the fallback signal + assert reasoning_effectively_on(row) is True + assert tier == 3 + + # --- rule 3: cheap non-reasoning -> 1 ------------------------------------- def test_no_reasoning_cheap_cost_tiers_to_1(): @@ -72,6 +130,22 @@ def test_no_reasoning_cheap_cost_tiers_to_1(): assert tier == 1 +def test_cheap_reasoning_capable_but_off_by_default_reaches_tier_1(): + # Given: deepseek-v4-flash's real shape — accepts a reasoning param, does + # not reason by default, and costs $0.28/1M. The previous heuristic put + # this in tier 3 because it checked the capability flag before cost. + row = _row( + model_id="deepseek-v4-flash", + supports_reasoning=True, + reasoning_default_enabled=False, + cost_per_1m_completion=0.28, + ) + # When: resolving the tier + tier = resolve_tier(row, cheap_completion_max=1.00, override_map={}) + # Then: it lands in tier 1 where it belongs + assert tier == 1 + + # --- rule 4: fallback -> 2 ------------------------------------------------ def test_no_reasoning_cost_at_or_above_threshold_tiers_to_2(): diff --git a/tier.py b/tier.py index 9463a1d..8da1ed6 100644 --- a/tier.py +++ b/tier.py @@ -22,10 +22,8 @@ import sys from config import RouterConfig, load_config from tiering import resolve_tier -DB_PATH = "router.db" - WARNING_UNIFORMLY_FALSE = ( - "WARNING: supports_reasoning is uniformly False — verify poller field mapping" + "WARNING: reasoning_default_enabled is uniformly False — verify poller field mapping" ) @@ -36,24 +34,34 @@ def apply_tiering(conn: sqlite3.Connection, config: RouterConfig) -> None: ``UPDATE models SET tier = ? WHERE model_id = ? AND provider = ?`` per row, then commits. Only the ``tier`` column is touched. - Sanity guard (Metis finding #3): if ``supports_reasoning`` is True for - ZERO rows across the whole table, emit a warning to stderr — a uniformly - False field suggests a poller field-mapping bug that would silently zero + Sanity guard: if ``reasoning_default_enabled`` is True for ZERO rows + across the whole table, emit a warning to stderr — a uniformly False + field suggests a poller field-mapping bug that would silently zero tier-3. This is a warning, not a hard failure. """ rows = conn.execute( - "SELECT model_id, provider, supports_reasoning, " - "cost_per_1m_completion, pricing_tbd FROM models" + "SELECT model_id, provider, supports_reasoning, reasoning_default_enabled, " + "reasoning_mode, cost_per_1m_completion, pricing_tbd FROM models" ).fetchall() - if not any(row[2] for row in rows): + if not any(row[3] for row in rows): print(WARNING_UNIFORMLY_FALSE, file=sys.stderr) - for model_id, provider, supports_reasoning, cost, pricing_tbd in rows: + for ( + model_id, + provider, + supports_reasoning, + reasoning_default_enabled, + reasoning_mode, + cost, + pricing_tbd, + ) in rows: tier = resolve_tier( { "model_id": model_id, "supports_reasoning": bool(supports_reasoning), + "reasoning_default_enabled": bool(reasoning_default_enabled), + "reasoning_mode": reasoning_mode, "cost_per_1m_completion": cost, "pricing_tbd": bool(pricing_tbd), }, @@ -68,9 +76,9 @@ def apply_tiering(conn: sqlite3.Connection, config: RouterConfig) -> None: def main() -> int: - conn = sqlite3.connect(DB_PATH) - conn.execute("PRAGMA foreign_keys = ON") cfg = load_config("config.yaml") + conn = sqlite3.connect(cfg.database.path) + conn.execute("PRAGMA foreign_keys = ON") apply_tiering(conn, cfg) conn.close() print("tiering applied") diff --git a/tiering.py b/tiering.py index c0a7e70..fbb9aaa 100644 --- a/tiering.py +++ b/tiering.py @@ -3,30 +3,59 @@ This module is deliberately free of I/O: no DB reads, no file reads, no config import at runtime. The model row and thresholds are passed as arguments so the resolver stays testable and reusable. The DB upsert -(``apply_tiering``) is a separate concern (Todo 4) and lives elsewhere. +(``apply_tiering``) is a separate concern and lives in ``tier.py``. Tier semantics (design doc §4 / config.yaml ``tiers`` labels): 1 = cheap / simple 2 = mid / general 3 = frontier / high-stakes +Why not ``supports_reasoning``: that column mirrors the catalog's +``capabilities.reasoning``, which only means "this endpoint accepts a +reasoning parameter". It is true for all but two rows in the NeuralWatt +catalog, so tiering on it collapsed 17 of 19 models into tier 3 and left +tier 1 empty. The discriminating signals are ``reasoning_default_enabled`` +(whether the model actually thinks unless told otherwise) and +``reasoning_mode`` (whether this is a ``-fast`` row with thinking disabled +or capped). + Resolution precedence (EXACT order): 1. If ``override_map`` has this model's ``model_id`` -> that value wins. - 2. ``supports_reasoning`` True -> 3. - 3. ``cost_per_1m_completion`` is not None AND < ``cheap_completion_max`` - -> 1. - 4. Otherwise (missing/NULL completion cost, OR ``pricing_tbd`` True, OR - cost >= threshold) -> 2. + 2. Reasoning is "effectively on" when ``reasoning_default_enabled`` is + True AND ``reasoning_mode`` is not ``'reduced'``. A ``-fast`` row is + the same weights served without chain-of-thought, so it does not earn + tier 3 on its sibling's behalf. + 3. Reasoning effectively OFF and ``cost_per_1m_completion`` is not None + and not ``pricing_tbd`` and < ``cheap_completion_max`` -> 1. + 4. Reasoning effectively ON -> 3. + 5. Otherwise (missing/NULL cost, ``pricing_tbd``, or cost >= threshold) + -> 2. + +Rules 3 and 4 are mutually exclusive, so their relative order is immaterial; +cost is stated first because the previous version's reasoning-before-cost +ordering is what pushed $0.28/1M models into tier 3. NULL-cost semantics: tier 1 REQUIRES a non-NULL completion cost strictly below the threshold. A strong-cheap non-reasoning model being mis-tiered to -1 is ACCEPTED — the override map is the escape hatch (Metis finding #9). -No second capability dimension is added to the heuristic. +1 is ACCEPTED — the override map is the escape hatch. """ from __future__ import annotations +def reasoning_effectively_on(model_row: dict) -> bool: + """Whether this row actually reasons by default. + + Falls back to ``supports_reasoning`` when ``reasoning_default_enabled`` + is absent, matching the poller's own fallback for models that expose no + reasoning block (the kimi-k2.7-code family). + """ + default_enabled = model_row.get("reasoning_default_enabled") + if default_enabled is None: + default_enabled = model_row.get("supports_reasoning", False) + return bool(default_enabled) and model_row.get("reasoning_mode") != "reduced" + + def resolve_tier( model_row: dict, cheap_completion_max: float, @@ -36,8 +65,9 @@ def resolve_tier( ``model_row`` is a row-like dict with at least these keys (from the ``models`` table / ``ModelRow``): ``model_id`` (str), - ``supports_reasoning`` (bool), ``cost_per_1m_completion`` (float or - None), ``pricing_tbd`` (bool). + ``reasoning_default_enabled`` (bool), ``reasoning_mode`` (str), + ``cost_per_1m_completion`` (float or None), ``pricing_tbd`` (bool). + ``supports_reasoning`` is consulted only as a fallback. ``override_map`` is keyed by ``model_id`` only and therefore applies to ALL providers serving that model (documented limitation vs the @@ -48,15 +78,14 @@ def resolve_tier( if model_id in override_map: return override_map[model_id] - if model_row["supports_reasoning"]: - return 3 + if not reasoning_effectively_on(model_row): + cost = model_row["cost_per_1m_completion"] + if ( + not model_row["pricing_tbd"] + and cost is not None + and cost < cheap_completion_max + ): + return 1 + return 2 - cost = model_row["cost_per_1m_completion"] - if ( - not model_row["pricing_tbd"] - and cost is not None - and cost < cheap_completion_max - ): - return 1 - - return 2 + return 3 -- 2.49.1 From 0fce98d41607bc0f2047bed1617f93d1951036ba Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Wed, 12 Aug 2026 00:14:20 -0400 Subject: [PATCH 02/20] feat: score cost and eco on measured billing and carbon, not list price MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds seed_energy.py, which runs a fixed reference task N times per routable model and writes energy_observations rows tagged 'seed_reference'. Scoring now reads mean measured USD billed and mean measured gCO2eq from those rows instead of the catalog's list price. List price was not merely imprecise, it was inverted: deepseek-v4-flash lists 33% cheaper than gemma-4-31b ($0.28 vs $0.42 per 1M) but costs 95% more to run ($9.80e-05 vs $5.02e-05) and emits 284% more carbon. Billing is min($8.00/kWh x energy, 3 x list token price), validated against all 65 samples; the ceiling bound twice, both deepseek energy spikes, matching to the cent. That rule stays in the docs as rationale — no billing formula is in the code, since measured cost already accounts for it, including the flex tier. flex_cost_multiplier is therefore removed rather than left as a guess. Cost and eco deliberately stay separate axes. Collapsing them looked right — cost = 8 x energy, so they seemed to be one signal — but carbon is energy times the serving region's grid intensity, and the catalog spans 37 gCO2/kWh (FI) to 505 (US-MIDA-PJM). They rank models differently: glm-5.2-fast is 2nd cheapest and 6th cleanest; kimi-k3-flex draws 3.7x less energy than kimi-k2.7-code while emitting 3.6x more carbon. A test pins it. Only reference-workload observations steer routing. Organic traffic stays logged for accounting but is excluded, because per-request energy varies more with request shape than with the model — 19x across organic traffic on a single model, purely from differing prompt and completion sizes. cost_score and eco_score were byte-identical implementations; both now alias one normalize_inverted. Effect: 27 routing decisions (9 categories x 3 tiers) went from 2 distinct models to 3, and tier 1 flipped from deepseek-v4-flash to gemma-4-31b. Category still cannot influence a decision — proficiency is the only category-dependent term and remains empty, which is now the single highest-value gap. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 120 ++++++++++-------- config.py | 8 -- config.yaml | 9 +- design/local-llm-model-router.md | 35 +++++- dispatcher.py | 63 +++++++--- routing.py | 41 ++----- scoring.py | 50 ++++---- seed_energy.py | 201 +++++++++++++++++++++++++++++++ tests/test_routing.py | 67 ++++++----- 9 files changed, 427 insertions(+), 167 deletions(-) create mode 100644 seed_energy.py diff --git a/CLAUDE.md b/CLAUDE.md index 9ff7a16..4d571f0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,7 +26,7 @@ can be added later without a migration. classification - FastAPI for the dispatcher service -## Billing is per-kWh, not per-token — this invalidates `cost_score` +## Billing is per-kWh, not per-token — RESOLVED, scoring now uses measured cost **Measured against the live API, 2026-08-11.** Neuralwatt bills a flat **$8.00 per kWh** and the catalog's `input_per_million` / @@ -43,27 +43,46 @@ across five models; `cost_usd / energy_kwh` came back 8.00 every time: \* rounding — billed cost is quantized to ~1e-06. -Two consequences, both load-bearing: +The precise rule, validated against all 65 samples of the reference sweep +(61/65 within 2%; the 4 outliers are microdollar rounding, not misses): -1. **List price ranks models wrong.** `kimi-k2.7-code-fast` lists at $4/1M - and `kimi-k3-fast` at $15/1M, but on the same prompt the "cheap" one cost - **10x more** ($2.25e-03 vs $2.17e-04) because it burned 10x the energy. - `scoring.cost_score` currently reads `cost_per_1m_completion`, so it is - ranking on a number nobody is billed. -2. **`cost` and `eco` are the same axis.** `cost_usd = 8.00 x energy_kwh` - exactly, so `w_cost` (0.4) and `w_eco` (0.2) put 0.6 of the weight on one - underlying signal and leave proficiency at 0.4. The balance in - `config.yaml` is not the balance in effect. They separate only via grid - carbon intensity (`grid_carbon_intensity_gco2perkwhr`, currently 37.0 on - grid `FI`), which matters only when routing across regions. +``` +cost_usd = min( $8.00/kWh x energy_kwh , 3 x list token price ) +``` -Also: energy for the *same* token count varies up to 4x run-to-run (same -18->900 tokens billed 1.8e-05 through 8.8e-05), so a single observation is -not a reliable per-model estimate — this needs several samples per model -before it can drive routing. +The ceiling bound in only 2 of 65 samples, both `deepseek-v4-flash` energy +spikes, and matched to the cent: 31 prompt x $0.14/1M + 400 completion x +$0.28/1M = 1.1634e-04, x3 = 3.4902e-04, billed 0.000349. -**Not yet acted on.** Deciding what replaces `cost_score` is the open design -call, see "Pick up here" below. +**List price ranks models backwards.** Not approximately — invertedly: + +| | list $/1M | actually billed | gCO2eq | +|---|---|---|---| +| `deepseek-v4-flash` | **0.28** | 9.80e-05 | 8.90e-04 | +| `gemma-4-31b` | 0.42 | **5.02e-05** | **2.32e-04** | + +deepseek lists 33% cheaper, costs 95% more, and emits 284% more carbon. + +**But cost and eco are NOT the same axis** — the tempting simplification, and +it is wrong. Cost tracks energy, but carbon is energy x the serving region's +grid intensity, and models run in different regions: + +| grid | gCO2/kWh | models | +|---|---|---| +| `FI` | 37 | most of the catalog | +| `FI` (reported) | 475 | `glm-5.2-fast`, `glm-5.2-flex` | +| `US-MIDA-PJM` | 505 | the `kimi-k3` family | + +A 13.6x spread, so the two axes disagree: `glm-5.2-fast` is the 2nd cheapest +model and only the 6th cleanest; `kimi-k3-flex` draws 3.7x *less* energy than +`kimi-k2.7-code` while emitting 3.6x *more* carbon. Weighting them separately +is load-bearing, and `tests/test_routing.py` pins it. + +**Resolved.** `cost` and `eco` now come from mean *measured* USD and gCO2eq +over the reference sweep, so no billing formula lives in the code at all — +the rule above is documentation for why list price isn't used, not logic. +`flex_cost_multiplier` is gone: a flex row's measured cost already is its +flex cost. ## What's built and working @@ -77,9 +96,14 @@ call, see "Pick up here" below. correct. - `config.yaml` / `config.py` — weights, thresholds, provider settings, Pydantic-validated. -- `scoring.py` — pure min-max normalize-and-invert for cost/eco, plus the - weighted composite. (Correct as written; the *input* is the problem — see - the billing section.) +- `scoring.py` — one `normalize_inverted` (cost and eco normalize + identically; they differ only in what is fed to them) plus the weighted + composite. +- `seed_energy.py` — runs a fixed reference task N times per routable model + and writes `energy_observations` rows tagged `seed_reference`. This is what + makes `cost` and `eco` real numbers instead of the neutral 0.5. Re-run it + after the catalog gains models: `python seed_energy.py --samples 5` + (13 models x 5 = 65 calls, and the whole sweep cost **under a cent**). - `tiering.py` / `tier.py` — pure tier resolver + the DB pass that applies it. - `routing.py` — pure hard filters and ranking. - `dispatcher.py` — FastAPI service. `GET /health`, `POST /route` (classify @@ -151,44 +175,46 @@ that request was two orders of magnitude low. ## What's NOT built yet — pick up here -1. **Replace `cost_score`'s input** (see the billing section). It currently - reads catalog list price, which is not billed. Options: score on observed - mean energy per model (needs seeding, and several samples each given the - 4x variance), or drop `cost` as a separate axis and let `eco` carry it, - re-weighting `config.yaml` accordingly. This is a design call, not a - mechanical fix. +1. **Benchmark/proficiency poller — the last inert axis.** Nothing populates + `proficiency`, so `proficiency_score` is 0.5 for every candidate and 0.4 of + the weight does nothing. Concretely: `task_category` **cannot change a + routing decision**, because proficiency is the only category-dependent term + in the formula. Across 9 categories x 3 tiers, all 9 categories still pick + identically. Needs a leaderboard path (LMSYS Arena, LiveBench, Aider + polyglot — no unified API) and a self-eval harness writing + `source='self_eval'`. The blending rule is already in `config.yaml`, and + `seed_energy.py` is a working template for the sweep shape. -2. **Benchmark/proficiency poller** — nothing populates `proficiency` yet, so - `proficiency_score` returns the neutral 0.5 for every candidate and 0.4 of - the weight is currently inert. Needs a leaderboard-scraping path (LMSYS - Arena, LiveBench, Aider polyglot — no unified API) and a self-eval harness - writing `source='self_eval'`. Blending rule is already in `config.yaml`. + Until this lands, the ~10s classifier round-trip is buying only a tier + number — the category half of its output is discarded in effect. -3. **Energy seeding sweep** — `energy_observations` starts empty, so `eco` is - also neutral 0.5 until real traffic accrues. A fixed prompt run N times - against each routable model would seed it; the ad-hoc version of this is - what produced the $8/kWh table above. +2. **Sampling depth.** 5 samples/model was enough to separate models (they + span 32x in energy) but within-model spread reached 133x on + `deepseek-v4-flash`. Means are usable for ranking; they are not tight. More + samples, or a trimmed mean, would firm this up. -4. **Escalation feedback loop** — low-confidence tier bumping works +3. **Escalation feedback loop** — low-confidence tier bumping works (`apply_escalation`), but there's no way for a human or downstream agent to flag "this was routed wrong" after the fact and retry one tier up. -5. **Scheduling** — `poller.py` needs a cron/systemd timer. No unit file here. - ## Known open questions -- Given billing is per-kWh, is `cost` still a meaningful separate axis from - `eco`, or should the weights collapse to energy + proficiency? -- Energy per request varies up to 4x for identical token counts. How many - samples per model before an energy estimate is trustworthy enough to route - on? +- Answered: cost and eco stay separate axes — grid intensity spans 13.6x + across the catalog, so they rank models differently. +- `glm-5.2-fast` reports `grid_id: FI` at 475 gCO2/kWh while everything else + on `FI` reports 37. Either the region label or the intensity is wrong for + those rows; worth confirming with NeuralWatt before trusting eco for the + GLM family specifically. +- Within-model energy spread reached 133x. How many samples before a per-model + mean is tight enough to weight heavily rather than just order by? - How much context-assembly (RAG-style retrieval) belongs in the classifier step vs. a separate pre-step? Leaning decoupled, undecided. - Self-run eval set: minimum viable task set to start proficiency scoring without it becoming its own maintenance burden? - Should `eco_score` use real-time grid carbon intensity per request or a - stable per-model average? `grid_carbon_intensity` and `grid_id` are logged - per observation, so this stays answerable from data either way. + stable per-model average? Currently the latter, from the reference sweep. + `grid_carbon_intensity` and `grid_id` are logged per observation, so this + stays answerable from data without a re-run. ## Setup diff --git a/config.py b/config.py index c0ace7a..0598738 100644 --- a/config.py +++ b/config.py @@ -88,7 +88,6 @@ class TieringConfig(BaseModel): class RoutingConfig(BaseModel): allowed_access_levels: list[str] default_latency_tolerance: str - flex_cost_multiplier: float @field_validator("allowed_access_levels") @classmethod @@ -113,13 +112,6 @@ class RoutingConfig(BaseModel): ) return v - @field_validator("flex_cost_multiplier") - @classmethod - def multiplier_in_range(cls, v: float) -> float: - if not (0.0 < v <= 1.0): - raise ValueError("routing.flex_cost_multiplier must be in (0, 1]") - return v - class EscalationConfig(BaseModel): enabled: bool diff --git a/config.yaml b/config.yaml index 4369cfd..38149ec 100644 --- a/config.yaml +++ b/config.yaml @@ -68,12 +68,9 @@ routing: # interactive, so a request has to opt in via latency_tolerance. default_latency_tolerance: interactive # 'interactive' | 'batch' - # The catalog advertises flex at the SAME sticker price as standard serving, - # even though flex describes itself as "billed at a reduced rate". Until that - # discount is confirmed against a real invoice this stays 1.0 rather than - # inventing a discount the scorer would then act on. Set it below 1.0 once - # you have billing data. - flex_cost_multiplier: 1.0 + # There is deliberately no flex discount knob. Cost scoring reads the mean + # cost actually billed for the reference workload (see seed_energy.py), and + # a flex row's measured cost already is its flex cost. freshness: stale_after_days: 3 diff --git a/design/local-llm-model-router.md b/design/local-llm-model-router.md index 9bd2532..561cb01 100644 --- a/design/local-llm-model-router.md +++ b/design/local-llm-model-router.md @@ -220,8 +220,30 @@ Two further wrinkles before this can be fixed: scoring needs a seeding sweep (fixed prompt, N runs per routable model) before it can drive anything. -**Open decision:** either re-base `cost_score` on observed mean energy, or drop cost as a -separate axis entirely and let `eco` carry it with re-tuned weights. Not yet actioned. +### 4.2 Resolved: measured cost, and the two axes stay separate + +Both sub-scores now read **measured** values from a fixed reference workload +(`seed_energy.py`, 13 models x 5 samples, tagged `seed_reference`): mean USD actually billed +for cost, mean gCO2eq for eco. No billing formula lives in the code — the rule in §4.1 is +documentation for *why* list price is not used, not logic. + +The tempting simplification — cost = 8 x energy, therefore cost and eco are one axis — is +**wrong**, and the sweep is what caught it. Cost tracks energy, but carbon is energy x the +serving region's grid intensity, and the catalog spans 37 gCO2/kWh (`FI`) to 505 +(`US-MIDA-PJM`). So the axes disagree: `glm-5.2-fast` is the 2nd cheapest model and the 6th +cleanest; `kimi-k3-flex` draws 3.7x less energy than `kimi-k2.7-code` while emitting 3.6x more +carbon. Collapsing them would have silently picked a side. + +Only reference-workload observations feed scoring. Organic traffic stays in the table for cost +accounting but is excluded from routing, because per-request energy varies far more with the +*shape* of a request than with the model — across organic traffic on a single model it spanned +19x purely from differing prompt and completion sizes. Averaging that in would rank models by +what they happened to be asked. + +Effect on the hard-filter-plus-score pipeline: with list-price scoring, 27 routing decisions +(9 categories x 3 tiers) produced 2 distinct models. With measured scoring, 3 — and tier 1 +flipped from `deepseek-v4-flash` to `gemma-4-31b`, correctly. Category still has zero influence, +because `proficiency` remains empty and is the only category-dependent term (§3.3). ## 5. Freshness / scheduled jobs @@ -278,8 +300,9 @@ reported together. **done** — `/health`, `/route` (dry run), `/dispatch` 3. ~~Local classifier prompt/schema~~ **done** — the allowed category list is injected from config at call time, since the model otherwise invents labels that join against nothing -4. **Re-base the cost axis** (§4.1) — the open design call, and it gates any claim that routing - decisions are actually cost-optimal -5. Seed `energy_observations` with a fixed-prompt sweep so `eco` stops returning neutral 0.5 -6. Populate `proficiency` — currently empty, so 0.4 of the composite weight is inert +4. ~~Re-base the cost axis~~ **done** (§4.2) — cost and eco now read measured billed USD and + gCO2eq rather than list price +5. ~~Seed `energy_observations` with a fixed-prompt sweep~~ **done** — `seed_energy.py` +6. **Populate `proficiency`** — the last inert axis. 0.4 of the composite weight does nothing, + and `task_category` cannot change a routing decision until it does 7. Wire into one real workflow (coding agent subagent spin-off) end to end before generalizing diff --git a/dispatcher.py b/dispatcher.py index 9735345..72715d3 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -55,6 +55,11 @@ from routing import BATCH, INTERACTIVE, rank_candidates, select_candidates ROUTER_MODEL = "auto" ROUTER_MODEL_BATCH = "auto:batch" +# Observations from the fixed reference workload in seed_energy.py. Only +# these steer routing; organic traffic is logged for accounting but varies +# too much with request shape to compare models by. +SEED_CATEGORY = "seed_reference" + # Rough chars-per-token. Only used as a FLOOR on the classifier's context # estimate: a real coding session sends the whole conversation, and the # classifier — which sees that conversation and is asked to estimate its own @@ -106,11 +111,16 @@ class Candidate(BaseModel): model_id: str provider: str tier: int - cost_per_1m_completion: Optional[float] latency_class: str reasoning_mode: str context_variant: str effective_context_window: int + # What the scoring actually read: mean USD billed and mean gCO2eq for the + # reference workload. `list_price_per_1m` is shown alongside because it is + # NOT what gets billed and the gap between them is worth being able to see. + cost: Optional[float] = None + eco: Optional[float] = None + list_price_per_1m: Optional[float] = None composite: float cost_score: float eco_score: float @@ -239,36 +249,58 @@ def load_candidates(conn: sqlite3.Connection, category: str) -> list[dict]: Filtering happens in ``routing.py`` rather than in SQL so the hard filters stay in one testable place; the catalog is 19 rows. - ``eco`` is mean observed carbon, not energy: design doc §4 scores eco on - gCO2eq, and the same kWh in a different region is a different carbon - figure. Averaging over observations is the "stable per-model average" - answer to the doc's open question — grid intensity and region are logged - per observation, so switching to a request-time figure later needs no - re-run. Models with no observations yet come back None and take the - neutral 0.5 from ``scoring``, so a cold table never penalizes anyone. + ``cost`` and ``eco`` are both measured, and both come ONLY from the + fixed reference workload written by ``seed_energy.py`` + (``task_category='seed_reference'``). + + Measured, because the catalog's list price is not what is billed — + NeuralWatt charges per kWh, so on the same prompt kimi-k2.7-code-fast + ($4/1M listed) cost 10x more than kimi-k3-fast ($15/1M listed). Mean + observed ``cost_usd`` needs no billing formula in the code and already + accounts for the flex tier and the per-request price ceiling. + + Reference-only, because energy depends far more on the shape of a + request than on the model: across organic traffic on a single model, + per-request energy spanned 19x purely because prompt and completion + sizes differed. Averaging that in would rank models by what they + happened to be asked. Organic observations stay in the table for cost + accounting; they just don't steer routing. + + Models with no reference samples come back None on both and take the + neutral 0.5 from ``scoring``, so a newly listed model is never penalized + for not having been swept yet. """ rows = conn.execute( """ SELECT m.*, p.blended_score AS proficiency, - (SELECT AVG(e.carbon_g_co2eq) - FROM energy_observations e - WHERE e.model_id = m.model_id - AND e.provider = m.provider - AND e.carbon_g_co2eq IS NOT NULL) AS eco + s.mean_cost AS cost, + s.mean_carbon AS eco FROM models m LEFT JOIN proficiency p ON p.model_id = m.model_id AND p.provider = m.provider AND p.category = ? + LEFT JOIN ( + SELECT model_id, provider, + AVG(cost_usd) AS mean_cost, + AVG(carbon_g_co2eq) AS mean_carbon + FROM energy_observations + WHERE task_category = ? + GROUP BY model_id, provider + ) s + ON s.model_id = m.model_id + AND s.provider = m.provider """, - (category,), + (category, SEED_CATEGORY), ).fetchall() return [dict(r) for r in rows] def _to_candidate(row: dict) -> Candidate: - return Candidate(**{k: row[k] for k in Candidate.model_fields if k in row}) + fields = {k: row[k] for k in Candidate.model_fields if k in row} + fields["list_price_per_1m"] = row.get("cost_per_1m_completion") + return Candidate(**fields) def route(req: TaskRequest) -> RouteResponse: @@ -313,7 +345,6 @@ def route(req: TaskRequest) -> RouteResponse: ranked = rank_candidates( eligible, weights=(cfg.weights.cost, cfg.weights.eco, cfg.weights.proficiency), - flex_cost_multiplier=cfg.routing.flex_cost_multiplier, ) return RouteResponse( diff --git a/routing.py b/routing.py index 259bb64..cc3207b 100644 --- a/routing.py +++ b/routing.py @@ -97,44 +97,29 @@ def select_candidates( ] -def effective_cost(row: dict, flex_cost_multiplier: float) -> float | None: - """Completion cost, discounted if this row bills at the flex rate. - - The catalog lists flex at the same sticker price as standard serving even - though it advertises itself as reduced-rate, so the discount is a config - knob (default 1.0 — no assumed discount) rather than a number invented - here. See ``routing.flex_cost_multiplier`` in config.yaml. - """ - cost = row.get("cost_per_1m_completion") - if cost is None: - return None - if row.get("latency_class") == "flex": - return cost * flex_cost_multiplier - return cost - - def rank_candidates( rows: Sequence[dict], *, weights: tuple[float, float, float], - flex_cost_multiplier: float = 1.0, ) -> list[dict]: """Score and sort candidates best-first. - Each row is expected to carry ``cost_per_1m_completion``, an optional - ``eco`` (mean energy per request, from ``energy_observations``), and an - optional ``proficiency`` (blended score for the task's category). Missing - eco/proficiency data yields the neutral 0.5 from ``scoring``, so a model + Each row carries an optional ``cost`` (mean USD actually billed for the + reference workload), an optional ``eco`` (mean gCO2eq for the same), and + an optional ``proficiency`` (blended score for the task's category). + Any of them missing yields the neutral 0.5 from ``scoring``, so a model is never penalized merely for being new to the table. + ``cost`` is the billed figure rather than the catalog's list price, which + NeuralWatt does not charge — see ``dispatcher.load_candidates``. That + also removes the need to guess a flex discount: a flex row's measured + cost already is its flex cost. + Returns a list of dicts: the original row plus ``composite`` and the three sub-scores, so a caller can log exactly why a model won. """ - costs = [effective_cost(r, flex_cost_multiplier) for r in rows] - ecos = [r.get("eco") for r in rows] - - cost_scores = cost_score(costs) - eco_scores = eco_score(ecos) + cost_scores = cost_score([r.get("cost") for r in rows]) + eco_scores = eco_score([r.get("eco") for r in rows]) ranked = [] for row, c_s, e_s in zip(rows, cost_scores, eco_scores): @@ -155,9 +140,7 @@ def rank_candidates( ranked.sort( key=lambda r: ( -r["composite"], - r["cost_per_1m_completion"] - if r["cost_per_1m_completion"] is not None - else float("inf"), + r["cost"] if r.get("cost") is not None else float("inf"), r["model_id"], ) ) diff --git a/scoring.py b/scoring.py index cb40f35..0210b1b 100644 --- a/scoring.py +++ b/scoring.py @@ -16,43 +16,37 @@ from __future__ import annotations from collections.abc import Sequence -def cost_score(costs: Sequence[float | None]) -> list[float]: - """Inverted min-max cost scores over a candidate set. +def normalize_inverted(values: Sequence[float | None]) -> list[float]: + """Inverted min-max over a candidate set: lower input = higher score. - score = (max_cost - cost) / (max_cost - min_cost); lower cost = higher - score. A $0-cheapest model legitimately scores 1.0 (free is cheapest); - paid models scale relative to max — this compression is intended, not a - bug (Metis finding #5). + score = (max - value) / (max - min). A cheapest/cleanest candidate scores + 1.0; the rest scale relative to max — this compression is intended. - None costs are excluded from the min/max computation and get a neutral - 0.5. If no candidate has a known cost, all get 0.5. If min == max - (all equal or a single candidate), present-data candidates get 1.0. + None values are excluded from the min/max computation and get a neutral + 0.5, so a model is never penalized merely for having no data yet. If no + candidate has data, all get 0.5. If min == max (all equal, or a single + candidate), present-data candidates get 1.0. """ - known = [c for c in costs if c is not None] + known = [v for v in values if v is not None] if not known: - return [0.5 for _ in costs] + return [0.5 for _ in values] lo, hi = min(known), max(known) if lo == hi: - return [1.0 if c is not None else 0.5 for c in costs] + return [1.0 if v is not None else 0.5 for v in values] span = hi - lo - return [0.5 if c is None else (hi - c) / span for c in costs] + return [0.5 if v is None else (hi - v) / span for v in values] -def eco_score(eco_values: Sequence[float | None]) -> list[float]: - """Inverted min-max eco scores over candidates that have eco data. - - Lower eco (carbon/energy) = higher score. Candidates without eco data - (None) get a neutral 0.5. If no candidate has eco data, all get 0.5. - If min == max among present-data candidates, they get 1.0. - """ - known = [e for e in eco_values if e is not None] - if not known: - return [0.5 for _ in eco_values] - lo, hi = min(known), max(known) - if lo == hi: - return [1.0 if e is not None else 0.5 for e in eco_values] - span = hi - lo - return [0.5 if e is None else (hi - e) / span for e in eco_values] +# Cost and eco normalize identically; they differ only in what is fed to +# them. Both are kept as named axes because they rank models DIFFERENTLY: +# cost tracks energy (NeuralWatt bills per kWh) while carbon is energy times +# the serving region's grid intensity, and that intensity spans 37 gCO2/kWh +# (FI) to 505 (US-MIDA-PJM) across the catalog. glm-5.2-fast is the second +# cheapest model and only the sixth cleanest; kimi-k3-flex draws 3.7x less +# energy than kimi-k2.7-code while emitting 3.6x more carbon. Collapsing +# these into one axis would silently pick a side. +cost_score = normalize_inverted +eco_score = normalize_inverted def proficiency_score(blended: float | None) -> float: diff --git a/seed_energy.py b/seed_energy.py new file mode 100644 index 0000000..9972555 --- /dev/null +++ b/seed_energy.py @@ -0,0 +1,201 @@ +#!/usr/bin/env python3 +"""Seed `energy_observations` by running a fixed reference task per model. + +Why this exists: `eco_score` returns the neutral 0.5 for every candidate +until a model has observations, and observations only accrue from real +traffic. That is a cold start the router cannot route its way out of — every +model looks identical on eco, so eco contributes nothing to the very +decisions that would generate the data. This sweep breaks the cycle. + +Method: one fixed prompt, one fixed `max_tokens`, `temperature=0`, run N +times per model. The prompt is held constant so the resulting energy figures +are comparable *across* models rather than reflecting who happened to be +asked a harder question. N > 1 because energy for identical token counts was +observed to vary up to 4x run to run — a single sample is not an estimate. + +What lands in the table are real observations of real calls, identical in +kind to what the dispatcher logs; they are simply generated deliberately +rather than incidentally. They carry `task_category='seed_reference'` so +they can be identified or purged later. + +Usage: + python seed_energy.py # 5 samples of every routable model + python seed_energy.py --samples 3 + python seed_energy.py --models kimi-k3,gemma-4-31b + python seed_energy.py --dry-run # show the plan and cost, call nothing +""" + +from __future__ import annotations + +import argparse +import os +import sqlite3 +import statistics +import sys +import time + +import requests + +from config import load_config +from dispatcher import extract_telemetry, log_observation + +# Held constant across models so the energy numbers are comparable. Long +# enough that most models run to the token cap rather than stopping early +# (a short answer burns less energy and would look misleadingly efficient), +# and generic enough that no model is advantaged by domain fit. +REFERENCE_PROMPT = ( + "Explain how a B-tree works, including its structure, how lookups " + "descend the tree, and how splits keep it balanced on insert." +) +REFERENCE_MAX_TOKENS = 400 +SEED_CATEGORY = "seed_reference" + + +def routable_models(conn: sqlite3.Connection, allowed_levels: list[str]) -> list[dict]: + placeholders = ",".join("?" * len(allowed_levels)) + rows = conn.execute( + f""" + SELECT model_id, provider, latency_class, tier, cost_per_1m_completion + FROM models + WHERE access_level IN ({placeholders}) + AND availability = 'active' + ORDER BY model_id + """, + tuple(allowed_levels), + ).fetchall() + return [dict(r) for r in rows] + + +def sample_once(base_url: str, api_key: str, model_id: str, timeout: int = 300) -> dict: + resp = requests.post( + f"{base_url}/chat/completions", + headers={"authorization": f"Bearer {api_key}"}, + json={ + "model": model_id, + "messages": [{"role": "user", "content": REFERENCE_PROMPT}], + "max_tokens": REFERENCE_MAX_TOKENS, + "temperature": 0, + }, + timeout=timeout, + ) + resp.raise_for_status() + return resp.json() + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--samples", type=int, default=5, help="samples per model (default 5)") + ap.add_argument("--models", help="comma-separated model_ids; default is all routable") + ap.add_argument("--max-tokens", type=int, default=REFERENCE_MAX_TOKENS) + ap.add_argument("--dry-run", action="store_true", help="print the plan, call nothing") + args = ap.parse_args() + + cfg = load_config("config.yaml") + conn = sqlite3.connect(cfg.database.path) + conn.row_factory = sqlite3.Row + models = routable_models(conn, cfg.routing.allowed_access_levels) + conn.close() + + if args.models: + wanted = {m.strip() for m in args.models.split(",")} + models = [m for m in models if m["model_id"] in wanted] + missing = wanted - {m["model_id"] for m in models} + if missing: + print(f"not routable / unknown: {', '.join(sorted(missing))}", file=sys.stderr) + + if not models: + print("no models to sweep", file=sys.stderr) + return 1 + + total_calls = len(models) * args.samples + print(f"{len(models)} models x {args.samples} samples = {total_calls} calls") + print(f"prompt: {REFERENCE_PROMPT[:60]}... (max_tokens={args.max_tokens}, temperature=0)") + if args.dry_run: + for m in models: + print(f" {m['model_id']:26s} tier {m['tier']} {m['latency_class']}") + return 0 + + settings = cfg.dispatch_providers["neuralwatt"] + api_key = os.environ.get(settings.api_key_env) + if not api_key: + print(f"{settings.api_key_env} is not set", file=sys.stderr) + return 1 + + results: dict[str, list[dict]] = {} + allowance_start = allowance_end = None + + for m in models: + model_id = m["model_id"] + results[model_id] = [] + for i in range(args.samples): + try: + payload = sample_once(settings.base_url, api_key, model_id) + except requests.RequestException as e: + print(f" {model_id:26s} sample {i + 1}: FAILED {type(e).__name__}: {e}") + continue + + usage = payload.get("usage") or {} + telemetry = extract_telemetry(payload) + log_observation( + model_id, + m["provider"], + SEED_CATEGORY, + usage.get("prompt_tokens"), + usage.get("completion_tokens"), + telemetry, + ) + if telemetry.allowance_remaining_usd is not None: + if allowance_start is None: + allowance_start = telemetry.allowance_remaining_usd + allowance_end = telemetry.allowance_remaining_usd + + results[model_id].append( + { + "completion_tokens": usage.get("completion_tokens"), + "energy_kwh": telemetry.energy_kwh, + "carbon": telemetry.carbon_g_co2eq, + "cost": telemetry.cost_usd, + } + ) + # Be a considerate neighbour on a shared endpoint. + time.sleep(0.3) + done = results[model_id] + if done: + mean_kwh = statistics.fmean(d["energy_kwh"] for d in done if d["energy_kwh"]) + print(f" {model_id:26s} {len(done)}/{args.samples} ok mean {mean_kwh:.3e} kWh") + + # --- summary --------------------------------------------------------- + print() + print( + f"{'model':26s}{'n':>3}{'tokens':>8}{'kWh mean':>12}{'kWh spread':>12}" + f"{'gCO2eq':>11}{'$ mean':>11}{'$/kWh':>8}" + ) + for model_id, rows in results.items(): + rows = [r for r in rows if r["energy_kwh"]] + if not rows: + print(f"{model_id:26s} 0 (no successful samples)") + continue + kwhs = [r["energy_kwh"] for r in rows] + toks = [r["completion_tokens"] or 0 for r in rows] + costs = [r["cost"] for r in rows if r["cost"] is not None] + carbons = [r["carbon"] for r in rows if r["carbon"] is not None] + spread = max(kwhs) / min(kwhs) if min(kwhs) else float("nan") + rate = (statistics.fmean(costs) / statistics.fmean(kwhs)) if costs else float("nan") + print( + f"{model_id:26s}{len(rows):>3}{statistics.fmean(toks):>8.0f}" + f"{statistics.fmean(kwhs):>12.3e}{spread:>11.1f}x" + f"{statistics.fmean(carbons) if carbons else 0:>11.2e}" + f"{statistics.fmean(costs) if costs else 0:>11.2e}{rate:>8.2f}" + ) + + if allowance_start is not None and allowance_end is not None: + print() + print( + f"allowance: {allowance_start:.6f} -> {allowance_end:.6f} USD " + f"(spent {allowance_start - allowance_end:.6f})" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_routing.py b/tests/test_routing.py index 2a63dc0..92d0472 100644 --- a/tests/test_routing.py +++ b/tests/test_routing.py @@ -6,12 +6,7 @@ proving it rejects and a case proving it does not over-reject. import pytest -from routing import ( - effective_cost, - is_eligible, - rank_candidates, - select_candidates, -) +from routing import is_eligible, rank_candidates, select_candidates WEIGHTS = (0.4, 0.2, 0.4) @@ -22,7 +17,7 @@ def _row(**overrides) -> dict: "model_id": "m", "provider": "neuralwatt", "tier": 2, - "cost_per_1m_completion": 1.0, + "cost": 1.0, "effective_context_window": 100_000, "availability": "active", "deprecated": 0, @@ -148,28 +143,21 @@ def test_select_candidates_filters_and_preserves_order(): assert [r["model_id"] for r in selected] == ["keep-1", "keep-2"] -# --- flex pricing --------------------------------------------------------- +# --- measured cost -------------------------------------------------------- -def test_flex_cost_is_unchanged_at_the_default_multiplier(): - # The catalog lists flex at the same sticker price, so the default asserts - # no discount rather than inventing one - assert effective_cost(_row(latency_class="flex"), 1.0) == 1.0 - - -def test_flex_cost_multiplier_discounts_only_flex_rows(): - assert effective_cost(_row(latency_class="flex"), 0.5) == 0.5 - assert effective_cost(_row(latency_class="standard"), 0.5) == 1.0 - - -def test_effective_cost_of_unknown_price_is_none(): - assert effective_cost(_row(cost_per_1m_completion=None), 0.5) is None +def test_unknown_cost_is_neutral_not_worst(): + # Given: a model never covered by the reference sweep. It must not be + # ranked last for lacking data, or a newly listed model could never be + # picked and so could never acquire data. + ranked = rank_candidates([_row(model_id="unswept", cost=None)], weights=WEIGHTS) + assert ranked[0]["cost_score"] == 0.5 # --- ranking -------------------------------------------------------------- def test_cheaper_model_wins_when_only_cost_differs(): - rows = [_row(model_id="pricey", cost_per_1m_completion=15.0), - _row(model_id="cheap", cost_per_1m_completion=0.28)] + rows = [_row(model_id="pricey", cost=15.0), + _row(model_id="cheap", cost=0.28)] ranked = rank_candidates(rows, weights=WEIGHTS) assert ranked[0]["model_id"] == "cheap" assert ranked[0]["composite"] > ranked[1]["composite"] @@ -183,8 +171,8 @@ def test_opposed_full_spreads_tie_at_equal_weight(): # normalization always produces a full spread, which makes the default # weights much tie-ier than they look. rows = [ - _row(model_id="cheap-bad", cost_per_1m_completion=0.1, proficiency=0.0), - _row(model_id="dear-good", cost_per_1m_completion=0.2, proficiency=1.0), + _row(model_id="cheap-bad", cost=0.1, proficiency=0.0), + _row(model_id="dear-good", cost=0.2, proficiency=1.0), ] ranked = rank_candidates(rows, weights=WEIGHTS) assert ranked[0]["composite"] == ranked[1]["composite"] @@ -193,8 +181,8 @@ def test_opposed_full_spreads_tie_at_equal_weight(): def test_proficiency_wins_when_weighted_above_cost(): rows = [ - _row(model_id="cheap-bad", cost_per_1m_completion=0.1, proficiency=0.0), - _row(model_id="dear-good", cost_per_1m_completion=0.2, proficiency=1.0), + _row(model_id="cheap-bad", cost=0.1, proficiency=0.0), + _row(model_id="dear-good", cost=0.2, proficiency=1.0), ] ranked = rank_candidates(rows, weights=(0.2, 0.2, 0.6)) assert ranked[0]["model_id"] == "dear-good" @@ -226,3 +214,28 @@ def test_ranking_reports_the_sub_scores_that_produced_the_winner(): def test_empty_candidate_set_ranks_to_empty(): assert rank_candidates([], weights=WEIGHTS) == [] + + +def test_cost_and_eco_can_rank_models_oppositely(): + # Given: the real catalog shape. NeuralWatt bills per kWh, so cost tracks + # energy — but carbon is energy x the serving region's grid intensity, and + # that spans 37 gCO2/kWh (FI) to 505 (US-MIDA-PJM). glm-5.2-fast draws + # little energy on a dirty grid; kimi-k2.7-code draws a lot on a clean one. + rows = [ + _row(model_id="glm-5.2-fast", cost=6.72e-05, eco=3.973e-03), + _row(model_id="kimi-k2.7-code", cost=1.18e-03, eco=5.459e-03), + ] + # Then: whichever axis carries the weight decides, and they disagree — + # which is why these must stay two axes rather than being collapsed. + by_cost = rank_candidates(rows, weights=(1.0, 0.0, 0.0)) + by_eco = rank_candidates(rows, weights=(0.0, 1.0, 0.0)) + assert by_cost[0]["model_id"] == "glm-5.2-fast" + assert by_eco[0]["model_id"] == "glm-5.2-fast" + + # And with a genuinely inverted pair the orders flip outright. + rows = [ + _row(model_id="low-cost-dirty", cost=1.0, eco=100.0), + _row(model_id="high-cost-clean", cost=10.0, eco=1.0), + ] + assert rank_candidates(rows, weights=(1.0, 0.0, 0.0))[0]["model_id"] == "low-cost-dirty" + assert rank_candidates(rows, weights=(0.0, 1.0, 0.0))[0]["model_id"] == "high-cost-clean" -- 2.49.1 From b3127d0eb0a2f0ecbc898c96080f08c0f3e90525 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Wed, 12 Aug 2026 01:05:01 -0400 Subject: [PATCH 03/20] fix: score on attributed cost and carbon, and drop fabricated grid data Two corrections, both surfaced by a routing decision that looked wrong to a human before it looked wrong in the data. 1. Carbon reported as carbon_source='static_fallback' is excluded from eco scoring. NeuralWatt substitutes a constant (475.0 gCO2/kWh, roughly a global average) when it cannot resolve live grid data, while still echoing the original grid_id -- so glm-5.2-fast appeared to sit on a grid 13.6x dirtier than its neighbours on the strength of a placeholder. Affected models now carry no eco figure and score the neutral 0.5, which is honest; ranking them against an invented number was not. 2. Cost and eco use the median rather than the mean. Distributions are right-skewed: deepseek-v4-flash sampled 7.7e-07 .. 1.0e-04, one spike 33x the median, enough to rank it at roughly twice gemma-4-31b's cost when by median it is a fraction. Also records avg_power_watts, duration_seconds and attribution_ratio, which made a third question answerable. Billed energy decomposes exactly as power x duration x attribution_ratio, and that ratio looks like noise up close -- eight rapid identical calls spanned 20x, correlating +0.997 with it. Scoring on the pre-attribution product was implemented and then reverted: the median ratio spans 750x BETWEEN models against ~1.8x within one, and the values are quantized (0.001, 0.25, 0.5, 0.75). That is serving concurrency, a stable per-model property and real money -- deepseek-v4-flash bills ~1000x under its share of pool gross. Normalizing it away would discard a 750x signal to suppress a 1.8x one. gross_energy_kwh survives as a diagnostic. Split-half validation on a 7-sample sweep: 10 of 13 models agree within 1.4x. kimi-k2.7-code-fast (29x), kimi-k3 (14x) and glm-5.2-flex (2.2x) do not, and are flagged as needing more samples. Effect: tier 1 now routes to deepseek-v4-flash, which is both the cheapest (9.0e-06, 5x under the next) and the cleanest (4.7e-05, 5x under). Category still cannot influence a decision -- proficiency remains the only category-dependent term and is still empty. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 61 +++++++++++++-- dispatcher.py | 128 ++++++++++++++++++++++++------ schema.sql | 23 +++++- seed_energy.py | 64 ++++++++++----- tests/test_load_candidates.py | 141 ++++++++++++++++++++++++++++++++++ 5 files changed, 364 insertions(+), 53 deletions(-) create mode 100644 tests/test_load_candidates.py diff --git a/CLAUDE.md b/CLAUDE.md index 4d571f0..65239ee 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -78,12 +78,53 @@ model and only the 6th cleanest; `kimi-k3-flex` draws 3.7x *less* energy than `kimi-k2.7-code` while emitting 3.6x *more* carbon. Weighting them separately is load-bearing, and `tests/test_routing.py` pins it. -**Resolved.** `cost` and `eco` now come from mean *measured* USD and gCO2eq +**Resolved.** `cost` and `eco` come from the *median* measured USD and gCO2eq over the reference sweep, so no billing formula lives in the code at all — the rule above is documentation for why list price isn't used, not logic. `flex_cost_multiplier` is gone: a flex row's measured cost already is its flex cost. +## Energy attribution: signal that looks like noise + +Billed energy decomposes exactly: + +``` +energy_kwh = avg_power_watts x duration_seconds x attribution_ratio +``` + +`attribution_ratio` is the request's share of a shared multi-tenant GPU pool. +Up close it looks like pure noise — eight rapid identical calls to one model +spanned 20x in billed energy, correlating **+0.997** with the ratio while +power and duration held steady. Two sweeps of the same 13 models with the +same prompt disagreed by up to 36x. + +Scoring on the pre-attribution product (`power x duration`) was tried, and it +is **wrong**. Across the sweep: + +| | spread | +|---|---| +| median attribution, **between** models | **750x** | +| typical spread **within** one model | **1.8x** | + +The ratios are quantized (0.001, 0.25, 0.5, 0.75) — that is serving +concurrency, a stable per-model property, not weather. A model whose GPUs +carry far more concurrent requests genuinely costs less per request, and +that is most of the real cost difference in the catalog: `deepseek-v4-flash` +bills ~1000x under its share of pool gross. Stripping attribution discards a +750x real signal to suppress a 1.8x one. + +So scoring reads the attributed figures, and the **median** absorbs what +noise remains. A split-half check on the 7-sample sweep (median of first +three vs last four) shows that working: + +- **10 of 13 models agree within 1.4x** — stable enough to route on +- **3 do not**: `kimi-k2.7-code-fast` (29x), `kimi-k3` (14x), + `glm-5.2-flex` (2.2x). Those need more samples before their position is + trustworthy. + +`dispatcher.gross_energy_kwh` remains as a diagnostic on the identity, not a +scoring input. + ## What's built and working - `schema.sql` — `models`, `proficiency`, `energy_observations`. Applies @@ -188,10 +229,11 @@ that request was two orders of magnitude low. Until this lands, the ~10s classifier round-trip is buying only a tier number — the category half of its output is discarded in effect. -2. **Sampling depth.** 5 samples/model was enough to separate models (they - span 32x in energy) but within-model spread reached 133x on - `deepseek-v4-flash`. Means are usable for ranking; they are not tight. More - samples, or a trimmed mean, would firm this up. +2. **Sampling depth for three models.** 7 samples/model gives split-half + agreement within 1.4x for 10 of 13, but `kimi-k2.7-code-fast` (29x), + `kimi-k3` (14x) and `glm-5.2-flex` (2.2x) are still unsettled. Re-run + `seed_energy.py --models ... --samples 21` for those before trusting where + they land. 3. **Escalation feedback loop** — low-confidence tier bumping works (`apply_escalation`), but there's no way for a human or downstream agent to @@ -205,8 +247,13 @@ that request was two orders of magnitude low. on `FI` reports 37. Either the region label or the intensity is wrong for those rows; worth confirming with NeuralWatt before trusting eco for the GLM family specifically. -- Within-model energy spread reached 133x. How many samples before a per-model - mean is tight enough to weight heavily rather than just order by? +- Three models still fail a split-half stability check at 7 samples. Is the + instability real (variable serving conditions) or an artifact of when the + sweep ran? Re-sweeping at a different hour would tell. +- The tier-1 composites sit within 0.009 of each other across a 40x cost + range, because min-max compresses once one candidate is far cheaper than + the rest. Ranking is right, but margins are thin — proficiency data will + swing these easily, which is the intent. - How much context-assembly (RAG-style retrieval) belongs in the classifier step vs. a separate pre-step? Leaning decoupled, undecided. - Self-run eval set: minimum viable task set to start proficiency scoring diff --git a/dispatcher.py b/dispatcher.py index 72715d3..d048ca0 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -38,6 +38,7 @@ import json import os import sqlite3 from datetime import datetime, timezone +from statistics import median from typing import Any, Literal, Optional import requests @@ -60,6 +61,30 @@ ROUTER_MODEL_BATCH = "auto:batch" # too much with request shape to compare models by. SEED_CATEGORY = "seed_reference" +# NeuralWatt reports this when it cannot resolve live grid data and falls back +# to a constant (475.0 gCO2/kWh) while still echoing the original grid_id. The +# figure is a placeholder, not a measurement, so it must not steer eco scoring. +FALLBACK_CARBON_SOURCE = "static_fallback" + +# Measured: cost_usd / energy_kwh came back 8.00 across every model. Kept for +# reporting and for reasoning about bills; scoring reads billed cost directly +# and so does not need it. +USD_PER_KWH = 8.00 + + +def gross_energy_kwh(avg_power_watts: float, duration_seconds: float) -> float: + """Pool energy over a request, BEFORE multi-tenant attribution. + + Diagnostic only — scoring does NOT use this. It exists to make the + decomposition inspectable: billed ``energy_kwh`` equals this times + ``attribution_ratio``, so comparing the two shows how much of a model's + cost comes from serving concurrency rather than from the work itself. + Ranking on this figure was tried and is wrong: it strips a term that + varies 750x between models (real, and real money) to suppress one that + varies about 1.8x within a model (noise). + """ + return avg_power_watts * duration_seconds / 3_600_000 + # Rough chars-per-token. Only used as a FLOOR on the classifier's context # estimate: a real coding session sends the whole conversation, and the # classifier — which sees that conversation and is asked to estimate its own @@ -115,9 +140,9 @@ class Candidate(BaseModel): reasoning_mode: str context_variant: str effective_context_window: int - # What the scoring actually read: mean USD billed and mean gCO2eq for the - # reference workload. `list_price_per_1m` is shown alongside because it is - # NOT what gets billed and the gap between them is worth being able to see. + # What the scoring actually read: median USD actually billed and median + # gCO2eq over the reference workload. `list_price_per_1m` rides along + # because it is NOT what gets billed, and the gap is worth seeing. cost: Optional[float] = None eco: Optional[float] = None list_price_per_1m: Optional[float] = None @@ -266,35 +291,75 @@ def load_candidates(conn: sqlite3.Connection, category: str) -> list[dict]: happened to be asked. Organic observations stay in the table for cost accounting; they just don't steer routing. - Models with no reference samples come back None on both and take the - neutral 0.5 from ``scoring``, so a newly listed model is never penalized - for not having been swept yet. + Both read the provider's own attributed figures — the billed + ``cost_usd`` and the reported ``carbon_g_co2eq`` — deliberately. + + That attribution (``energy_kwh = avg_power_watts x duration_seconds x + attribution_ratio``, where the ratio is the request's share of a shared + GPU pool) looks like noise up close: eight rapid identical calls varied + 20x, correlating +0.997 with the ratio. It is not. Across the reference + sweep the median ratio spans **750x between models** while the typical + within-model spread is **1.8x** — it is a stable per-model property, and + the values are quantized (0.001, 0.25, 0.5, 0.75), which is what serving + concurrency looks like. A model sharing its GPUs with far more concurrent + requests genuinely costs less per request, and that is most of the real + cost difference in the catalog. Scoring on pre-attribution energy was + tried and discards a 750x signal to suppress a 1.8x one. + + The MEDIAN, not the mean, handles what noise remains: a few models + (kimi-k3, kimi-k3-fast, qwen3.6-35b-fast) still show ~50-90x within-model + outliers, and with 7 samples the median ignores up to three of them. + + Carbon is dropped where ``carbon_source`` is ``static_fallback``. That is + a constant NeuralWatt substitutes when it cannot resolve live grid data, + so scoring on it would rank a model against a placeholder; without it the + model simply has no eco data and scores the neutral 0.5. + + Models with no usable reference samples come back None and take that + same neutral 0.5, so a newly listed model is never penalized for not + having been swept yet. """ rows = conn.execute( """ - SELECT m.*, - p.blended_score AS proficiency, - s.mean_cost AS cost, - s.mean_carbon AS eco + SELECT m.*, p.blended_score AS proficiency FROM models m LEFT JOIN proficiency p ON p.model_id = m.model_id AND p.provider = m.provider AND p.category = ? - LEFT JOIN ( - SELECT model_id, provider, - AVG(cost_usd) AS mean_cost, - AVG(carbon_g_co2eq) AS mean_carbon - FROM energy_observations - WHERE task_category = ? - GROUP BY model_id, provider - ) s - ON s.model_id = m.model_id - AND s.provider = m.provider """, - (category, SEED_CATEGORY), + (category,), ).fetchall() - return [dict(r) for r in rows] + + # Medians are computed here rather than in SQL — SQLite has no MEDIAN, and + # the catalog is 19 rows, so clarity beats a window-function expression. + costs: dict[tuple, list[float]] = {} + carbons: dict[tuple, list[float]] = {} + for o in conn.execute( + """ + SELECT model_id, provider, cost_usd, carbon_g_co2eq, carbon_source + FROM energy_observations WHERE task_category = ? + """, + (SEED_CATEGORY,), + ): + key = (o["model_id"], o["provider"]) + if o["cost_usd"] is not None: + costs.setdefault(key, []).append(o["cost_usd"]) + if ( + o["carbon_g_co2eq"] is not None + and o["carbon_source"] != FALLBACK_CARBON_SOURCE + ): + carbons.setdefault(key, []).append(o["carbon_g_co2eq"]) + + out = [] + for r in rows: + row = dict(r) + key = (row["model_id"], row["provider"]) + row["cost"] = median(costs[key]) if key in costs else None + row["eco"] = median(carbons[key]) if key in carbons else None + row["samples"] = len(costs.get(key, [])) + out.append(row) + return out def _to_candidate(row: dict) -> Candidate: @@ -363,9 +428,13 @@ class Telemetry(BaseModel): energy_kwh: Optional[float] = None energy_btu: Optional[float] = None + avg_power_watts: Optional[float] = None + duration_seconds: Optional[float] = None + attribution_ratio: Optional[float] = None carbon_g_co2eq: Optional[float] = None grid_carbon_intensity: Optional[float] = None grid_id: Optional[str] = None + carbon_source: Optional[str] = None cost_usd: Optional[float] = None allowance_remaining_usd: Optional[float] = None service_tier: Optional[str] = None @@ -387,9 +456,13 @@ def extract_telemetry(payload: dict) -> Telemetry: return Telemetry( energy_kwh=kwh, energy_btu=kwh * BTU_PER_KWH if kwh is not None else None, + avg_power_watts=energy.get("avg_power_watts"), + duration_seconds=energy.get("duration_seconds"), + attribution_ratio=energy.get("attribution_ratio"), carbon_g_co2eq=energy.get("carbon_g_co2eq"), grid_carbon_intensity=energy.get("grid_carbon_intensity_gco2perkwhr"), grid_id=energy.get("grid_id"), + carbon_source=energy.get("carbon_source"), # The provider's billed figure, not a tokens x list-price estimate. # Flex rows bill under their standard sibling despite identical # advertised pricing, so the estimate is wrong for every flex call. @@ -415,9 +488,10 @@ def log_observation( """ INSERT INTO energy_observations ( model_id, provider, task_category, prompt_tokens, completion_tokens, - energy_kwh, energy_btu, carbon_g_co2eq, grid_carbon_intensity, grid_id, - cost_usd, allowance_remaining_usd, service_tier, observed_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + energy_kwh, energy_btu, avg_power_watts, duration_seconds, + attribution_ratio, carbon_g_co2eq, grid_carbon_intensity, grid_id, + carbon_source, cost_usd, allowance_remaining_usd, service_tier, observed_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( model_id, @@ -427,9 +501,13 @@ def log_observation( completion_tokens, telemetry.energy_kwh, telemetry.energy_btu, + telemetry.avg_power_watts, + telemetry.duration_seconds, + telemetry.attribution_ratio, telemetry.carbon_g_co2eq, telemetry.grid_carbon_intensity, telemetry.grid_id, + telemetry.carbon_source, telemetry.cost_usd, telemetry.allowance_remaining_usd, telemetry.service_tier, diff --git a/schema.sql b/schema.sql index 54b77ef..d59853f 100644 --- a/schema.sql +++ b/schema.sql @@ -76,9 +76,24 @@ CREATE TABLE IF NOT EXISTS energy_observations ( task_category TEXT, prompt_tokens INTEGER, completion_tokens INTEGER, - energy_kwh REAL, -- response.energy.energy_kwh + -- energy_kwh is what NeuralWatt BILLS, but it is not a measure of model + -- efficiency: it equals avg_power_watts * duration_seconds * + -- attribution_ratio, where attribution_ratio is this request's share of a + -- shared multi-tenant GPU pool. Eight identical calls to one model inside + -- one minute varied 20x in energy_kwh and 22x in attribution_ratio, + -- correlation +0.997, while duration and power held steady. Ranking models + -- on this ranks how busy the provider was, not how efficient the model is. + energy_kwh REAL, -- response.energy.energy_kwh (attributed, noisy) energy_btu REAL, -- energy_kwh * 3412.14, purely for comedic dashboard value + -- The pre-attribution terms. avg_power_watts * duration_seconds is the + -- pool's energy over the request, independent of how many other tenants + -- shared it, and it is ~10x more stable (1.85x spread vs 20x on the same + -- eight calls). That product is what scoring ranks on. + avg_power_watts REAL, -- response.energy.avg_power_watts + duration_seconds REAL, -- response.energy.duration_seconds + attribution_ratio REAL, -- kept so the noise term stays inspectable + -- Carbon is what design doc §4 actually scores eco on, and NeuralWatt -- reports it per-request rather than making us derive it. Grid intensity -- and region are stored alongside because the same energy in a different @@ -88,6 +103,12 @@ CREATE TABLE IF NOT EXISTS energy_observations ( carbon_g_co2eq REAL, -- response.energy.carbon_g_co2eq grid_carbon_intensity REAL, -- gCO2/kWh at call time grid_id TEXT, -- e.g. 'FI' + -- How the carbon figure was obtained. 'static_fallback' means NeuralWatt + -- could not resolve live grid data and substituted a constant (475.0, + -- about a global average) while still reporting the original grid_id -- + -- so the number is a placeholder, not a measurement. Routing on it would + -- penalize a model against a made-up figure, so eco scoring excludes it. + carbon_source TEXT, -- 'agent_cache' | 'static_fallback' | ... -- The provider's own billed figure (response.cost.request_cost_usd), NOT -- a tokens x list-price estimate. These disagree: flex rows bill roughly diff --git a/seed_energy.py b/seed_energy.py index 9972555..e8b69c2 100644 --- a/seed_energy.py +++ b/seed_energy.py @@ -8,10 +8,21 @@ model looks identical on eco, so eco contributes nothing to the very decisions that would generate the data. This sweep breaks the cycle. Method: one fixed prompt, one fixed `max_tokens`, `temperature=0`, run N -times per model. The prompt is held constant so the resulting energy figures -are comparable *across* models rather than reflecting who happened to be -asked a harder question. N > 1 because energy for identical token counts was -observed to vary up to 4x run to run — a single sample is not an estimate. +times per model. The prompt is held constant so the resulting figures are +comparable *across* models rather than reflecting who happened to be asked a +harder question. + +N > 1 for a specific reason. The provider's billed `energy_kwh` is +`avg_power_watts * duration_seconds * attribution_ratio`, and that last term +is the request's share of a shared multi-tenant GPU pool — eight identical +calls to one model inside one minute spanned 20x, correlating +0.997 with +the attribution ratio while power and duration held steady. Two earlier +sweeps of these same 13 models disagreed by up to 36x on that basis. + +So routing scores on `power * duration` (see `dispatcher.gross_energy_kwh`), +which held to 1.85x across those same eight calls. The summary prints both +spreads side by side; if the billed column is wildly wider than the gross +column, that is the artifact, not the models. What lands in the table are real observations of real calls, identical in kind to what the dispatcher logs; they are simply generated deliberately @@ -37,7 +48,7 @@ import time import requests from config import load_config -from dispatcher import extract_telemetry, log_observation +from dispatcher import extract_telemetry, gross_energy_kwh, log_observation # Held constant across models so the energy numbers are comparable. Long # enough that most models run to the token cap rather than stopping early @@ -149,43 +160,56 @@ def main() -> int: allowance_start = telemetry.allowance_remaining_usd allowance_end = telemetry.allowance_remaining_usd + gross = None + if telemetry.avg_power_watts and telemetry.duration_seconds: + gross = gross_energy_kwh( + telemetry.avg_power_watts, telemetry.duration_seconds + ) results[model_id].append( { "completion_tokens": usage.get("completion_tokens"), "energy_kwh": telemetry.energy_kwh, + "gross_kwh": gross, "carbon": telemetry.carbon_g_co2eq, "cost": telemetry.cost_usd, } ) # Be a considerate neighbour on a shared endpoint. time.sleep(0.3) - done = results[model_id] + done = [d for d in results[model_id] if d["gross_kwh"]] if done: - mean_kwh = statistics.fmean(d["energy_kwh"] for d in done if d["energy_kwh"]) - print(f" {model_id:26s} {len(done)}/{args.samples} ok mean {mean_kwh:.3e} kWh") + med = statistics.median(d["gross_kwh"] for d in done) + print( + f" {model_id:26s} {len(done)}/{args.samples} ok " + f"median gross {med:.3e} kWh" + ) # --- summary --------------------------------------------------------- + # Two spread columns, because the whole point of this sweep is that they + # differ: the billed figure carries a multi-tenancy attribution term that + # the pre-attribution product does not. print() print( - f"{'model':26s}{'n':>3}{'tokens':>8}{'kWh mean':>12}{'kWh spread':>12}" - f"{'gCO2eq':>11}{'$ mean':>11}{'$/kWh':>8}" + f"{'model':26s}{'n':>3}{'gross kWh':>12}{'gross spr':>11}" + f"{'billed kWh':>12}{'billed spr':>12}{'gCO2eq':>11}" ) for model_id, rows in results.items(): - rows = [r for r in rows if r["energy_kwh"]] + rows = [r for r in rows if r["gross_kwh"]] if not rows: print(f"{model_id:26s} 0 (no successful samples)") continue - kwhs = [r["energy_kwh"] for r in rows] - toks = [r["completion_tokens"] or 0 for r in rows] - costs = [r["cost"] for r in rows if r["cost"] is not None] + gross = [r["gross_kwh"] for r in rows] + billed = [r["energy_kwh"] for r in rows if r["energy_kwh"]] carbons = [r["carbon"] for r in rows if r["carbon"] is not None] - spread = max(kwhs) / min(kwhs) if min(kwhs) else float("nan") - rate = (statistics.fmean(costs) / statistics.fmean(kwhs)) if costs else float("nan") + + def spread(xs): + return max(xs) / min(xs) if xs and min(xs) else float("nan") + print( - f"{model_id:26s}{len(rows):>3}{statistics.fmean(toks):>8.0f}" - f"{statistics.fmean(kwhs):>12.3e}{spread:>11.1f}x" - f"{statistics.fmean(carbons) if carbons else 0:>11.2e}" - f"{statistics.fmean(costs) if costs else 0:>11.2e}{rate:>8.2f}" + f"{model_id:26s}{len(rows):>3}{statistics.median(gross):>12.3e}" + f"{spread(gross):>10.1f}x{statistics.median(billed) if billed else 0:>12.3e}" + f"{spread(billed):>11.1f}x" + f"{statistics.median(carbons) if carbons else 0:>11.2e}" ) if allowance_start is not None and allowance_end is not None: diff --git a/tests/test_load_candidates.py b/tests/test_load_candidates.py new file mode 100644 index 0000000..555940d --- /dev/null +++ b/tests/test_load_candidates.py @@ -0,0 +1,141 @@ +"""Tests for dispatcher.load_candidates — how measured data reaches scoring. + +Every behaviour here was found by a wrong routing decision rather than by +reasoning, so each gets explicit coverage: + +- scoring reads the provider's ATTRIBUTED figures, because the attribution + ratio turned out to be a stable per-model property (750x between models, + ~1.8x within one) rather than the noise it resembles up close +- the median, not the mean, because a few models still throw 50-90x outliers +- carbon reported as a static fallback is not data and must not rank anything +""" + +import sqlite3 +from pathlib import Path + +import pytest + +from dispatcher import ( + FALLBACK_CARBON_SOURCE, + SEED_CATEGORY, + USD_PER_KWH, + gross_energy_kwh, + load_candidates, +) + +ROOT = Path(__file__).resolve().parent.parent +SCHEMA_SQL = (ROOT / "schema.sql").read_text() + +# 1000 W for 3.6 s = 1e-3 kWh, so the arithmetic below stays readable. +WATTS = 1000.0 +SECONDS = 3.6 +GROSS_KWH = 1e-3 + + +@pytest.fixture +def db(tmp_path): + conn = sqlite3.connect(tmp_path / "test.db") + conn.row_factory = sqlite3.Row + conn.executescript(SCHEMA_SQL) + conn.execute( + """ + INSERT INTO models (model_id, provider, tier, availability, last_updated) + VALUES ('m', 'neuralwatt', 2, 'active', '2026-08-12T00:00:00+00:00') + """ + ) + conn.commit() + yield conn + conn.close() + + +def _observe(conn, *, cost=1.0, carbon=37.0, source="agent_cache", category=SEED_CATEGORY): + conn.execute( + """ + INSERT INTO energy_observations ( + model_id, provider, task_category, cost_usd, + carbon_g_co2eq, carbon_source, observed_at + ) VALUES ('m', 'neuralwatt', ?, ?, ?, ?, '2026-08-12T00:00:00+00:00') + """, + (category, cost, carbon, source), + ) + conn.commit() + + +def _only(conn): + return load_candidates(conn, "coding_general")[0] + + +# --- attribution is signal, not noise ------------------------------------- + +def test_gross_energy_identity_holds(): + # The decomposition behind the finding: billed = power x duration x + # attribution. 1260.9 W x 1.523 s x 0.0158 = 8.43e-06 kWh, and the API + # reported 8.438e-06. gross_energy_kwh is a diagnostic on that identity. + assert gross_energy_kwh(1000.0, 3.6) == pytest.approx(1e-3) + assert gross_energy_kwh(1260.9, 1.523) * 0.0158 == pytest.approx(8.43e-06, rel=1e-2) + + +def test_scoring_reads_billed_cost_not_pre_attribution_energy(db): + # Given: a model whose billed cost is far below its share of pool gross, + # because many concurrent requests share its GPUs — deepseek-v4-flash + # bills ~1000x under gross. That discount is real money, so it must reach + # the score rather than being normalized away. + _observe(db, cost=1.154e-06) + _observe(db, cost=1.154e-06) + assert _only(db)["cost"] == pytest.approx(1.154e-06) + + +# --- median, not mean ----------------------------------------------------- + +def test_cost_uses_the_median_so_one_spike_cannot_dominate(db): + # Given: kimi-k3's real shape — ordinary samples plus a ~90x outlier + for c in (1.0, 1.0, 1.0, 1.0, 90.0): + _observe(db, cost=c) + assert _only(db)["cost"] == 1.0 + + +def test_eco_uses_the_median_too(db): + for carbon in (1.0, 2.0, 3.0, 4.0, 500.0): + _observe(db, cost=1.0, carbon=carbon) + assert _only(db)["eco"] == 3.0 + + +# --- fabricated carbon is not data ---------------------------------------- + +def test_static_fallback_carbon_is_excluded(db): + # Given: the glm-5.2-fast case. NeuralWatt could not resolve live grid + # data and substituted 475.0 while still reporting grid_id 'FI'. + for _ in range(3): + _observe(db, cost=1.0, carbon=8.5e-03, source=FALLBACK_CARBON_SOURCE) + row = _only(db) + # Then: no eco figure at all rather than a placeholder one. scoring turns + # that into a neutral 0.5, which is honest; ranking a model against a + # constant nobody measured would not be. + assert row["eco"] is None + # Cost is unaffected — it comes from the billing block, not the estimate. + assert row["cost"] == 1.0 + + +def test_measured_carbon_survives_alongside_fallback_rows(db): + _observe(db, cost=1.0, carbon=100.0, source=FALLBACK_CARBON_SOURCE) + _observe(db, cost=1.0, carbon=2.0) + _observe(db, cost=1.0, carbon=4.0) + assert _only(db)["eco"] == 3.0 + + +# --- scope ---------------------------------------------------------------- + +def test_only_reference_workload_observations_are_used(db): + # Given: organic traffic, whose energy tracks request shape far more than + # model efficiency (19x spread on one model), alongside the sweep + _observe(db, cost=1.0) + _observe(db, cost=1.0) + _observe(db, cost=999.0, category="coding_general") + assert _only(db)["cost"] == 1.0 + + +def test_unswept_model_has_no_measurements(db): + row = _only(db) + assert row["cost"] is None + assert row["eco"] is None + assert row["samples"] == 0 -- 2.49.1 From 7675c61e2dd92055186572da83abf502958dff8b Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Wed, 12 Aug 2026 01:05:17 -0400 Subject: [PATCH 04/20] docs: mark the GLM grid question answered Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 65239ee..cd7170d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -243,10 +243,11 @@ that request was two orders of magnitude low. - Answered: cost and eco stay separate axes — grid intensity spans 13.6x across the catalog, so they rank models differently. -- `glm-5.2-fast` reports `grid_id: FI` at 475 gCO2/kWh while everything else - on `FI` reports 37. Either the region label or the intensity is wrong for - those rows; worth confirming with NeuralWatt before trusting eco for the - GLM family specifically. +- Answered: the GLM rows reporting `grid_id: FI` at 475 gCO2/kWh were + `carbon_source: static_fallback` — a substituted constant, not a + measurement. They are now excluded from eco rather than trusted. Still + worth asking NeuralWatt why the fallback keeps the original `grid_id`, + since that is what made it look like a real regional difference. - Three models still fail a split-half stability check at 7 samples. Is the instability real (variable serving conditions) or an artifact of when the sweep ran? Re-sweeping at a different hour would tell. -- 2.49.1 From 3d264c8a8dda8c250cb49bd1e489c11fc1f33868 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Wed, 12 Aug 2026 01:51:04 -0400 Subject: [PATCH 05/20] fix: make the classifier path survive a slow or unhappy local model Three defects, all found by pointing opencode at the running service rather than by testing the classifier in isolation. 1. max_retries=0 on the classifier client. The OpenAI SDK retries twice by default, so classifier.timeout_seconds was silently a 3x wall-clock bound: a request hung past 250s against a 120s setting and logged nothing at all. 2. Cap the classifier's generation at 1024 tokens. qwen3.5 is a reasoning model and will otherwise emit an unbounded chain of thought. That does not fail in isolation, it cascades -- Ollama keeps generating after the client gives up AND serializes per model, so one runaway request queues every later request behind it. Observed as alternating 120s timeouts and 14s successes on an idle GPU with the model resident. 256 was tried first and was too tight: the thinking trace consumed the whole budget and the model was truncated before emitting any JSON, which surfaced as "Classifier returned non-JSON output: ''". Prompt-dependent, so /route passed six consecutive calls while /v1/chat/completions failed. 3. Classifier failure degrades instead of aborting. A timeout, transport error, or unparseable response now yields a configured fallback tier and category flagged source='fallback', rather than 502/503. The caller is a coding agent that would rather have a mid-tier answer than an error. Escalation skips fallbacks deliberately: bumping a low-confidence fallback would send every request to the frontier tier precisely when the local model is unavailable, which is the expensive failure mode. Effect: eight varied prompts through /v1/chat/completions all return 200 (4-25s, the spread being how much the model chooses to think), where before the same path cascaded into timeouts. opencode round-trips cleanly. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 28 +++++++++++++++----- config.py | 3 +++ config.yaml | 18 +++++++++++++ dispatcher.py | 72 +++++++++++++++++++++++++++++++++++---------------- 4 files changed, 92 insertions(+), 29 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index cd7170d..ce5633c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -195,13 +195,29 @@ single upstream token is requested. Measured on `qwen3.5:latest`: | | latency | |---|---| -| cold (model not resident) | **43s** — exceeded the old 30s ceiling and returned 503 | -| warm | **5–10s** | +| cold (model not resident) | **43s+** — a reload can exceed even the 120s ceiling | +| warm, simple prompt | **~4s** | +| warm, prompt that triggers a long thinking trace | **up to ~25s** | -`timeout_seconds` is now 120 so a cold load no longer 503s, and -`temperature` is 0 because at the default the same prompt classified tier 2 -then tier 1 on consecutive calls and routed to two different models — -routing that moves under an identical prompt can't be reasoned about. +Four settings keep this usable, each fixing a failure seen in practice: + +- **`max_retries=0` on the classifier client.** The OpenAI SDK retries twice + by default, so `timeout_seconds` silently became a 3x wall-clock bound — a + request hung past 250s on a 120s setting and logged nothing. +- **`max_output_tokens: 1024`.** qwen3.5 is a reasoning model and will + otherwise emit an unbounded chain of thought. That cascades: Ollama keeps + generating after the client gives up *and* serializes per model, so one + runaway request queues every later request behind it and the timeouts + spread. 256 was too tight — the trace consumed the budget and the model was + truncated before emitting any JSON, which surfaced as an empty response. +- **`fallback_tier` / `fallback_category`.** A classifier that times out, + errors, or returns garbage now degrades to a configured mid tier flagged + `source: "fallback"` instead of returning 502/503. The caller is a coding + agent that would rather have a mid-tier answer than an error. Escalation + deliberately skips fallbacks, so an unavailable local model does not + silently promote every request to the frontier tier. +- **`temperature: 0`.** At the default, the same prompt classified tier 2 + then tier 1 on consecutive calls and routed to two different models. But ~10s of local overhead on every message is a real tax for an interactive agent, where the upstream answer itself may take 2s. Unaddressed options: diff --git a/config.py b/config.py index 0598738..e1a4e7a 100644 --- a/config.py +++ b/config.py @@ -135,6 +135,9 @@ class ClassifierConfig(BaseModel): model: str timeout_seconds: int temperature: float = 0.0 + max_output_tokens: int = 1024 + fallback_tier: int = 2 + fallback_category: str = "general_chat" response_format: str system_prompt: str diff --git a/config.yaml b/config.yaml index 38149ec..3724616 100644 --- a/config.yaml +++ b/config.yaml @@ -96,6 +96,24 @@ classifier: # it to two different models. Routing that changes under an identical # prompt is untraceable. temperature: 0 + # Hard cap on the classifier's generation, to bound a failure mode that + # cascades: qwen3.5 is a reasoning model and will sometimes emit a long + # chain of thought, blowing past timeout_seconds — and Ollama keeps + # generating after the client gives up AND serializes per model, so one + # runaway request queues every later request behind it. + # + # Must leave room for the thinking trace, not just the ~45-token answer. At + # 256 the trace consumed the whole budget on some prompts and the model was + # truncated before emitting any JSON at all, which read as an empty + # response. 1024 bounds the runaway while leaving the answer reachable. + max_output_tokens: 1024 + + # Where routing lands when the classifier times out, errors, or returns + # something unparseable. A local model being slow should degrade routing, + # not refuse the request — the caller is a coding agent that would rather + # have a mid-tier answer than a 502. + fallback_tier: 2 + fallback_category: general_chat response_format: "json" # ask Ollama to constrain output to valid JSON # The dispatcher appends the authoritative category list from # proficiency.categories to this prompt at call time. Do not enumerate the diff --git a/dispatcher.py b/dispatcher.py index d048ca0..6c9426f 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -37,6 +37,7 @@ from __future__ import annotations import json import os import sqlite3 +import sys from datetime import datetime, timezone from statistics import median from typing import Any, Literal, Optional @@ -129,7 +130,7 @@ class Classification(BaseModel): required_context_tokens: int confidence: float escalated: bool = False - source: Literal["classifier", "override"] = "classifier" + source: Literal["classifier", "override", "fallback"] = "classifier" class Candidate(BaseModel): @@ -178,7 +179,16 @@ def _db() -> sqlite3.Connection: def _classifier_client() -> OpenAI: # Ollama ignores the key but the SDK requires one to be set. - return OpenAI(base_url=cfg.classifier.base_url, api_key="ollama") + # + # max_retries=0 matters: the SDK retries twice by default, so + # classifier.timeout_seconds silently becomes a 3x wall-clock bound and a + # cold model load can block a caller for six minutes on a 120s setting — + # observed as a request that hung past 250s and logged nothing. Retrying a + # local model that is busy loading only queues more work behind it, so the + # configured timeout should be the real one. + return OpenAI( + base_url=cfg.classifier.base_url, api_key="ollama", max_retries=0 + ) def _provider_client(provider: str) -> OpenAI: @@ -214,7 +224,27 @@ def classify(task: str, context: Optional[str]) -> Classification: client = _classifier_client() try: - resp = client.chat.completions.create( + resp = _classify_once(client, system_prompt, user_content) + except (OpenAIError, ValueError, KeyError, TypeError, json.JSONDecodeError) as e: + # A local model that is slow, restarting, or mid-thought must not take + # the caller down with it. Routing degrades to a configured mid tier, + # flagged as source='fallback' so the response says so. + print(f"classifier fallback ({type(e).__name__}: {e})", file=sys.stderr) + return Classification( + task_category=cfg.classifier.fallback_category, + task_tier=cfg.classifier.fallback_tier, + required_context_tokens=0, + confidence=0.0, + source="fallback", + ) + + return resp + + +def _classify_once(client: OpenAI, system_prompt: str, user_content: str) -> Classification: + """One classifier round-trip. Raises on anything unusable.""" + categories = cfg.proficiency.categories + resp = client.chat.completions.create( model=cfg.classifier.model, messages=[ {"role": "system", "content": system_prompt}, @@ -223,19 +253,13 @@ def classify(task: str, context: Optional[str]) -> Classification: response_format={"type": "json_object"} if cfg.classifier.response_format == "json" else None, - temperature=cfg.classifier.temperature, - timeout=cfg.classifier.timeout_seconds, - ) - except OpenAIError as e: - raise HTTPException( - 503, f"Classifier ({cfg.classifier.model} @ {cfg.classifier.base_url}) failed: {e}" - ) + temperature=cfg.classifier.temperature, + max_tokens=cfg.classifier.max_output_tokens, + timeout=cfg.classifier.timeout_seconds, + ) raw = resp.choices[0].message.content or "" - try: - parsed = json.loads(raw) - except json.JSONDecodeError: - raise HTTPException(502, f"Classifier returned non-JSON output: {raw[:300]!r}") + parsed = json.loads(raw) category = parsed.get("task_category") if category not in categories: @@ -243,21 +267,23 @@ def classify(task: str, context: Optional[str]) -> Classification: # proficiency row, so fall back rather than routing on a phantom join. category = "general_chat" if "general_chat" in categories else categories[0] - try: - return Classification( - task_category=category, - task_tier=int(parsed["task_tier"]), - required_context_tokens=int(parsed["required_context_tokens"]), - confidence=float(parsed["confidence"]), - ) - except (KeyError, TypeError, ValueError) as e: - raise HTTPException(502, f"Classifier JSON missing/invalid fields: {e}; got {raw[:300]!r}") + return Classification( + task_category=category, + task_tier=int(parsed["task_tier"]), + required_context_tokens=int(parsed["required_context_tokens"]), + confidence=float(parsed["confidence"]), + ) def apply_escalation(c: Classification) -> Classification: """Bump the tier when the classifier isn't confident in its own call.""" if not cfg.escalation.enabled: return c + # A fallback tier is a deliberate default, not a shaky guess — escalating + # it would silently send every request to the frontier tier whenever the + # local model is unavailable, which is the expensive failure mode. + if c.source == "fallback": + return c if c.confidence >= cfg.escalation.min_confidence_before_bump: return c bumped = min(c.task_tier + 1, cfg.escalation.max_tier) -- 2.49.1 From ef06e7d0d1bc3c8bb7dbcf90afb8da8957e3bf05 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Sun, 16 Aug 2026 16:49:25 -0400 Subject: [PATCH 06/20] feat: populate proficiency, so task_category finally changes routing proficiency_score is the only category-dependent term in the composite, so with the table empty the classifier's category output was computed, paid for at ~10s a request, and then discarded. Across 27 decisions (9 categories x 3 tiers) routing produced 2 distinct models under list-price scoring and 3 under measured cost/eco. It now produces 7, with four different models winning tier 1 depending on category. Adds: - proficiency.py / proficiency_store.py -- pure blending plus the single write path, so blended_score and source cannot drift from their inputs. Scores accumulate into a running mean rather than replacing, so re-running the harness tightens estimates instead of discarding history. - leaderboards.yaml / leaderboard.py -- curated per-family priors and their importer, for cold start: a newly listed NeuralWatt family has no self-eval history and would otherwise be indistinguishable from a model measured and found average. Ships EMPTY on purpose; inventing benchmark numbers would put fabricated data into routing, the same failure as the provider's static_fallback carbon constant this project already excludes. `leaderboard.py --check` names every family missing a prior. - evals/tasks.yaml / eval_proficiency.py -- 23 tasks over all 9 categories, scored objectively wherever the category admits it: code executed against checks, exact answers compared, tool calls inspected structurally. Only the four prose categories use a judge, and a judge never grades its own family. - base_model_id on models, so -flex rows inherit their family's scores rather than being re-measured: same weights, different queue. The blending rule needed a fallback the design doc did not specify. Read literally, a model with no leaderboard prior and 9 real samples scores nothing. Self-eval now carries it, labelled self_eval_thin so thin evidence stays distinguishable from evidence that cleared the threshold. Findings: coding does NOT discriminate this catalog -- all 13 rows score 1.00 on all three coding categories even after the tasks were hardened with touching intervals, present-but-falsy defaults, late-binding closures and a binary search that infinite-loops. What discriminates is tool use, arithmetic traps and prose. deepseek-v4-flash scores 1.00 on coding but 0.33 on tool_use_agentic: given a prompt containing both times it needed, it calls two tools instead of subtracting. The router now avoids it there while still choosing it for coding. Three harness defects were found and fixed along the way, each of which scored the rig rather than the model: a token budget shared between a reasoning trace and the answer (empty completions scored 0.00), a single leading space making valid code an IndentationError, and judge malfunctions recorded as model failures. tests/test_task_set.py now validates every task against a reference solution so a broken check cannot masquerade as difficulty -- it caught one on its first run. Tests 134 -> 153. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 97 ++++++- eval_proficiency.py | 503 +++++++++++++++++++++++++++++++++++ evals/tasks.yaml | 490 ++++++++++++++++++++++++++++++++++ leaderboard.py | 169 ++++++++++++ leaderboards.yaml | 41 +++ poller.py | 30 ++- proficiency.py | 103 +++++++ proficiency_store.py | 190 +++++++++++++ schema.sql | 5 + tests/test_eval_scoring.py | 209 +++++++++++++++ tests/test_poller_parsing.py | 31 ++- tests/test_proficiency.py | 100 +++++++ tests/test_task_set.py | 232 ++++++++++++++++ 13 files changed, 2184 insertions(+), 16 deletions(-) create mode 100644 eval_proficiency.py create mode 100644 evals/tasks.yaml create mode 100644 leaderboard.py create mode 100644 leaderboards.yaml create mode 100644 proficiency.py create mode 100644 proficiency_store.py create mode 100644 tests/test_eval_scoring.py create mode 100644 tests/test_proficiency.py create mode 100644 tests/test_task_set.py diff --git a/CLAUDE.md b/CLAUDE.md index ce5633c..a007c1b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -148,7 +148,19 @@ scoring input. - `tiering.py` / `tier.py` — pure tier resolver + the DB pass that applies it. - `routing.py` — pure hard filters and ranking. - `dispatcher.py` — FastAPI service. `GET /health`, `POST /route` (classify - and pick, no provider call), `POST /dispatch` (route, call, log). + and pick, no provider call), `POST /dispatch` (route, call, log), plus an + OpenAI-compatible `GET /v1/models` and `POST /v1/chat/completions`. +- `proficiency.py` / `proficiency_store.py` — pure blending arithmetic, and + the only path that writes the `proficiency` table (so `blended_score` and + `source` can never drift from the inputs that produced them). +- `leaderboards.yaml` / `leaderboard.py` — curated per-family priors and their + importer. **Ships empty by design**; `python leaderboard.py --check` names + every active family lacking a prior. +- `evals/tasks.yaml` / `eval_proficiency.py` — the self-eval task set and + runner. Four scoring kinds, objective wherever the category allows it: + `code` executes the model's Python against checks, `exact` compares a + normalized answer, `tool` inspects the tool call structurally, and only the + four prose categories fall back to a `judge`. - `tests/` — 74 tests, all passing. ### Serving class: one base model, many rows @@ -188,6 +200,70 @@ tier 3 and left tier 1 empty. A `-fast` row does not inherit its sibling's tier 3. Cost is checked before the reasoning rule so $0.28/1M models can reach tier 1. Current distribution: **4 / 6 / 9**. +## Proficiency: category now changes routing + +`proficiency_score` is the ONLY category-dependent term in the composite, so +until this table had data, `task_category` could not change a decision at +all — the classifier computed it, the router paid ~10s for it, and then it +made no difference. It does now: + +| scoring | distinct models across 27 decisions (9 categories x 3 tiers) | +|---|---| +| catalog list price | 2 | +| measured cost + eco | 3 | +| **+ proficiency** | **7** | + +Within tier 1 alone, four different models win depending on category. + +### What the task set actually found + +**Coding does not discriminate these models.** All 13 rows score exactly +1.00 on `coding_general`, `coding_refactor` and `debugging` — and that is +after the tasks were deliberately hardened with touching intervals, full +semver, present-but-falsy defaults, late-binding closures and a binary search +that infinite-loops. Every model in this catalog is simply good at that class +of problem, so cost and eco decide coding routes, which is the right outcome. + +**What does discriminate is tool use, arithmetic traps, and prose.** +`deepseek-v4-flash` scores 1.00 on all three coding categories yet **0.33 on +`tool_use_agentic`** and 0.67 on `reasoning_math`. Verified live, not an +artifact: given "It is 1:20pm and my meeting starts at 3pm, how many minutes +away?" — both times supplied — it calls *two* tools rather than subtracting. +It over-reaches for tools, which is exactly the failure mode that matters in +an agent loop. The router now avoids it for those categories while still +picking it for coding. + +Everything currently reads `source='self_eval_thin'`: real measurement, but +below `self_eval_min_samples` (2-3 tasks per category per run). Re-run +`eval_proficiency.py` to accumulate — scores fold into a running mean rather +than replacing, so samples add up across runs. + +### Harness bugs this shook out + +Three separate defects, each of which scored the rig rather than the model, +and each caught by reading per-task detail rather than the summary: + +- **Token budget.** `max_tokens` was shared between a reasoning model's trace + and its answer. At 1200, qwen3.6-35b spent ~4,200 characters thinking and + returned an EMPTY content field, scoring 0.00 on tasks it can plainly do. + Now 24000, clamped per model (gemma-4-31b caps at 16384), and + `finish_reason: length` skips the sample instead of scoring it. +- **One leading space.** kimi-k2.7-code returns `" def f(...)"`, which becomes + IndentationError once the harness prepends its imports — 0.00 across all + nine coding tasks for a model with "code" in its name. +- **Judge failures scored as model failures.** 44% of judge calls returned + unparseable output (the judge is itself a reasoning model and leaks its + thinking despite `response_format`). Each was recorded as 0.0. Now the JSON + is extracted from surrounding prose and an unusable reply yields no sample. + +`tests/test_task_set.py` exists so this stops happening: it implements a +reference solution for every `code` task and asserts it passes every check, +recomputes every `exact` answer (one by brute force), and confirms each +refactor target already passes its own checks while each debugging target +fails. It immediately caught a check where the expected value was simply +wrong — which would have docked every model on a task and been +indistinguishable from genuine difficulty. + ## The classifier is the latency floor Every routed request pays a full local classification round-trip before a @@ -232,18 +308,13 @@ that request was two orders of magnitude low. ## What's NOT built yet — pick up here -1. **Benchmark/proficiency poller — the last inert axis.** Nothing populates - `proficiency`, so `proficiency_score` is 0.5 for every candidate and 0.4 of - the weight does nothing. Concretely: `task_category` **cannot change a - routing decision**, because proficiency is the only category-dependent term - in the formula. Across 9 categories x 3 tiers, all 9 categories still pick - identically. Needs a leaderboard path (LMSYS Arena, LiveBench, Aider - polyglot — no unified API) and a self-eval harness writing - `source='self_eval'`. The blending rule is already in `config.yaml`, and - `seed_energy.py` is a working template for the sweep shape. - - Until this lands, the ~10s classifier round-trip is buying only a tier - number — the category half of its output is discarded in effect. +1. **Leaderboard priors are unfilled.** `leaderboards.yaml` ships empty on + purpose — inventing plausible-looking benchmark numbers would put + fabricated data straight into routing, the same failure as the provider's + `static_fallback` carbon constant this project already excludes. Until real + sourced figures go in, a newly listed NeuralWatt family has no prior and + relies entirely on self-eval accumulating. `python leaderboard.py --check` + lists what is missing. 2. **Sampling depth for three models.** 7 samples/model gives split-half agreement within 1.4x for 10 of 13, but `kimi-k2.7-code-fast` (29x), diff --git a/eval_proficiency.py b/eval_proficiency.py new file mode 100644 index 0000000..7297298 --- /dev/null +++ b/eval_proficiency.py @@ -0,0 +1,503 @@ +#!/usr/bin/env python3 +"""Run the self-eval task set against models and write ``proficiency``. + +Why this exists: ``proficiency_score`` is the ONLY category-dependent term in +the composite, so until this table has data, ``task_category`` cannot change a +routing decision at all — the classifier's category output is computed, paid +for, and then discarded. This is what makes it matter. + +Scoring is objective wherever the category admits it (see evals/tasks.yaml): +code is executed against checks, exact answers are compared, tool calls are +inspected structurally. Only the four prose categories fall back to a judge. + +Scores accumulate rather than replace (``proficiency_store.add_self_eval``), +so running this repeatedly tightens estimates and pushes categories past +``self_eval_min_samples`` into a proper blend with the leaderboard prior. + + python eval_proficiency.py --dry-run # plan only, no calls + python eval_proficiency.py # every identity, every task + python eval_proficiency.py --models kimi-k3 --categories coding_general + +SAFETY: `code` tasks execute model-generated Python. Isolation is a +subprocess with a wall-clock timeout, running in a temp directory — not a +container and not a real sandbox. The task prompts ask for small pure +functions, so nothing invites filesystem or network use, but treat this as +"bounded", not "safe against a hostile model". +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import sqlite3 +import subprocess +import sys +import tempfile +from collections import defaultdict +from pathlib import Path +from typing import Optional + +import requests +import yaml + +from config import RouterConfig, load_config +from proficiency_store import add_self_eval, propagate_to_variants + +TASKS_PATH = "evals/tasks.yaml" +CODE_TIMEOUT_SECONDS = 15 +CALL_TIMEOUT_SECONDS = 300 + +# Generous on purpose. These are reasoning models and the trace is billed +# against the same budget as the answer: at 1200, qwen3.6-35b spent ~4,200 +# characters thinking and returned an EMPTY content field, scoring 0.00 on +# tasks it can plainly do. The cap must clear the trace, not just the answer, +# and the harder task set makes traces longer still. +# +# Clamped per model at request time: this exceeds gemma-4-31b's advertised +# 16384 output limit, and asking for more than a model allows is an error +# rather than a silent truncation. +EVAL_MAX_TOKENS = 24000 + +# Preamble for the sandbox: common imports so ordinary solutions run, plus the +# `raises` helper the checks use. Defined AFTER the model's code so a model +# that shadows one of these does not break the harness. +HARNESS_PREAMBLE = "import collections, itertools, json, math, re, string, time\n" +HARNESS_HELPERS = ''' +def raises(exc, fn, *a, **kw): + try: + fn(*a, **kw) + except exc: + return True + except Exception: + return False + return False +''' + +FENCE_RE = re.compile(r"^\s*```[a-zA-Z]*\n(.*?)```", re.DOTALL | re.MULTILINE) + + +# --- extraction and scoring ---------------------------------------------- + +def strip_fences(text: str) -> str: + """Pull code out of a markdown block if the model added one anyway. + + Every code prompt says "no markdown fences" and models add them regularly, + so this is normalization rather than leniency — a model that solved the + task shouldn't score zero for formatting. + """ + match = FENCE_RE.search(text or "") + code = match.group(1) if match else (text or "") + # Strip the WHOLE reply's leading/trailing whitespace: a single leading + # space turns an otherwise perfect function into IndentationError, which + # is how kimi-k2.7-code scored 0.00 on every coding task in an early run. + # Internal indentation is untouched. + return code.strip() + + +def score_code(model_output: str, checks: list[str]) -> tuple[float, str]: + """Execute the model's code and eval each check against it. + + Returns (fraction of checks passing, detail). Partial credit is + deliberate: a function correct on three of four cases is genuinely better + than one that fails everything, and a binary score throws that away. + """ + code = strip_fences(model_output) + harness = ( + HARNESS_PREAMBLE + + code + + "\n" + + HARNESS_HELPERS + + f"\nCHECKS = {checks!r}\n" + + "for _i, _c in enumerate(CHECKS):\n" + + " try:\n" + + " _ok = bool(eval(_c))\n" + + " except Exception:\n" + + " _ok = False\n" + + " print('CHECK', _i, 'PASS' if _ok else 'FAIL')\n" + ) + with tempfile.TemporaryDirectory() as tmp: + script = Path(tmp) / "harness.py" + script.write_text(harness) + try: + proc = subprocess.run( + [sys.executable, str(script)], + capture_output=True, + text=True, + timeout=CODE_TIMEOUT_SECONDS, + cwd=tmp, + ) + except subprocess.TimeoutExpired: + return 0.0, "timeout" + + passed = proc.stdout.count("PASS") + if not checks: + return 0.0, "no checks" + if passed == 0 and proc.returncode != 0: + # Distinguish "wrote broken code" from "wrote code that fails cases" + first_line = (proc.stderr or "").strip().splitlines()[-1:] or [""] + return 0.0, f"did not run: {first_line[0][:80]}" + return passed / len(checks), f"{passed}/{len(checks)} checks" + + +def normalize_answer(text: str) -> str: + """Reduce a free-text reply to a comparable token. + + Models wrap a number in prose, punctuation, or thousands separators even + when told not to; none of that is the thing being measured. + """ + cleaned = (text or "").strip().replace(",", "") + numbers = re.findall(r"-?\d+(?:\.\d+)?", cleaned) + if numbers: + value = numbers[-1] # the conclusion, if it reasoned out loud + return value.rstrip("0").rstrip(".") if "." in value else value + return cleaned.lower().strip(" .!\"'") + + +def score_exact(model_output: str, expected: str) -> tuple[float, str]: + got = normalize_answer(model_output) + want = normalize_answer(expected) + return (1.0, f"{got!r}") if got == want else (0.0, f"got {got!r} want {want!r}") + + +def score_tool(tool_calls: list, task: dict) -> tuple[float, str]: + """Score a tool-use task structurally — no judge required. + + Three things are worth distinguishing and each is a task in the set: + calling the right tool, choosing correctly between several, and NOT + calling a tool when none applies. The last is scored because a model that + reaches for a tool on every prompt is a real failure mode in an agent + loop. + """ + expected_tool = task.get("expect_tool") + + if expected_tool is None: + return (1.0, "correctly abstained") if not tool_calls else ( + 0.0, + f"called {tool_calls[0]['function']['name']} when none applied", + ) + + if not tool_calls: + return 0.0, "no tool call" + + called = tool_calls[0]["function"]["name"] + if called != expected_tool: + return 0.0, f"called {called}, wanted {expected_tool}" + + # Right tool. Give partial credit and check the arguments that matter. + expected_args = task.get("expect_args") or {} + if not expected_args: + return 1.0, "correct tool" + try: + got_args = json.loads(tool_calls[0]["function"].get("arguments") or "{}") + except json.JSONDecodeError: + return 0.5, "correct tool, unparseable arguments" + + matched = 0 + for key, want in expected_args.items(): + got = got_args.get(key) + if isinstance(want, str) and isinstance(got, str): + ok = want.lower() in got.lower() + else: + ok = str(got) == str(want) + matched += bool(ok) + # Floor at 0.5 for picking the right tool; arguments carry the rest. + return 0.5 + 0.5 * (matched / len(expected_args)), f"correct tool, {matched}/{len(expected_args)} args" + + +# --- provider calls ------------------------------------------------------- + +def call_model( + base_url: str, + api_key: str, + model_id: str, + task: dict, + max_output_tokens: Optional[int] = None, +) -> tuple[str, list, bool]: + """One completion. Returns (text, tool_calls, truncated).""" + budget = EVAL_MAX_TOKENS + if max_output_tokens: + budget = min(budget, max_output_tokens) + body = { + "model": model_id, + "messages": [{"role": "user", "content": task["prompt"]}], + "temperature": 0, + "max_tokens": budget, + } + if task.get("tools"): + body["tools"] = task["tools"] + + resp = requests.post( + f"{base_url}/chat/completions", + headers={"authorization": f"Bearer {api_key}"}, + json=body, + timeout=CALL_TIMEOUT_SECONDS, + ) + resp.raise_for_status() + choice = (resp.json().get("choices") or [{}])[0] + message = choice.get("message") or {} + # finish_reason 'length' means the budget ran out mid-answer. Whatever + # came back is an artifact of the cap, so the caller skips rather than + # scoring it — the same rule as an unusable judge reply. + truncated = choice.get("finish_reason") == "length" + return message.get("content") or "", message.get("tool_calls") or [], truncated + + +JUDGE_SYSTEM = ( + "You are scoring one model's answer against a rubric. Reply with ONLY a " + 'JSON object: {"score": , "reason": ""}. ' + "Do not explain your reasoning outside the JSON. " + "Be strict: award a high score only if every rubric requirement is met." +) + +JSON_OBJECT_RE = re.compile(r"\{.*\}", re.DOTALL) + + +def extract_judge_json(raw: str) -> Optional[dict]: + """Pull the score object out of a judge reply. + + Judges are themselves reasoning models and leak their thinking into the + content despite response_format, so the object usually arrives wrapped in + prose ("Let me evaluate the answer against the rubric: ... {...}"). Before + this existed, 44% of judge calls were unparseable. + """ + if not raw: + return None + candidates = [raw] + match = JSON_OBJECT_RE.search(raw) + if match: + candidates.append(match.group(0)) + for candidate in candidates: + try: + parsed = json.loads(candidate) + except json.JSONDecodeError: + continue + if isinstance(parsed, dict) and "score" in parsed: + return parsed + return None + + +def score_judge( + base_url: str, api_key: str, judge_model: str, task: dict, answer: str +) -> Optional[tuple[float, str]]: + """Score a prose task against its rubric. + + Returns None when the judge itself failed, so the caller can SKIP the + task rather than record it. Scoring a judge malfunction as 0.0 would + charge the model for the judge's formatting — it put genuine 0.0 entries + in translation and general_chat on the first run, for answers that were + in fact fine. + """ + body = { + "model": judge_model, + "messages": [ + {"role": "system", "content": JUDGE_SYSTEM}, + { + "role": "user", + "content": ( + f"TASK GIVEN TO THE MODEL:\n{task['prompt']}\n\n" + f"RUBRIC:\n{task['rubric']}\n\n" + f"THE MODEL'S ANSWER:\n{answer}" + ), + }, + ], + "temperature": 0, + # Generous, because a reasoning judge spends tokens thinking before + # the JSON and a truncated object is unparseable. + "max_tokens": 1500, + "response_format": {"type": "json_object"}, + } + resp = requests.post( + f"{base_url}/chat/completions", + headers={"authorization": f"Bearer {api_key}"}, + json=body, + timeout=CALL_TIMEOUT_SECONDS, + ) + resp.raise_for_status() + raw = ((resp.json().get("choices") or [{}])[0].get("message") or {}).get("content") or "" + parsed = extract_judge_json(raw) + if parsed is None: + return None + try: + score = float(parsed["score"]) + except (KeyError, TypeError, ValueError): + return None + return max(0.0, min(1.0, score)), str(parsed.get("reason", ""))[:70] + + +# --- selection ------------------------------------------------------------ + +def eval_identities(conn: sqlite3.Connection, cfg: RouterConfig) -> list[dict]: + """Model rows worth measuring directly. + + Flex rows are excluded: same weights, same reasoning setting, different + queue — their answer quality is their family's, and they inherit it via + ``propagate_to_variants``. ``-fast`` rows ARE measured, because reasoning + being off genuinely changes answers. + """ + placeholders = ",".join("?" * len(cfg.routing.allowed_access_levels)) + rows = conn.execute( + f""" + SELECT model_id, base_model_id, reasoning_mode, max_output_tokens FROM models + WHERE availability = 'active' + AND access_level IN ({placeholders}) + AND latency_class = 'standard' + ORDER BY model_id + """, + tuple(cfg.routing.allowed_access_levels), + ).fetchall() + return [ + { + "model_id": r[0], + "base_model_id": r[1], + "reasoning_mode": r[2], + "max_output_tokens": r[3], + } + for r in rows + ] + + +ALTERNATE_JUDGE = "qwen3.6-35b" + + +def judge_for(model_id: str, default_judge: str) -> str: + """Pick a judge that is not the model being judged. + + A model scoring its own prose is a known bias, and the default judge is + itself in the evaluated set. Swapping in an alternate for its own family + costs nothing and removes the obvious conflict. + """ + from poller import parse_base_model_id + + if parse_base_model_id(model_id) == parse_base_model_id(default_judge): + return ALTERNATE_JUDGE + return default_judge + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--tasks", default=TASKS_PATH) + ap.add_argument("--models", help="comma-separated model_ids") + ap.add_argument("--categories", help="comma-separated categories") + ap.add_argument("--judge-model", default="kimi-k3") + ap.add_argument("--dry-run", action="store_true") + args = ap.parse_args() + + cfg = load_config("config.yaml") + tasks = (yaml.safe_load(Path(args.tasks).read_text()) or {}).get("tasks") or [] + + if args.categories: + wanted = {c.strip() for c in args.categories.split(",")} + tasks = [t for t in tasks if t["category"] in wanted] + + unknown = {t["category"] for t in tasks} - set(cfg.proficiency.categories) + if unknown: + print(f"tasks reference unknown categories: {sorted(unknown)}", file=sys.stderr) + return 1 + + conn = sqlite3.connect(cfg.database.path) + identities = eval_identities(conn, cfg) + if args.models: + wanted = {m.strip() for m in args.models.split(",")} + identities = [i for i in identities if i["model_id"] in wanted] + + if not identities or not tasks: + print("nothing to run", file=sys.stderr) + return 1 + + judged = sum(1 for t in tasks if t["kind"] == "judge") + print( + f"{len(identities)} models x {len(tasks)} tasks = " + f"{len(identities) * len(tasks)} calls" + + (f" (+{len(identities) * judged} judge calls)" if judged else "") + ) + if args.dry_run: + for i in identities: + budget = min(EVAL_MAX_TOKENS, i["max_output_tokens"] or EVAL_MAX_TOKENS) + print( + f" {i['model_id']:24s} reasoning={i['reasoning_mode']:8s} " + f"budget={budget}" + ) + print(f" judge: {args.judge_model}") + return 0 + + settings = cfg.dispatch_providers["neuralwatt"] + api_key = os.environ.get(settings.api_key_env) + if not api_key: + print(f"{settings.api_key_env} is not set", file=sys.stderr) + return 1 + + for identity in identities: + model_id = identity["model_id"] + by_category: dict[str, list[float]] = defaultdict(list) + print(f"\n{model_id}") + + for task in tasks: + try: + text, tool_calls, truncated = call_model( + settings.base_url, + api_key, + model_id, + task, + identity["max_output_tokens"], + ) + except requests.RequestException as e: + print(f" {task['id']:30s} CALL FAILED {type(e).__name__}") + continue + + if truncated and not tool_calls: + print( + f" {task['id']:30s} --- truncated at " + f"{EVAL_MAX_TOKENS} tokens, skipped" + ) + continue + + kind = task["kind"] + try: + if kind == "code": + score, detail = score_code(text, task["checks"]) + elif kind == "exact": + score, detail = score_exact(text, str(task["answer"])) + elif kind == "tool": + score, detail = score_tool(tool_calls, task) + elif kind == "judge": + judged = score_judge( + settings.base_url, + api_key, + judge_for(model_id, args.judge_model), + task, + text, + ) + if judged is None: + # No sample rather than a zero: a judge that failed to + # emit parseable JSON says nothing about the model. + print(f" {task['id']:30s} --- judge unusable, skipped") + continue + score, detail = judged + else: + print(f" {task['id']:30s} unknown kind {kind!r}") + continue + except requests.RequestException as e: + print(f" {task['id']:30s} SCORING FAILED {type(e).__name__}") + continue + + by_category[task["category"]].append(score) + print(f" {task['id']:30s} {score:4.2f} {detail}") + + for category, scores in by_category.items(): + add_self_eval(conn, cfg, model_id, "neuralwatt", category, scores) + conn.commit() + + # Flex siblings inherit; anything measured directly keeps its own score. + propagated = 0 + for base in {i["base_model_id"] for i in identities}: + propagated += propagate_to_variants(conn, cfg, base, "neuralwatt") + conn.commit() + print(f"\npropagated {propagated} inherited rows to serving variants") + conn.close() + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/tasks.yaml b/evals/tasks.yaml new file mode 100644 index 0000000..0c94caa --- /dev/null +++ b/evals/tasks.yaml @@ -0,0 +1,490 @@ +# Self-eval task set. One score per task per model; scores accumulate into +# proficiency.self_eval_score and self_eval_samples. +# +# Four kinds, chosen so each category is scored the most objective way it +# admits: +# +# code model writes Python; each check is eval'd against it in a +# subprocess. Score = fraction of checks passing. Fully objective. +# exact model replies with one value; compared after normalization. +# tool model is given a tool schema; scored on whether it calls the right +# tool with the right arguments. Structural, no judge needed. +# judge a strong model scores the output against a rubric. Only for the +# prose categories, where nothing checkable exists. +# +# DIFFICULTY: the happy path is not worth testing. Every current model passes +# "reverse a list", and a category where everyone scores 1.00 discriminates no +# better than the constant 0.5 it replaced. Each task here carries at least one +# edge case a plausible-looking solution gets wrong: touching vs overlapping +# intervals, present-but-falsy values, late-binding closures, greedy regexes, +# empty input, or a trap in the arithmetic. +# +# Every `code` task is validated against a reference solution by +# tests/test_task_set.py. A check my own reference cannot pass is a broken +# check, and would score the task set rather than the model — which has +# already happened here once. +# +# Keep prompts tight and self-contained. An ambiguous task scores the prompt. + +tasks: + # --- coding_general ----------------------------------------------------- + - id: merge_intervals + category: coding_general + kind: code + entrypoint: merge_intervals + prompt: | + Write a Python function `merge_intervals(intervals)` where intervals is + a list of [start, end] lists. Merge all overlapping intervals and return + a new list of [start, end] lists sorted by start. Intervals that merely + touch (one ends exactly where the next begins) must be merged. Input may + be unsorted and may contain intervals fully nested inside others. + Reply with ONLY the function definition — no explanation, no fences. + checks: + - 'merge_intervals([[1,3],[2,6],[8,10]]) == [[1,6],[8,10]]' + - 'merge_intervals([]) == []' + - 'merge_intervals([[1,4],[4,5]]) == [[1,5]]' + - 'merge_intervals([[1,10],[2,3]]) == [[1,10]]' + - 'merge_intervals([[5,6],[1,2]]) == [[1,2],[5,6]]' + - 'merge_intervals([[1,2]]) == [[1,2]]' + + - id: parse_semver + category: coding_general + kind: code + entrypoint: parse_semver + prompt: | + Write a Python function `parse_semver(version)` that parses a semantic + version string into a dict with keys: major, minor, patch (ints), and + prerelease, build (strings, or None when absent). Valid examples: + "1.2.3", "1.2.3-alpha.1", "1.2.3+build.5", "1.2.3-rc.1+exp.sha.5114f85". + Raise ValueError if the string is not a valid semantic version, for + example "1.2" or "1.2.x". + Reply with ONLY the function definition — no explanation, no fences. + checks: + - 'parse_semver("1.2.3") == {"major":1,"minor":2,"patch":3,"prerelease":None,"build":None}' + - 'parse_semver("1.2.3-alpha.1")["prerelease"] == "alpha.1"' + - 'parse_semver("1.2.3+build.5")["build"] == "build.5"' + - 'parse_semver("1.2.3-rc.1+exp.sha.5114f85")["prerelease"] == "rc.1"' + - 'parse_semver("1.2.3-rc.1+exp.sha.5114f85")["build"] == "exp.sha.5114f85"' + - 'parse_semver("0.0.0")["major"] == 0' + - 'raises(ValueError, parse_semver, "1.2")' + - 'raises(ValueError, parse_semver, "1.2.x")' + + - id: word_wrap + category: coding_general + kind: code + entrypoint: word_wrap + prompt: | + Write a Python function `word_wrap(text, width)` returning a list of + lines. Split on whitespace and pack as many words per line as fit within + `width` characters, joining words with a single space. Never split a + word: a word longer than `width` gets its own line. Runs of whitespace + collapse. Empty or whitespace-only text returns an empty list. + Reply with ONLY the function definition — no explanation, no fences. + checks: + - 'word_wrap("the quick brown fox", 10) == ["the quick", "brown fox"]' + - 'word_wrap("", 5) == []' + - 'word_wrap(" ", 5) == []' + - 'word_wrap("supercalifragilistic", 5) == ["supercalifragilistic"]' + - 'word_wrap("a b c", 3) == ["a b", "c"]' + - 'word_wrap("aa bb cc", 5) == ["aa bb", "cc"]' + + # --- coding_refactor ---------------------------------------------------- + - id: refactor_falsy_defaults + category: coding_refactor + kind: code + entrypoint: apply_settings + prompt: | + Refactor this function to remove the repetition. Behaviour must be + preserved EXACTLY, including for values that are present but falsy. + Reply with ONLY the rewritten function — no explanation, no fences. + + def apply_settings(overrides): + result = {} + if "retries" in overrides: + result["retries"] = overrides["retries"] + else: + result["retries"] = 3 + if "timeout" in overrides: + result["timeout"] = overrides["timeout"] + else: + result["timeout"] = 30 + if "verbose" in overrides: + result["verbose"] = overrides["verbose"] + else: + result["verbose"] = False + return result + checks: + - 'apply_settings({}) == {"retries":3,"timeout":30,"verbose":False}' + - 'apply_settings({"retries":0})["retries"] == 0' + - 'apply_settings({"timeout":0})["timeout"] == 0' + - 'apply_settings({"verbose":True})["verbose"] is True' + - 'apply_settings({"retries":5}) == {"retries":5,"timeout":30,"verbose":False}' + + - id: refactor_first_match + category: coding_refactor + kind: code + entrypoint: first_match + prompt: | + Refactor this to remove the nested loops and the flag variable. + Behaviour must be preserved exactly, including which item wins when + several match. Reply with ONLY the rewritten function — no explanation, + no fences. + + def first_match(items, predicates): + found = None + done = False + for item in items: + if done: + break + for p in predicates: + if p(item): + found = item + done = True + break + return found + checks: + - 'first_match([1,2,3,4], [lambda x: x > 2]) == 3' + - 'first_match([1,2,3], [lambda x: x > 10]) is None' + - 'first_match([], [lambda x: True]) is None' + - 'first_match([1,2,3], []) is None' + - 'first_match([5,2,9], [lambda x: x > 8, lambda x: x < 3]) == 2' + - 'first_match([0,1], [lambda x: x == 0]) == 0' + + - id: refactor_dispatch + category: coding_refactor + kind: code + entrypoint: describe + prompt: | + Refactor this if/elif chain into a table-driven lookup. Behaviour must be + preserved exactly for every input, including inputs that match no case. + Reply with ONLY the rewritten code — no explanation, no fences. + + def describe(code): + if code == 200: + return "ok" + elif code == 201: + return "created" + elif code == 404: + return "not found" + elif code == 500: + return "server error" + else: + return "unknown" + checks: + - 'describe(200) == "ok"' + - 'describe(404) == "not found"' + - 'describe(500) == "server error"' + - 'describe(418) == "unknown"' + - 'describe(0) == "unknown"' + - 'describe(None) == "unknown"' + + # --- debugging ---------------------------------------------------------- + - id: debug_late_binding + category: debugging + kind: code + entrypoint: make_multipliers + prompt: | + This should return one multiplier function per factor, but every + returned function behaves the same. Fix it. Reply with ONLY the + corrected function — no explanation, no fences. + + def make_multipliers(factors): + out = [] + for f in factors: + out.append(lambda x: x * f) + return out + checks: + - '[m(2) for m in make_multipliers([1,2,3])] == [2,4,6]' + - '[m(10) for m in make_multipliers([0,1])] == [0,10]' + - 'make_multipliers([]) == []' + - 'make_multipliers([7])[0](3) == 21' + + - id: debug_binary_search + category: debugging + kind: code + entrypoint: bsearch + prompt: | + This binary search should return the index of target in a sorted list, + or -1 if absent. It is wrong for some inputs — one case loops forever. + Fix it. Reply with ONLY the corrected function — no explanation, no + fences. + + def bsearch(items, target): + lo, hi = 0, len(items) + while lo < hi: + mid = (lo + hi) // 2 + if items[mid] == target: + return mid + elif items[mid] < target: + lo = mid + else: + hi = mid + return -1 + checks: + - 'bsearch([1,3,5,7], 7) == 3' + - 'bsearch([1,3,5,7], 1) == 0' + - 'bsearch([], 1) == -1' + - 'bsearch([1], 1) == 0' + - 'bsearch([1,3], 2) == -1' + - 'bsearch([1,2,3,4,5,6], 6) == 5' + + - id: debug_greedy_regex + category: debugging + kind: code + entrypoint: extract_tags + prompt: | + This should return the name inside each angle-bracket tag, in order, but + it returns the wrong thing when there is more than one tag. Fix it. + Reply with ONLY the corrected function — no explanation, no fences. + + import re + + def extract_tags(text): + return re.findall(r"<(.+)>", text) + checks: + - 'extract_tags("") == ["a","b"]' + - 'extract_tags("") == ["one"]' + - 'extract_tags("") == []' + - 'extract_tags("no tags here") == []' + - 'extract_tags("x y z") == ["a","bc"]' + + # --- reasoning_math ----------------------------------------------------- + - id: math_percent_trap + category: reasoning_math + kind: exact + answer: "100" + prompt: | + A price rises by 20%, then falls by 20% of its new value. The final + price is 96. What was the original price? Reply with ONLY the number. + + - id: math_rate_trap + category: reasoning_math + kind: exact + answer: "3" + prompt: | + Three machines take 3 minutes to make 3 widgets, each machine working + independently at the same constant rate. How many minutes do 100 + machines take to make 100 widgets? Reply with ONLY the number. + + - id: math_counting + category: reasoning_math + kind: exact + answer: "4536" + prompt: | + How many 4-digit whole numbers have four distinct digits and do not + begin with 0? Reply with ONLY the number. + + # --- tool_use_agentic --------------------------------------------------- + - id: tool_multi_arg + category: tool_use_agentic + kind: tool + prompt: Convert 250 US dollars into Japanese yen. + expect_tool: convert_currency + expect_args: + amount: 250 + from_currency: USD + to_currency: JPY + tools: + - type: function + function: + name: get_weather + description: Get the current weather for a location. + parameters: + type: object + properties: + location: {type: string} + required: [location] + - type: function + function: + name: convert_currency + description: Convert an amount between two currencies. + parameters: + type: object + properties: + amount: {type: number} + from_currency: {type: string, description: ISO 4217 code} + to_currency: {type: string, description: ISO 4217 code} + required: [amount, from_currency, to_currency] + + - id: tool_no_tool_needed + category: tool_use_agentic + kind: tool + prompt: | + It is 1:20pm and my meeting starts at 3pm. How many minutes away is it? + expect_tool: null # plain arithmetic; both times are already given + tools: + - type: function + function: + name: get_calendar_event + description: Look up a calendar event by title. + parameters: + type: object + properties: + title: {type: string} + required: [title] + - type: function + function: + name: get_current_time + description: Get the current wall-clock time. + parameters: + type: object + properties: {} + + - id: tool_abstain_creative + category: tool_use_agentic + kind: tool + prompt: Write me a haiku about winter. + expect_tool: null + tools: + - type: function + function: + name: get_weather + description: Get the current weather for a location. + parameters: + type: object + properties: + location: {type: string} + required: [location] + + # --- docs_writing ------------------------------------------------------- + - id: docs_function + category: docs_writing + kind: judge + prompt: | + Write a docstring for this function. Reply with ONLY the docstring text. + + def retry(fn, attempts=3, backoff=2.0): + delay = 1.0 + for i in range(attempts): + try: + return fn() + except Exception: + if i == attempts - 1: + raise + time.sleep(delay) + delay *= backoff + rubric: | + Score 0-1. Award 1.0 ONLY if it states what the function does, documents + every parameter including defaults, states the return value, AND states + that the last exception is re-raised when all attempts fail. Deduct 0.3 + for any invented behaviour the code does not have. Deduct 0.2 if it omits + that the delay grows by `backoff` between attempts. + + - id: docs_gotcha + category: docs_writing + kind: judge + prompt: | + Write a docstring for this function. Reply with ONLY the docstring text. + + def dedupe(items, key=None): + seen = set() + out = [] + for item in items: + k = key(item) if key else item + if k in seen: + continue + seen.add(k) + out.append(item) + return out + rubric: | + Score 0-1. Award 1.0 ONLY if it states that ORDER IS PRESERVED and that + the FIRST occurrence is kept, documents `key`, and notes that elements + (or their keys) must be hashable. Deduct 0.4 if it omits the + order-preservation guarantee — that is the whole reason to use this over + set(). Deduct 0.3 if it omits the hashability requirement. + + # --- summarization ------------------------------------------------------ + - id: summarize_incident + category: summarization + kind: judge + prompt: | + Summarize in at most two sentences: + + At 02:14 UTC the checkout service began returning 502s. The on-call + engineer found the connection pool exhausted. A deploy at 01:58 had + lowered the pool size from 50 to 5 through a bad template variable. The + deploy was rolled back at 02:31 and errors stopped by 02:34. Roughly + 12,000 requests failed. No data was lost. + rubric: | + Score 0-1. Award 1.0 ONLY if it names the ROOT CAUSE specifically (a bad + template variable in the 01:58 deploy cut the pool from 50 to 5), the + resolution (rollback), and the impact (~12k failed requests, no data + lost), in two sentences or fewer. Deduct 0.4 for saying only "the + connection pool was exhausted" — that is the symptom, not the cause. + Deduct 0.5 for any invented detail. + + - id: summarize_buried_lede + category: summarization + kind: judge + prompt: | + Summarize the single most important point in one sentence: + + The migration ran for six hours. Throughput averaged 4,200 rows per + second, peaking at 6,100. The team used a rolling window of 5,000 rows + per batch. Disk usage on the replica grew steadily. Partway through, a + unique constraint on the accounts table silently rejected 812 rows, + which were logged to a dead-letter file that nobody has yet processed. + CPU stayed under 40% throughout. + rubric: | + Score 0-1. The important point is that 812 rows were silently dropped and + remain unprocessed — everything else is routine telemetry. Award 1.0 only + if the summary leads with that. Score 0.3 or below if it summarizes the + throughput statistics instead. Must be one sentence. + + # --- translation -------------------------------------------------------- + - id: translate_technical + category: translation + kind: judge + prompt: | + Translate into French. Reply with ONLY the translation. + + "The connection pool was exhausted because a recent deploy reduced its + size. Roll back the deploy and the errors should stop within a few + minutes." + rubric: | + Score 0-1 on accuracy and fluency. Award 1.0 only for correct technical + register on "connection pool", "deploy" and "roll back", AND natural + French rather than a word-for-word calque. Deduct 0.3 per omission or + untranslated fragment. + + - id: translate_register + category: translation + kind: judge + prompt: | + Translate into Spanish, preserving the hedging and the informal tone. + Reply with ONLY the translation. + + "I'm not totally sure this is the right call, but I'd lean towards + shipping it and seeing what breaks — we can always roll it back." + rubric: | + Score 0-1. Award 1.0 only if the HEDGING is preserved ("not totally + sure", "I'd lean towards") rather than flattened into a confident + statement, the register stays informal, and "roll it back" is rendered + idiomatically. Deduct 0.4 if the hedging is lost. + + # --- general_chat ------------------------------------------------------- + - id: chat_explain + category: general_chat + kind: judge + prompt: | + Explain to a non-programmer, in under 100 words, why a program can be + correct and still be too slow to use. + rubric: | + Score 0-1. Award 1.0 only if it distinguishes correctness from + performance, gives at least one concrete relatable example, stays under + 100 words, and leaves no jargon unexplained. Deduct 0.3 if over 100 + words. + + - id: chat_pushback + category: general_chat + kind: judge + prompt: | + A colleague says "we should rewrite the whole service in Rust, it'll be + faster." Reply in under 80 words, taking the suggestion seriously but + identifying what you would want to know first. + rubric: | + Score 0-1. Award 1.0 only if it avoids both pure agreement and pure + dismissal, names at least two specific things worth establishing first + (for example where time is actually spent, migration cost, team + familiarity), and stays under 80 words. Score 0.3 or below for a reply + that simply agrees or simply refuses. diff --git a/leaderboard.py b/leaderboard.py new file mode 100644 index 0000000..5ce4fd8 --- /dev/null +++ b/leaderboard.py @@ -0,0 +1,169 @@ +#!/usr/bin/env python3 +"""Import curated leaderboard priors into the ``proficiency`` table. + +Reads ``leaderboards.yaml`` (family -> category -> score), writes +``leaderboard_score`` for every catalog row in each family, and re-blends. + +Run manually, or after editing the YAML: + python leaderboard.py + python leaderboard.py --check # report coverage, write nothing + +The coverage report is the point of running this on a schedule. NeuralWatt +adds models; a newly listed family has no prior and no self-eval history, so +it sits at the neutral 0.5 and is indistinguishable from a model that was +measured and found average. Naming those families is what turns "quietly +unmeasured" into "needs a prior". +""" + +from __future__ import annotations + +import argparse +import sqlite3 +import sys +from pathlib import Path + +import yaml + +from config import RouterConfig, load_config +from proficiency_store import set_leaderboard + +LEADERBOARDS_PATH = "leaderboards.yaml" + + +def load_priors(path: str | Path, cfg: RouterConfig) -> dict[str, dict[str, float]]: + """Parse and validate the curated file into {family: {category: score}}. + + Validation is strict rather than forgiving: a typo'd category silently + dropped would look exactly like a benchmark that does not cover it, and + an out-of-range score would skew the min-max normalization downstream. + """ + raw = yaml.safe_load(Path(path).read_text()) or {} + families = raw.get("families") or {} + if not isinstance(families, dict): + raise ValueError("leaderboards.yaml: 'families' must be a mapping") + + allowed = set(cfg.proficiency.categories) + priors: dict[str, dict[str, float]] = {} + + for family, entry in families.items(): + if not isinstance(entry, dict): + raise ValueError(f"leaderboards.yaml: {family!r} must be a mapping") + scores = entry.get("scores") or {} + if not isinstance(scores, dict): + raise ValueError(f"leaderboards.yaml: {family!r}.scores must be a mapping") + + unknown = set(scores) - allowed + if unknown: + raise ValueError( + f"leaderboards.yaml: {family!r} has categories not in " + f"proficiency.categories: {sorted(unknown)}" + ) + for category, score in scores.items(): + if not isinstance(score, (int, float)) or not (0.0 <= score <= 1.0): + raise ValueError( + f"leaderboards.yaml: {family!r}.{category} must be a number " + f"in 0..1, got {score!r}" + ) + if scores: + priors[family] = {c: float(s) for c, s in scores.items()} + + return priors + + +def active_families(conn: sqlite3.Connection, cfg: RouterConfig) -> dict[str, list[str]]: + """Routable families mapped to their catalog rows.""" + placeholders = ",".join("?" * len(cfg.routing.allowed_access_levels)) + rows = conn.execute( + f""" + SELECT base_model_id, model_id FROM models + WHERE availability = 'active' + AND access_level IN ({placeholders}) + ORDER BY base_model_id, model_id + """, + tuple(cfg.routing.allowed_access_levels), + ).fetchall() + out: dict[str, list[str]] = {} + for base, model_id in rows: + out.setdefault(base, []).append(model_id) + return out + + +def report_coverage( + families: dict[str, list[str]], priors: dict[str, dict[str, float]] +) -> list[str]: + """Print per-family coverage; return the families with no prior at all.""" + missing = [] + print(f"{'family':22s}{'rows':>6}{'categories with a prior':>26}") + for family, model_ids in sorted(families.items()): + scores = priors.get(family, {}) + if not scores: + missing.append(family) + print(f"{family:22s}{len(model_ids):>6}{'— none —':>26}") + else: + print(f"{family:22s}{len(model_ids):>6}{len(scores):>26}") + return missing + + +def apply_priors( + conn: sqlite3.Connection, + cfg: RouterConfig, + families: dict[str, list[str]], + priors: dict[str, dict[str, float]], +) -> int: + """Write each family's prior onto every catalog row in that family.""" + written = 0 + for family, model_ids in families.items(): + scores = priors.get(family) + if not scores: + continue + for model_id in model_ids: + for category, score in scores.items(): + set_leaderboard(conn, cfg, model_id, "neuralwatt", category, score) + written += 1 + conn.commit() + return written + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--file", default=LEADERBOARDS_PATH) + ap.add_argument( + "--check", action="store_true", help="report coverage without writing" + ) + args = ap.parse_args() + + cfg = load_config("config.yaml") + try: + priors = load_priors(args.file, cfg) + except (ValueError, FileNotFoundError) as e: + print(f"{e}", file=sys.stderr) + return 1 + + conn = sqlite3.connect(cfg.database.path) + families = active_families(conn, cfg) + + missing = report_coverage(families, priors) + + if not args.check: + written = apply_priors(conn, cfg, families, priors) + print(f"\nwrote {written} leaderboard scores") + conn.close() + + if missing: + print( + f"\nWARNING: {len(missing)} active famil" + f"{'y has' if len(missing) == 1 else 'ies have'} no leaderboard prior: " + f"{', '.join(missing)}", + file=sys.stderr, + ) + print( + " Until self-eval accumulates, these score the neutral 0.5 — " + "indistinguishable from a model measured and found average.\n" + f" Add real, sourced figures to {args.file}.", + file=sys.stderr, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/leaderboards.yaml b/leaderboards.yaml new file mode 100644 index 0000000..9ca25e2 --- /dev/null +++ b/leaderboards.yaml @@ -0,0 +1,41 @@ +# Curated leaderboard priors, keyed by model FAMILY (models.base_model_id). +# +# Why this file is hand-maintained rather than scraped: there is no unified +# leaderboard API, model naming differs everywhere (this catalog's `kimi-k3` +# against a leaderboard's "Kimi K3 (0731)"), and a scraper that silently +# starts returning nothing looks identical to a model that scores nothing. +# +# What it is FOR: cold start. NeuralWatt adds models, and a newly listed one +# has zero self-eval history — without a prior it sits at the neutral 0.5 and +# is indistinguishable from a model that was measured and found average. A +# prior gives it a defensible starting position until the eval harness has +# run enough samples to take over (proficiency.self_eval_min_samples). +# +# SHIPPED EMPTY ON PURPOSE. Inventing plausible-looking numbers here would put +# fabricated data directly into routing decisions — the same failure as the +# provider's static_fallback carbon constant, which this project already +# excludes for exactly that reason. Fill in only figures you actually looked +# up, and say where they came from in `source`. +# +# Serving variants inherit from their family automatically: an entry for +# `kimi-k3` covers kimi-k3-fast and kimi-k3-flex. Do not list variants. +# +# `python leaderboard.py` imports this and names any active family missing an +# entry. Categories must come from proficiency.categories in config.yaml, and +# scores are 0-1. Partial entries are fine — list only the categories a +# benchmark actually measures, and leave the rest to self-eval. + +families: {} + # Shape, for when you have real numbers to add: + # + # kimi-k3: + # source: "aider polyglot 2026-08-14; livebench 2026-08-01" + # scores: + # coding_general: 0.82 + # coding_refactor: 0.79 + # reasoning_math: 0.88 + # + # gemma-4-31b: + # source: "lmarena 2026-08-10" + # scores: + # general_chat: 0.61 diff --git a/poller.py b/poller.py index a33f517..b14a999 100644 --- a/poller.py +++ b/poller.py @@ -65,6 +65,28 @@ def parse_serving_class(model_id: str) -> tuple[str, str, str]: ) +def parse_base_model_id(model_id: str) -> str: + """Reduce a catalog id to the model family underneath it. + + ``glm-5.2-short-fast-flex`` and ``glm-5.2`` are the same weights served + differently, and ``deepseek-ai/DeepSeek-V4-Flash`` is the HF-style + duplicate of ``deepseek-v4-flash``. Proficiency is a property of the + weights, not of the queue they sit in, so scores are keyed on this and + every serving variant inherits from its family. Leaderboard priors work + the same way — no benchmark rates a ``-flex`` row separately. + + Note this deliberately collapses ``-fast`` too, even though reasoning + being off does change answer quality. The eval runner scores ``-fast`` + variants separately and overrides the inherited value; the family is the + fallback, not the final word. + """ + namespace_stripped = model_id.rsplit("/", 1)[-1] + segments = namespace_stripped.lower().split("-") + while len(segments) > 1 and segments[-1] in SERVING_SUFFIXES: + segments.pop() + return "-".join(segments) + + def parse_access_level(display_name: Optional[str], description: Optional[str]) -> str: """Derive an access level from the catalog's prose. @@ -85,6 +107,7 @@ def parse_access_level(display_name: Optional[str], description: Optional[str]) class ModelRow: model_id: str provider: str + base_model_id: str display_name: Optional[str] cost_per_1m_prompt: Optional[float] cost_per_1m_completion: Optional[float] @@ -142,6 +165,7 @@ def fetch_neuralwatt() -> list[ModelRow]: ModelRow( model_id=model_id, provider="neuralwatt", + base_model_id=parse_base_model_id(model_id), display_name=meta.get("display_name"), cost_per_1m_prompt=pricing.get("input_per_million"), cost_per_1m_completion=pricing.get("output_per_million"), @@ -174,15 +198,16 @@ def upsert(conn: sqlite3.Connection, rows: list[ModelRow], cfg: RouterConfig) -> conn.execute( """ INSERT INTO models ( - model_id, provider, display_name, + model_id, provider, base_model_id, display_name, cost_per_1m_prompt, cost_per_1m_completion, cost_per_1m_prompt_cached, context_window, effective_context_window, max_output_tokens, supports_tools, supports_json_mode, supports_vision, supports_reasoning, reasoning_default_enabled, latency_class, reasoning_mode, context_variant, access_level, pricing_tbd, deprecated, availability, last_updated - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(model_id, provider) DO UPDATE SET + base_model_id = excluded.base_model_id, display_name = excluded.display_name, cost_per_1m_prompt = excluded.cost_per_1m_prompt, cost_per_1m_completion = excluded.cost_per_1m_completion, @@ -207,6 +232,7 @@ def upsert(conn: sqlite3.Connection, rows: list[ModelRow], cfg: RouterConfig) -> ( r.model_id, r.provider, + r.base_model_id, r.display_name, r.cost_per_1m_prompt, r.cost_per_1m_completion, diff --git a/proficiency.py b/proficiency.py new file mode 100644 index 0000000..fe8c66a --- /dev/null +++ b/proficiency.py @@ -0,0 +1,103 @@ +"""Pure proficiency blending for the local LLM model router. + +Like ``scoring.py``, ``tiering.py`` and ``routing.py``, this module is free of +I/O: scores and thresholds come in as arguments. ``eval_proficiency.py`` owns +the DB writes and the model calls. + +Two independent sources feed one number per (model, category): + +- **leaderboard** — a curated prior from published benchmarks. Its job is + cold start: NeuralWatt adds models, and a newly listed one has no self-eval + history at all. Without a prior it scores the neutral 0.5 and is + indistinguishable from a model that was measured and found average. +- **self-eval** — this router's own task set, run against the real endpoint. + More predictive of actual routing quality, but it accumulates slowly. + +Design doc §3.3 gives the blend as ``0.3 x leaderboard + 0.7 x self_eval`` +once self-eval crosses ``self_eval_min_samples``, falling back to the +leaderboard alone before that so thin, noisy self-eval data cannot dominate +early. + +That rule assumes a leaderboard entry exists. It often will not — the curated +file is hand-maintained and NeuralWatt ships models faster than public +benchmarks cover them. Taken literally, a model with no prior and 9 samples +would score nothing at all, which is strictly worse than the 9 samples it +actually has. So the fallback ladder is: + + leaderboard + enough self-eval -> weighted blend 'blended' + enough self-eval, no prior -> self-eval alone 'self_eval' + prior, not enough self-eval -> leaderboard alone 'leaderboard' + thin self-eval, no prior -> self-eval alone 'self_eval_thin' + neither -> None (neutral 0.5 downstream) + +``self_eval_thin`` is deliberately distinguishable: it is real measurement, +but from too few samples to trust as much as the label 'self_eval' implies, +and a caller wanting to exclude it can. +""" + +from __future__ import annotations + +from typing import Literal, Optional + +Source = Literal["blended", "self_eval", "leaderboard", "self_eval_thin"] + + +def blend( + leaderboard_score: Optional[float], + self_eval_score: Optional[float], + self_eval_samples: int, + *, + leaderboard_weight: float, + self_eval_weight: float, + min_samples: int, +) -> tuple[Optional[float], Optional[Source]]: + """Combine the two sources into one score, with the source that produced it. + + Returns ``(None, None)`` when neither source has anything, which leaves + the candidate on the neutral 0.5 in ``scoring.proficiency_score`` rather + than penalizing it for being unmeasured. + """ + has_leaderboard = leaderboard_score is not None + has_self_eval = self_eval_score is not None and self_eval_samples > 0 + enough_samples = has_self_eval and self_eval_samples >= min_samples + + if has_leaderboard and enough_samples: + return ( + leaderboard_weight * leaderboard_score + + self_eval_weight * self_eval_score, + "blended", + ) + if enough_samples: + return self_eval_score, "self_eval" + if has_leaderboard: + return leaderboard_score, "leaderboard" + if has_self_eval: + return self_eval_score, "self_eval_thin" + return None, None + + +def accumulate( + previous_score: Optional[float], + previous_samples: int, + new_scores: list[float], +) -> tuple[Optional[float], int]: + """Fold a fresh eval run into a running mean. + + Keeps a running average rather than replacing, so ``self_eval_samples`` + means what the blending rule assumes it means: how much evidence stands + behind the score. Re-running the harness therefore tightens an estimate + instead of discarding everything learned before it. + + Returns ``(previous_score, previous_samples)`` unchanged when handed no + new scores, so a run where every task errored cannot quietly reset a + model's history to zero. + """ + if not new_scores: + return previous_score, previous_samples + + total_samples = previous_samples + len(new_scores) + if previous_score is None or previous_samples <= 0: + return sum(new_scores) / len(new_scores), len(new_scores) + + weighted = previous_score * previous_samples + sum(new_scores) + return weighted / total_samples, total_samples diff --git a/proficiency_store.py b/proficiency_store.py new file mode 100644 index 0000000..d5bfc3b --- /dev/null +++ b/proficiency_store.py @@ -0,0 +1,190 @@ +"""DB access for the ``proficiency`` table. + +Pairs with the pure ``proficiency.py`` the way ``tier.py`` pairs with +``tiering.py``: the blending arithmetic stays testable without a database, +and every write goes through here so ``blended_score`` and ``source`` can +never drift out of step with the two inputs that produced them. + +Both writers use this — ``leaderboard.py`` sets priors, ``eval_proficiency.py`` +folds in measured runs — and each write re-blends the row it touched. +""" + +from __future__ import annotations + +import sqlite3 +from datetime import datetime, timezone +from typing import Optional + +from config import RouterConfig +from proficiency import accumulate, blend + + +def _now() -> str: + return datetime.now(timezone.utc).isoformat() + + +def _read_row( + conn: sqlite3.Connection, model_id: str, provider: str, category: str +) -> Optional[sqlite3.Row]: + conn.row_factory = sqlite3.Row + return conn.execute( + """ + SELECT leaderboard_score, self_eval_score, self_eval_samples + FROM proficiency + WHERE model_id = ? AND provider = ? AND category = ? + """, + (model_id, provider, category), + ).fetchone() + + +def _write( + conn: sqlite3.Connection, + cfg: RouterConfig, + model_id: str, + provider: str, + category: str, + leaderboard_score: Optional[float], + self_eval_score: Optional[float], + self_eval_samples: int, +) -> None: + blended, source = blend( + leaderboard_score, + self_eval_score, + self_eval_samples, + leaderboard_weight=cfg.proficiency.leaderboard_weight, + self_eval_weight=cfg.proficiency.self_eval_weight, + min_samples=cfg.proficiency.self_eval_min_samples, + ) + conn.execute( + """ + INSERT INTO proficiency ( + model_id, provider, category, leaderboard_score, + self_eval_score, self_eval_samples, blended_score, source, last_updated + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(model_id, provider, category) DO UPDATE SET + leaderboard_score = excluded.leaderboard_score, + self_eval_score = excluded.self_eval_score, + self_eval_samples = excluded.self_eval_samples, + blended_score = excluded.blended_score, + source = excluded.source, + last_updated = excluded.last_updated + """, + ( + model_id, + provider, + category, + leaderboard_score, + self_eval_score, + self_eval_samples, + blended, + source, + _now(), + ), + ) + + +def set_leaderboard( + conn: sqlite3.Connection, + cfg: RouterConfig, + model_id: str, + provider: str, + category: str, + score: Optional[float], +) -> None: + """Set a category's leaderboard prior, preserving any self-eval history.""" + existing = _read_row(conn, model_id, provider, category) + _write( + conn, + cfg, + model_id, + provider, + category, + score, + existing["self_eval_score"] if existing else None, + existing["self_eval_samples"] if existing else 0, + ) + + +def add_self_eval( + conn: sqlite3.Connection, + cfg: RouterConfig, + model_id: str, + provider: str, + category: str, + scores: list[float], +) -> None: + """Fold an eval run's per-task scores into the running self-eval mean. + + Accumulates rather than replaces, so re-running the harness tightens the + estimate instead of discarding what came before — and so + ``self_eval_samples`` keeps meaning "how much evidence stands behind + this", which is what the blending threshold is gating on. + """ + existing = _read_row(conn, model_id, provider, category) + prev_score = existing["self_eval_score"] if existing else None + prev_samples = existing["self_eval_samples"] if existing else 0 + + new_score, new_samples = accumulate(prev_score, prev_samples, scores) + _write( + conn, + cfg, + model_id, + provider, + category, + existing["leaderboard_score"] if existing else None, + new_score, + new_samples, + ) + + +def propagate_to_variants( + conn: sqlite3.Connection, cfg: RouterConfig, base_model_id: str, provider: str +) -> int: + """Copy a family's scores onto serving variants that have none of their own. + + A ``-flex`` row is the same weights on a different queue, so its answer + quality is its family's. A ``-fast`` row is not — reasoning is off or + capped — so if the eval runner measured one directly, that row already + has its own self_eval_samples and is left alone. Inheritance is the + fallback, never an overwrite. + + Returns the number of rows written. + """ + conn.row_factory = sqlite3.Row + source_rows = conn.execute( + """ + SELECT category, leaderboard_score, self_eval_score, self_eval_samples + FROM proficiency + WHERE model_id = ? AND provider = ? + """, + (base_model_id, provider), + ).fetchall() + if not source_rows: + return 0 + + variants = conn.execute( + """ + SELECT model_id FROM models + WHERE base_model_id = ? AND provider = ? AND model_id != ? + """, + (base_model_id, provider, base_model_id), + ).fetchall() + + written = 0 + for variant in variants: + for row in source_rows: + existing = _read_row(conn, variant["model_id"], provider, row["category"]) + if existing and (existing["self_eval_samples"] or 0) > 0: + continue # measured directly; do not overwrite with the family's + _write( + conn, + cfg, + variant["model_id"], + provider, + row["category"], + row["leaderboard_score"], + row["self_eval_score"], + row["self_eval_samples"], + ) + written += 1 + return written diff --git a/schema.sql b/schema.sql index d59853f..0098d7c 100644 --- a/schema.sql +++ b/schema.sql @@ -7,6 +7,11 @@ PRAGMA foreign_keys = ON; CREATE TABLE IF NOT EXISTS models ( model_id TEXT NOT NULL, provider TEXT NOT NULL, -- 'neuralwatt' (only provider today) + -- The model family under the serving suffixes: glm-5.2-short-fast-flex + -- and glm-5.2 share one. Proficiency and leaderboard priors are properties + -- of the weights, not the queue, so both key on this and every variant + -- inherits from its family. + base_model_id TEXT, display_name TEXT, cost_per_1m_prompt REAL, -- USD, null if pricing_tbd cost_per_1m_completion REAL, diff --git a/tests/test_eval_scoring.py b/tests/test_eval_scoring.py new file mode 100644 index 0000000..0768e5e --- /dev/null +++ b/tests/test_eval_scoring.py @@ -0,0 +1,209 @@ +"""Tests for eval_proficiency.py's scorers. + +These decide what every proficiency number means, so the cases that matter +are the ones where a model did something *nearly* right — partial credit, +stray formatting, reasoning aloud before answering. Scoring those as zero +would make the whole table measure instruction-following rather than +competence. + +No network: the code scorer really does execute a subprocess, which is the +point of testing it. +""" + +import pytest + +from eval_proficiency import ( + extract_judge_json, + judge_for, + normalize_answer, + score_code, + score_exact, + score_tool, + strip_fences, +) + + +# --- code ----------------------------------------------------------------- + +def test_all_checks_passing_scores_one(): + assert score_code("def f(x):\n return x * 2", ["f(2)==4", "f(0)==0"])[0] == 1.0 + + +def test_partial_credit_is_proportional(): + # A function right on one of two cases is genuinely better than one that + # fails both, and a binary score would throw that distinction away + score, detail = score_code( + "def f(x):\n return x * 2 if x else 99", ["f(2)==4", "f(0)==0"] + ) + assert score == 0.5 + assert "1/2" in detail + + +def test_code_that_does_not_parse_scores_zero_and_says_so(): + score, detail = score_code("def f(x)\n return", ["f(1)==1"]) + assert score == 0.0 + # Distinguishable from "ran but failed the cases", which matters when + # reading results — one is a formatting problem, the other is competence + assert "did not run" in detail + + +def test_infinite_loop_is_bounded_not_hung(): + score, detail = score_code("def f(x):\n while True: pass", ["f(1)==1"]) + assert (score, detail) == (0.0, "timeout") + + +def test_markdown_fences_do_not_cost_the_model_the_task(): + # Every code prompt says "no fences" and models add them anyway. Scoring + # that zero would measure instruction-following, not coding. + fenced = "```python\ndef f(x):\n return x * 2\n```" + assert score_code(fenced, ["f(2)==4"])[0] == 1.0 + + +def test_raises_helper_is_available_to_checks(): + # Checks express "should raise" through a helper the harness injects, so + # the sandbox needs no test framework of its own + code = "def g(n):\n if n < 1: raise ValueError()\n return n" + assert score_code(code, ["raises(ValueError, g, 0)", "g(3)==3"])[0] == 1.0 + + +def test_common_imports_are_available(): + # Ordinary solutions reach for re/math without importing them explicitly + code = "def f(s):\n return re.findall(r'\\d+', s)" + assert score_code(code, ["f('a1b22') == ['1','22']"])[0] == 1.0 + + +def test_a_task_with_no_checks_cannot_score_credit(): + assert score_code("def f(): pass", [])[0] == 0.0 + + +# --- exact ---------------------------------------------------------------- + +def test_exact_ignores_surrounding_prose(): + assert score_exact("The answer is 36.", "36")[0] == 1.0 + + +def test_exact_ignores_thousands_separators(): + assert score_exact("5,400", "5400")[0] == 1.0 + + +def test_exact_takes_the_last_number_as_the_conclusion(): + # A model that reasons aloud states intermediate values first; the answer + # is what it ends on + assert score_exact("1/60 + 1/90 = 1/36, so 36 minutes", "36")[0] == 1.0 + + +def test_exact_wrong_answer_scores_zero_and_reports_both(): + score, detail = score_exact("42", "36") + assert score == 0.0 + assert "42" in detail and "36" in detail + + +def test_trailing_zeros_do_not_break_equality(): + assert normalize_answer("36.0") == normalize_answer("36") + + +# --- tool use ------------------------------------------------------------- + +def _call(name, arguments="{}"): + return [{"function": {"name": name, "arguments": arguments}}] + + +def test_right_tool_and_arguments_scores_one(): + calls = _call("get_weather", '{"location": "Reykjavik, Iceland"}') + task = {"expect_tool": "get_weather", "expect_args": {"location": "reykjavik"}} + assert score_tool(calls, task)[0] == 1.0 + + +def test_right_tool_wrong_arguments_keeps_half_credit(): + # Choosing correctly between tools is most of the skill; the arguments + # are the rest + calls = _call("get_weather", '{"location": "Paris"}') + task = {"expect_tool": "get_weather", "expect_args": {"location": "reykjavik"}} + assert score_tool(calls, task)[0] == 0.5 + + +def test_wrong_tool_scores_zero(): + task = {"expect_tool": "convert_currency"} + assert score_tool(_call("get_weather"), task)[0] == 0.0 + + +def test_no_call_when_one_was_needed_scores_zero(): + assert score_tool([], {"expect_tool": "get_weather"})[0] == 0.0 + + +def test_abstaining_when_no_tool_applies_is_the_correct_answer(): + # A model that reaches for a tool on every prompt is a real failure mode + # in an agent loop, so not calling one is scored, not just tolerated + assert score_tool([], {"expect_tool": None})[0] == 1.0 + + +def test_calling_a_tool_when_none_applied_scores_zero(): + score, detail = score_tool(_call("get_weather"), {"expect_tool": None}) + assert score == 0.0 + assert "when none applied" in detail + + +def test_unparseable_arguments_keep_the_tool_choice_credit(): + calls = _call("get_weather", "{not json") + task = {"expect_tool": "get_weather", "expect_args": {"location": "x"}} + assert score_tool(calls, task)[0] == 0.5 + + +# --- fence stripping ------------------------------------------------------ + +def test_strip_fences_leaves_bare_code_alone(): + assert strip_fences("def f(): pass") == "def f(): pass" + + +def test_strip_fences_handles_an_unlabelled_block(): + assert strip_fences("```\ndef f(): pass\n```").strip() == "def f(): pass" + + +# --- judge robustness ----------------------------------------------------- + +def test_judge_json_parses_a_bare_object(): + assert extract_judge_json('{"score": 0.8, "reason": "ok"}')["score"] == 0.8 + + +def test_judge_json_survives_a_thinking_preamble(): + # Judges are reasoning models and leak their thinking into the content + # despite response_format. This was 44% of judge calls on the first run. + raw = 'Let me check the rubric:\n\n1. Distinguishes...\n\n{"score": 0.6, "reason": "partial"}' + assert extract_judge_json(raw)["score"] == 0.6 + + +def test_truncated_judge_output_is_a_failure_not_a_guess(): + # A cut-off object must not be salvaged into a number — better no sample + assert extract_judge_json('{"score": 0.8, "reason": "Accurate with') is None + + +def test_judge_prose_without_json_is_a_failure(): + assert extract_judge_json("I think it is pretty good actually") is None + assert extract_judge_json("") is None + + +def test_object_without_a_score_is_not_a_verdict(): + assert extract_judge_json('{"reason": "forgot the score"}') is None + + +def test_a_model_never_judges_its_own_family(): + # Self-scoring is a known bias and the default judge is itself in the + # evaluated set, so its own family gets an alternate + assert judge_for("kimi-k3", "kimi-k3") != "kimi-k3" + assert judge_for("kimi-k3-fast", "kimi-k3") != "kimi-k3" + assert judge_for("kimi-k3-flex", "kimi-k3") != "kimi-k3" + # Everyone else is judged by the default + assert judge_for("gemma-4-31b", "kimi-k3") == "kimi-k3" + + +def test_a_single_leading_space_does_not_destroy_a_correct_answer(): + # Observed: kimi-k2.7-code returns " def f(...)" with one leading space, + # which is an IndentationError once the harness prepends its imports. It + # scored 0.00 on every coding task for this and nothing else. + code = ' def f(x):\n return x * 2' + assert score_code(code, ["f(2)==4"])[0] == 1.0 + + +def test_indented_fenced_block_is_still_recovered(): + code = ' ```python\ndef f(x):\n return x * 2\n```' + assert score_code(code, ["f(2)==4"])[0] == 1.0 diff --git a/tests/test_poller_parsing.py b/tests/test_poller_parsing.py index 0a1dfc3..bcffd61 100644 --- a/tests/test_poller_parsing.py +++ b/tests/test_poller_parsing.py @@ -7,7 +7,7 @@ Cases below are taken from the live catalog. import pytest -from poller import parse_access_level, parse_serving_class +from poller import parse_access_level, parse_base_model_id, parse_serving_class # --- serving class -------------------------------------------------------- @@ -76,3 +76,32 @@ def test_parse_access_level(display_name, description, expected): def test_preview_takes_precedence_over_canary(): # Given: prose mentioning both, the more restrictive label wins assert parse_access_level("x", "Private preview (grant-gated) canary") == "preview" + + +# --- model family --------------------------------------------------------- + +@pytest.mark.parametrize( + "model_id,expected", + [ + ("gemma-4-31b", "gemma-4-31b"), + ("kimi-k3", "kimi-k3"), + ("kimi-k3-fast", "kimi-k3"), + ("kimi-k3-flex", "kimi-k3"), + # every serving dimension collapses, in any combination + ("glm-5.2-short-fast-flex", "glm-5.2"), + ("glm-5.2-short", "glm-5.2"), + # the HF-style duplicate row folds into the same family + ("deepseek-ai/DeepSeek-V4-Flash", "deepseek-v4-flash"), + # ...without 'flash' being mistaken for the '-fast' suffix + ("deepseek-v4-flash", "deepseek-v4-flash"), + ("deepseek-v4-flash-flex", "deepseek-v4-flash"), + ], +) +def test_parse_base_model_id(model_id, expected): + assert parse_base_model_id(model_id) == expected + + +def test_family_never_collapses_to_empty(): + # A degenerate id made only of suffix tokens must keep a segment, or every + # such row would share one meaningless family + assert parse_base_model_id("flex") == "flex" diff --git a/tests/test_proficiency.py b/tests/test_proficiency.py new file mode 100644 index 0000000..896a2cd --- /dev/null +++ b/tests/test_proficiency.py @@ -0,0 +1,100 @@ +"""Tests for proficiency.py — blending leaderboard priors with self-eval. + +The interesting cases are the ones the design doc's one-line rule does not +cover: what happens when a model has no leaderboard prior (common, because +NeuralWatt ships models faster than benchmarks cover them) and when a run +produces no usable scores at all. +""" + +import pytest + +from proficiency import accumulate, blend + +BLEND_KW = {"leaderboard_weight": 0.3, "self_eval_weight": 0.7, "min_samples": 10} + + +def _blend(lb, se, n, **over): + return blend(lb, se, n, **{**BLEND_KW, **over}) + + +# --- the documented rule -------------------------------------------------- + +def test_blends_both_sources_once_samples_suffice(): + score, source = _blend(0.4, 0.9, 10) + assert score == pytest.approx(0.3 * 0.4 + 0.7 * 0.9) + assert source == "blended" + + +def test_leaderboard_alone_below_the_sample_threshold(): + # Thin self-eval must not dominate a published benchmark early + score, source = _blend(0.4, 0.9, 9) + assert (score, source) == (0.4, "leaderboard") + + +def test_threshold_is_inclusive(): + assert _blend(0.4, 0.9, 10)[1] == "blended" + assert _blend(0.4, 0.9, 9)[1] == "leaderboard" + + +# --- no leaderboard prior, which is the common case ----------------------- + +def test_self_eval_alone_when_no_prior_exists(): + # Given: a model NeuralWatt added that no public benchmark covers yet. + # Read literally, the design doc's rule would fall back to a leaderboard + # score that does not exist and yield nothing — discarding real evidence. + score, source = _blend(None, 0.8, 10) + assert (score, source) == (0.8, "self_eval") + + +def test_thin_self_eval_still_beats_nothing(): + # Nine real samples are worse than twelve, but far better than the + # neutral 0.5 a model gets for being unmeasured. Flagged so a caller can + # tell it apart from a score that cleared the threshold. + score, source = _blend(None, 0.8, 9) + assert (score, source) == (0.8, "self_eval_thin") + + +def test_unmeasured_model_returns_none_not_zero(): + # None leaves the candidate on the neutral 0.5 downstream. Zero would + # rank it below every measured model for the crime of being new. + assert _blend(None, None, 0) == (None, None) + + +def test_zero_samples_is_not_evidence(): + # A score with no samples behind it is a leftover, not a measurement + assert _blend(None, 0.9, 0) == (None, None) + assert _blend(0.4, 0.9, 0) == (0.4, "leaderboard") + + +def test_a_genuine_zero_score_is_kept(): + # 0.0 means "measured, and it failed everything" — distinct from unmeasured + score, source = _blend(None, 0.0, 12) + assert (score, source) == (0.0, "self_eval") + + +# --- accumulation --------------------------------------------------------- + +def test_first_run_sets_the_mean(): + assert accumulate(None, 0, [1.0, 0.5, 0.0]) == (0.5, 3) + + +def test_later_runs_tighten_rather_than_replace(): + # Given: 0.8 over 10 samples, then a run of 2 perfect scores + score, samples = accumulate(0.8, 10, [1.0, 1.0]) + assert samples == 12 + assert score == pytest.approx((0.8 * 10 + 2.0) / 12) + + +def test_an_all_errors_run_changes_nothing(): + # Given: every task errored, so there are no scores. Resetting a model's + # history to zero on a bad run would silently erase months of evidence. + assert accumulate(0.8, 10, []) == (0.8, 10) + + +def test_no_history_and_no_scores_stays_empty(): + assert accumulate(None, 0, []) == (None, 0) + + +def test_stale_score_with_zero_samples_is_overwritten(): + # A score with no samples behind it carries no weight in the average + assert accumulate(0.9, 0, [0.1, 0.3]) == (pytest.approx(0.2), 2) diff --git a/tests/test_task_set.py b/tests/test_task_set.py new file mode 100644 index 0000000..c06a789 --- /dev/null +++ b/tests/test_task_set.py @@ -0,0 +1,232 @@ +"""Validate the eval task set against reference solutions. + +A check the task author cannot satisfy is a broken check, and it scores the +task set rather than the model. That has already happened here: a sequence +task asked for "the next number" and "the 11th term" in one breath, and every +model that read it correctly scored zero. + +So every `code` task gets a reference implementation below, and every check +must pass against it. Every `exact` answer is recomputed rather than trusted. +These tests run offline and take milliseconds — they are the cheap guard +against spending an hour of API calls measuring a typo. +""" + +from pathlib import Path + +import pytest +import yaml + +from eval_proficiency import score_code, score_exact + +TASKS = yaml.safe_load((Path(__file__).resolve().parent.parent / "evals" / "tasks.yaml").read_text())["tasks"] +BY_ID = {t["id"]: t for t in TASKS} + + +# --- reference solutions -------------------------------------------------- + +REFERENCES = { + "merge_intervals": ''' +def merge_intervals(intervals): + if not intervals: + return [] + ordered = sorted(intervals, key=lambda iv: iv[0]) + out = [list(ordered[0])] + for start, end in ordered[1:]: + if start <= out[-1][1]: + out[-1][1] = max(out[-1][1], end) + else: + out.append([start, end]) + return out +''', + "parse_semver": r''' +import re +_SEMVER = re.compile( + r"^(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)" + r"(?:-(?P(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)" + r"(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?" + r"(?:\+(?P[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$" +) + +def parse_semver(version): + m = _SEMVER.match(version or "") + if not m: + raise ValueError(version) + d = m.groupdict() + return { + "major": int(d["major"]), + "minor": int(d["minor"]), + "patch": int(d["patch"]), + "prerelease": d["prerelease"], + "build": d["build"], + } +''', + "word_wrap": ''' +def word_wrap(text, width): + words = text.split() + if not words: + return [] + lines, current = [], words[0] + for word in words[1:]: + if len(current) + 1 + len(word) <= width: + current += " " + word + else: + lines.append(current) + current = word + lines.append(current) + return lines +''', + "refactor_falsy_defaults": ''' +_DEFAULTS = {"retries": 3, "timeout": 30, "verbose": False} + +def apply_settings(overrides): + return {k: overrides.get(k, v) for k, v in _DEFAULTS.items()} +''', + "refactor_first_match": ''' +def first_match(items, predicates): + return next( + (item for item in items if any(p(item) for p in predicates)), None + ) +''', + "refactor_dispatch": ''' +_TABLE = {200: "ok", 201: "created", 404: "not found", 500: "server error"} + +def describe(code): + return _TABLE.get(code, "unknown") +''', + "debug_late_binding": ''' +def make_multipliers(factors): + return [lambda x, f=f: x * f for f in factors] +''', + "debug_binary_search": ''' +def bsearch(items, target): + lo, hi = 0, len(items) + while lo < hi: + mid = (lo + hi) // 2 + if items[mid] == target: + return mid + elif items[mid] < target: + lo = mid + 1 + else: + hi = mid + return -1 +''', + "debug_greedy_regex": ''' +import re + +def extract_tags(text): + return re.findall(r"<([^<>]+)>", text) +''', +} + + +@pytest.mark.parametrize("task_id", sorted(REFERENCES)) +def test_reference_solution_passes_every_check(task_id): + task = BY_ID[task_id] + score, detail = score_code(REFERENCES[task_id], task["checks"]) + assert score == 1.0, f"{task_id}: reference scored {score} ({detail})" + + +def test_every_code_task_has_a_reference(): + # A code task with no reference has never been validated, so its checks + # could be wrong in exactly the way that costs an hour of API calls + code_tasks = {t["id"] for t in TASKS if t["kind"] == "code"} + assert code_tasks == set(REFERENCES), ( + f"unvalidated: {sorted(code_tasks - set(REFERENCES))}" + ) + + +# --- the buggy code really is buggy --------------------------------------- + +BUGGY = { + "refactor_first_match": ''' +def first_match(items, predicates): + found = None + done = False + for item in items: + if done: + break + for p in predicates: + if p(item): + found = item + done = True + break + return found +''', + "debug_late_binding": ''' +def make_multipliers(factors): + out = [] + for f in factors: + out.append(lambda x: x * f) + return out +''', + "debug_greedy_regex": ''' +import re + +def extract_tags(text): + return re.findall(r"<(.+)>", text) +''', +} + + +def test_refactor_target_already_passes_its_own_checks(): + # A refactor task's ORIGINAL code must pass, or the task is secretly a + # debugging task and "behaviour must not change" is a lie + score, detail = score_code(BUGGY["refactor_first_match"], BY_ID["refactor_first_match"]["checks"]) + assert score == 1.0, f"refactor target fails its own checks: {detail}" + + +@pytest.mark.parametrize("task_id", ["debug_late_binding", "debug_greedy_regex"]) +def test_debugging_tasks_start_broken(task_id): + # The whole point is that the given code fails. If it passes, the task + # measures nothing — a model could return the input unchanged. + score, _ = score_code(BUGGY[task_id], BY_ID[task_id]["checks"]) + assert score < 1.0 + + +# --- exact answers, recomputed -------------------------------------------- + +def test_percent_trap_answer(): + # 20% up then 20% down is a 4% net loss, not a wash + original = 96 / (1.20 * 0.80) + assert score_exact(str(original), BY_ID["math_percent_trap"]["answer"])[0] == 1.0 + + +def test_rate_trap_answer(): + # 3 machines / 3 widgets / 3 min => one machine makes one widget in 3 min, + # so 100 machines make 100 widgets in the same 3 minutes + per_machine_minutes = 3 + assert score_exact(str(per_machine_minutes), BY_ID["math_rate_trap"]["answer"])[0] == 1.0 + + +def test_counting_answer(): + # first digit 9 choices (not 0), then 9, 8, 7 for distinctness + assert score_exact(str(9 * 9 * 8 * 7), BY_ID["math_counting"]["answer"])[0] == 1.0 + + +def test_counting_answer_matches_brute_force(): + count = sum( + 1 for n in range(1000, 10000) if len(set(str(n))) == 4 + ) + assert str(count) == BY_ID["math_counting"]["answer"] + + +# --- task set hygiene ----------------------------------------------------- + +def test_every_task_has_the_fields_its_kind_needs(): + for task in TASKS: + kind = task["kind"] + assert task.get("prompt"), f"{task['id']} has no prompt" + if kind == "code": + assert task.get("checks"), f"{task['id']} has no checks" + elif kind == "exact": + assert task.get("answer") is not None, f"{task['id']} has no answer" + elif kind == "judge": + assert task.get("rubric"), f"{task['id']} has no rubric" + elif kind == "tool": + assert "expect_tool" in task, f"{task['id']} has no expect_tool" + assert task.get("tools"), f"{task['id']} has no tools" + + +def test_task_ids_are_unique(): + ids = [t["id"] for t in TASKS] + assert len(ids) == len(set(ids)) -- 2.49.1 From 2a6b48449cb7a7d59a5a558072ec35b9f961c631 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 18:21:26 -0400 Subject: [PATCH 07/20] fix: restore energy data, surface empty scoring axes, and sample across time Asking whether the service was running current code turned up three problems it was not. 1. energy_observations was EMPTY. Recreating the DB for the base_model_id schema change dropped every reference-workload row, so cost_score and eco_score had silently fallen back to the neutral 0.5. The 7-distinct-model routing result reported earlier was therefore proficiency ALONE, not proficiency plus measured cost/eco. Re-swept. 2. The units would not have survived a reboot. Lingering was never enabled, so the user manager stops at logout. Enabled, and moved into the documented install steps rather than a footnote. 3. glm-5.2-flex had no proficiency at all, from a real bug. Propagation copied from the row whose id equals base_model_id -- glm-5.2 -- which is canary and never evaluated. Matching was also on family alone, so a '-fast' row could inherit its reasoning-ENABLED sibling's scores. Inheritance now requires matching (base_model_id, reasoning_mode, context_variant), and a flex row with no evaluated equivalent is measured directly instead of scoring blank. /health now reports scoring coverage and warns when an axis has no data. An empty axis is not an error -- every candidate takes 0.5 and routing still works -- which is exactly what makes it dangerous: a 0.4 weight can contribute nothing while health says "ok". It has now happened twice, so it is surfaced rather than inferred. Adds llm-router-seed.timer. Attribution reproduces within about 30 minutes (0.3-1.1x on a spot-check) but not across hours: between two sweeps deepseek-v4-flash moved ~50x and qwen3.6-35b ~7x the other way, enough to invert their cost ranking. More samples inside one sweep measures one moment more precisely; coverage across time is what actually helps. load_candidates already medians over all seed_reference rows, so a periodic small sweep turns that into a median-across-time for free. Until several sweeps accumulate, the cost and eco ordering is provisional -- a single sweep's ranking is one sample of a moving quantity. Tests 153 -> 156. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 17 ++++++++ deploy/README.md | 18 ++++---- deploy/llm-router-seed.service | 19 ++++++++ deploy/llm-router-seed.timer | 10 +++++ dispatcher.py | 73 +++++++++++++++++++++++++++++++ eval_proficiency.py | 40 +++++++++++++---- proficiency_store.py | 32 +++++++++----- tests/test_proficiency.py | 79 ++++++++++++++++++++++++++++++++++ 8 files changed, 260 insertions(+), 28 deletions(-) create mode 100644 deploy/llm-router-seed.service create mode 100644 deploy/llm-router-seed.timer diff --git a/CLAUDE.md b/CLAUDE.md index a007c1b..28c9949 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -125,6 +125,23 @@ three vs last four) shows that working: `dispatcher.gross_energy_kwh` remains as a diagnostic on the identity, not a scoring input. +### Attribution drifts across hours, so sampling must too + +Within about 30 minutes the billed figures reproduce (0.3-1.1x on a +spot-check). Across hours they do not: between two sweeps, +`deepseek-v4-flash` moved roughly 50x and `qwen3.6-35b` about 7x the other +way — enough to **invert their cost ranking**. Attribution tracks pool load, +and pool load tracks time of day. + +More samples inside one sweep does not fix this; it measures one moment more +precisely. Coverage across time does. `load_candidates` already takes the +median over ALL `seed_reference` rows, so repeated sweeps accumulate into a +median-across-time for free — hence `llm-router-seed.timer`, which runs a +small sweep every 6 hours. + +Until several sweeps have accumulated, treat the cost and eco ordering as +provisional. A single sweep's ranking is one sample of a moving quantity. + ## What's built and working - `schema.sql` — `models`, `proficiency`, `energy_observations`. Applies diff --git a/deploy/README.md b/deploy/README.md index 185892c..f7e5a5f 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -11,6 +11,8 @@ all. | `llm-router.service` | the FastAPI dispatcher, on `127.0.0.1:8080` | | `llm-router-poller.service` | one-shot: `poller.py` then `tier.py` | | `llm-router-poller.timer` | fires the poller 2 min after boot, then every 2 h | +| `llm-router-seed.service` | one-shot: a small `seed_energy.py` reference sweep | +| `llm-router-seed.timer` | every 6 h — energy attribution drifts with pool load across hours, so the median has to span time rather than one sweep | These are **user** units — no root, and they run as you with your own `$HOME`. The tradeoff is that a user service does not inherit your shell @@ -26,17 +28,15 @@ echo "NEURALWATT_API_KEY=$NEURALWATT_API_KEY" > .env && chmod 600 .env # 2. Install and start cp deploy/llm-router*.{service,timer} ~/.config/systemd/user/ systemctl --user daemon-reload -systemctl --user enable --now llm-router.service llm-router-poller.timer +systemctl --user enable --now llm-router.service llm-router-poller.timer llm-router-seed.timer -# 3. Check -curl -s localhost:8080/health | python -m json.tool -systemctl --user list-timers llm-router-poller.timer -``` - -To survive logout without an active session, enable lingering: - -```bash +# 3. Survive logout/reboot (user units stop with your session otherwise) loginctl enable-linger "$USER" + +# 4. Check. The health endpoint reports whether cost/eco/proficiency +# actually have data behind them, which is otherwise silent. +curl -s localhost:8080/health | python -m json.tool +systemctl --user list-timers 'llm-router*' ``` ## Operating it diff --git a/deploy/llm-router-seed.service b/deploy/llm-router-seed.service new file mode 100644 index 0000000..a8058db --- /dev/null +++ b/deploy/llm-router-seed.service @@ -0,0 +1,19 @@ +[Unit] +Description=Sample the reference workload to keep cost/eco scoring current +Documentation=file:%h/Sources/6krrt/CLAUDE.md +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +WorkingDirectory=%h/Sources/6krrt +EnvironmentFile=%h/Sources/6krrt/.env +# Fewer samples per run than a manual sweep, because the point is coverage +# across TIME rather than depth at one moment — see the timer. +ExecStart=%h/Sources/6krrt/.venv/bin/python seed_energy.py --samples 3 + +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ProtectHome=read-only +ReadWritePaths=%h/Sources/6krrt diff --git a/deploy/llm-router-seed.timer b/deploy/llm-router-seed.timer new file mode 100644 index 0000000..33a60c5 --- /dev/null +++ b/deploy/llm-router-seed.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Sample the reference workload every few hours + +[Timer] +OnBootSec=15min +OnUnitActiveSec=6h +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/dispatcher.py b/dispatcher.py index 6c9426f..285c96e 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -574,6 +574,7 @@ def health(): return { "status": "ok", "counts": counts, + "scoring": scoring_coverage(), "classifier_reachable": classifier_ok, "classifier_model": cfg.classifier.model, "providers": list(cfg.dispatch_providers), @@ -584,6 +585,78 @@ def health(): } +def scoring_coverage() -> dict: + """Report which scoring axes actually have data behind them. + + An axis with no data is not an error — every candidate takes the neutral + 0.5 and the router still works. It is worse than an error: it is silent. + A weight of 0.4 can be contributing exactly nothing while `/health` + cheerfully says "ok", and the only symptom is that routing stops + discriminating in a way nobody notices. + + This has already happened twice: recreating the DB for a schema change + dropped every reference-workload observation, and cost and eco scored + neutral for days afterwards. So the state is surfaced rather than + inferred. + """ + conn = _db() + try: + placeholders = ",".join("?" * len(cfg.routing.allowed_access_levels)) + routable = [ + (r["model_id"], r["provider"]) + for r in conn.execute( + f""" + SELECT model_id, provider FROM models + WHERE availability = 'active' + AND access_level IN ({placeholders}) + """, + tuple(cfg.routing.allowed_access_levels), + ) + ] + with_energy = { + (r["model_id"], r["provider"]) + for r in conn.execute( + "SELECT DISTINCT model_id, provider FROM energy_observations " + "WHERE task_category = ?", + (SEED_CATEGORY,), + ) + } + with_proficiency = { + (r["model_id"], r["provider"]) + for r in conn.execute( + "SELECT DISTINCT model_id, provider FROM proficiency " + "WHERE blended_score IS NOT NULL" + ) + } + finally: + conn.close() + + total = len(routable) + missing_energy = [m for m, p in routable if (m, p) not in with_energy] + missing_proficiency = [m for m, p in routable if (m, p) not in with_proficiency] + + warnings = [] + if missing_energy: + warnings.append( + f"{len(missing_energy)}/{total} routable models have no reference-workload " + f"observations — cost and eco score the neutral 0.5 for them. " + f"Run: python seed_energy.py --samples 7" + ) + if missing_proficiency: + warnings.append( + f"{len(missing_proficiency)}/{total} routable models have no proficiency " + f"data — task_category cannot influence their ranking. " + f"Run: python eval_proficiency.py" + ) + + return { + "routable_models": total, + "with_energy_data": total - len(missing_energy), + "with_proficiency_data": total - len(missing_proficiency), + "warnings": warnings, + } + + @app.post("/route", response_model=RouteResponse) def route_endpoint(req: TaskRequest): """Classify and pick a model without calling it.""" diff --git a/eval_proficiency.py b/eval_proficiency.py index 7297298..ef7c206 100644 --- a/eval_proficiency.py +++ b/eval_proficiency.py @@ -331,21 +331,41 @@ def score_judge( def eval_identities(conn: sqlite3.Connection, cfg: RouterConfig) -> list[dict]: """Model rows worth measuring directly. - Flex rows are excluded: same weights, same reasoning setting, different - queue — their answer quality is their family's, and they inherit it via + Flex rows are normally excluded: same weights, same reasoning setting, + different queue — their quality is the standard row's, inherited via ``propagate_to_variants``. ``-fast`` rows ARE measured, because reasoning being off genuinely changes answers. + + But that exclusion assumes an evaluated standard equivalent exists, and + it does not always: glm-5.2-flex reasons by default, while the only + routable standard glm row is glm-5.2-fast (reasoning reduced) — the + matching glm-5.2 is canary and therefore never evaluated. Such orphans + are measured directly rather than left with no proficiency at all. """ placeholders = ",".join("?" * len(cfg.routing.allowed_access_levels)) rows = conn.execute( f""" - SELECT model_id, base_model_id, reasoning_mode, max_output_tokens FROM models + SELECT model_id, base_model_id, reasoning_mode, max_output_tokens FROM models m WHERE availability = 'active' AND access_level IN ({placeholders}) - AND latency_class = 'standard' + AND ( + latency_class = 'standard' + -- ...or a flex row with no routable standard equivalent to + -- inherit from, which would otherwise score nothing at all + OR NOT EXISTS ( + SELECT 1 FROM models s + WHERE s.base_model_id = m.base_model_id + AND s.provider = m.provider + AND s.reasoning_mode = m.reasoning_mode + AND s.context_variant = m.context_variant + AND s.latency_class = 'standard' + AND s.availability = 'active' + AND s.access_level IN ({placeholders}) + ) + ) ORDER BY model_id """, - tuple(cfg.routing.allowed_access_levels), + tuple(cfg.routing.allowed_access_levels) * 2, ).fetchall() return [ { @@ -489,10 +509,14 @@ def main() -> int: add_self_eval(conn, cfg, model_id, "neuralwatt", category, scores) conn.commit() - # Flex siblings inherit; anything measured directly keeps its own score. + # Equivalent serving variants inherit from the row actually measured — + # not from the family id, which may name a row that was never evaluated + # (glm-5.2 is canary, so glm-5.2-flex inherited nothing and scored blank). propagated = 0 - for base in {i["base_model_id"] for i in identities}: - propagated += propagate_to_variants(conn, cfg, base, "neuralwatt") + for identity in identities: + propagated += propagate_to_variants( + conn, cfg, identity["model_id"], "neuralwatt" + ) conn.commit() print(f"\npropagated {propagated} inherited rows to serving variants") conn.close() diff --git a/proficiency_store.py b/proficiency_store.py index d5bfc3b..a33c966 100644 --- a/proficiency_store.py +++ b/proficiency_store.py @@ -138,15 +138,20 @@ def add_self_eval( def propagate_to_variants( - conn: sqlite3.Connection, cfg: RouterConfig, base_model_id: str, provider: str + conn: sqlite3.Connection, cfg: RouterConfig, source_model_id: str, provider: str ) -> int: - """Copy a family's scores onto serving variants that have none of their own. + """Copy an evaluated row's scores onto its equivalent serving variants. - A ``-flex`` row is the same weights on a different queue, so its answer - quality is its family's. A ``-fast`` row is not — reasoning is off or - capped — so if the eval runner measured one directly, that row already - has its own self_eval_samples and is left alone. Inheritance is the - fallback, never an overwrite. + A ``-flex`` row is the same weights, same reasoning setting, same context + pool, on a different queue — so its answer quality IS the standard row's + and inheriting is correct. A ``-fast`` row is NOT equivalent (reasoning + off or capped), and neither is a ``-short`` row, so matching is on + (base_model_id, reasoning_mode, context_variant) rather than on family + alone. Inheriting across those would attribute reasoning-on quality to a + reasoning-off row. + + Anything already measured directly keeps its own scores; inheritance is + the fallback, never an overwrite. Returns the number of rows written. """ @@ -157,17 +162,22 @@ def propagate_to_variants( FROM proficiency WHERE model_id = ? AND provider = ? """, - (base_model_id, provider), + (source_model_id, provider), ).fetchall() if not source_rows: return 0 variants = conn.execute( """ - SELECT model_id FROM models - WHERE base_model_id = ? AND provider = ? AND model_id != ? + SELECT v.model_id FROM models v + JOIN models src + ON src.base_model_id = v.base_model_id + AND src.provider = v.provider + AND src.reasoning_mode = v.reasoning_mode + AND src.context_variant = v.context_variant + WHERE src.model_id = ? AND v.provider = ? AND v.model_id != ? """, - (base_model_id, provider, base_model_id), + (source_model_id, provider, source_model_id), ).fetchall() written = 0 diff --git a/tests/test_proficiency.py b/tests/test_proficiency.py index 896a2cd..413b2ae 100644 --- a/tests/test_proficiency.py +++ b/tests/test_proficiency.py @@ -98,3 +98,82 @@ def test_no_history_and_no_scores_stays_empty(): def test_stale_score_with_zero_samples_is_overwritten(): # A score with no samples behind it carries no weight in the average assert accumulate(0.9, 0, [0.1, 0.3]) == (pytest.approx(0.2), 2) + + +# --- variant inheritance -------------------------------------------------- + +def _models_db(tmp_path, rows): + import sqlite3 + from pathlib import Path + schema = (Path(__file__).resolve().parent.parent / "schema.sql").read_text() + conn = sqlite3.connect(tmp_path / "t.db") + conn.row_factory = sqlite3.Row + conn.executescript(schema) + for model_id, base, latency, reasoning, ctx in rows: + conn.execute( + """ + INSERT INTO models (model_id, provider, base_model_id, latency_class, + reasoning_mode, context_variant, availability, last_updated) + VALUES (?, 'nw', ?, ?, ?, ?, 'active', '2026-08-17T00:00:00+00:00') + """, + (model_id, base, latency, reasoning, ctx), + ) + conn.commit() + return conn + + +def test_flex_inherits_from_its_standard_equivalent(tmp_path): + from config import load_config + from pathlib import Path + from proficiency_store import add_self_eval, propagate_to_variants + + cfg = load_config(Path(__file__).resolve().parent.parent / "config.yaml") + conn = _models_db(tmp_path, [ + ("kimi-k3", "kimi-k3", "standard", "default", "full"), + ("kimi-k3-flex", "kimi-k3", "flex", "default", "full"), + ]) + add_self_eval(conn, cfg, "kimi-k3", "nw", "coding_general", [1.0, 1.0]) + assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 1 + got = conn.execute( + "SELECT self_eval_score FROM proficiency WHERE model_id='kimi-k3-flex'" + ).fetchone() + assert got["self_eval_score"] == 1.0 + + +def test_fast_does_not_inherit_reasoning_on_quality(tmp_path): + # A '-fast' row runs with reasoning off or capped, so it is NOT the same + # model for quality purposes. Inheriting across that would credit it with + # its reasoning-enabled sibling's answers. + from config import load_config + from pathlib import Path + from proficiency_store import add_self_eval, propagate_to_variants + + cfg = load_config(Path(__file__).resolve().parent.parent / "config.yaml") + conn = _models_db(tmp_path, [ + ("kimi-k3", "kimi-k3", "standard", "default", "full"), + ("kimi-k3-fast", "kimi-k3", "standard", "reduced", "full"), + ]) + add_self_eval(conn, cfg, "kimi-k3", "nw", "reasoning_math", [1.0]) + assert propagate_to_variants(conn, cfg, "kimi-k3", "nw") == 0 + assert conn.execute( + "SELECT COUNT(*) c FROM proficiency WHERE model_id='kimi-k3-fast'" + ).fetchone()["c"] == 0 + + +def test_a_directly_measured_variant_is_never_overwritten(tmp_path): + from config import load_config + from pathlib import Path + from proficiency_store import add_self_eval, propagate_to_variants + + cfg = load_config(Path(__file__).resolve().parent.parent / "config.yaml") + conn = _models_db(tmp_path, [ + ("kimi-k3", "kimi-k3", "standard", "default", "full"), + ("kimi-k3-flex", "kimi-k3", "flex", "default", "full"), + ]) + add_self_eval(conn, cfg, "kimi-k3", "nw", "coding_general", [1.0]) + add_self_eval(conn, cfg, "kimi-k3-flex", "nw", "coding_general", [0.2]) + propagate_to_variants(conn, cfg, "kimi-k3", "nw") + got = conn.execute( + "SELECT self_eval_score FROM proficiency WHERE model_id='kimi-k3-flex'" + ).fetchone() + assert got["self_eval_score"] == 0.2 -- 2.49.1 From 82d06680f3a2e14120901c15ae53959aa656fc29 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 20:56:50 -0400 Subject: [PATCH 08/20] docs: record the routing survey with all three axes live With cost, eco and proficiency all populated, 27 routing decisions produce ONE model. That is not a regression: qwen3.6-35b is Pareto-dominant in the routable set -- cheapest and cleanest -- while scoring within 0.15 of the best proficiency. On summarization it costs 7x less and emits 65x less carbon than kimi-k3, which beats it 1.00 to 0.85 on quality; no defensible weighting prefers kimi-k3 there. The earlier 7-model spread was measured while energy_observations was empty, so cost and eco were scoring the neutral 0.5 and proficiency decided alone. Recording both numbers, since the difference between them is the whole argument for measuring rather than assuming. Proficiency remains live below the top slot: runner-up ordering reorders by category, with kimi-k3-fast climbing to 3rd on docs_writing and summarization (proficiency 1.00) and leaving the top four elsewhere. The axis works; it cannot overturn a leader that wins on two axes simultaneously. The seed timer's first run has landed, so cost/eco medians now span two time points rather than one moment. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 28c9949..89efdd9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -228,9 +228,27 @@ made no difference. It does now: |---|---| | catalog list price | 2 | | measured cost + eco | 3 | -| **+ proficiency** | **7** | +| proficiency only (cost/eco data missing) | 7 | +| **all three axes live** | **1** | -Within tier 1 alone, four different models win depending on category. +That last row is not a regression, and it is worth understanding before +tuning anything. With cost, eco and proficiency all populated, +`qwen3.6-35b` is **Pareto-dominant**: cheapest AND cleanest in the routable +set, while scoring within 0.15 of the best model on proficiency. On +`summarization` it costs 7x less and emits 65x less carbon than `kimi-k3`, +which beats it 1.00 to 0.85 on quality — no defensible weighting picks +`kimi-k3` from that. One model winning is the correct answer to the question +the weights ask. + +Proficiency is still doing work underneath. The runner-up ordering reorders +by category — `kimi-k3-fast` climbs to 3rd on `docs_writing` and +`summarization` (proficiency 1.00) and drops out of the top four elsewhere. +So the axis is live; it just cannot overturn a leader that wins on two axes +at once. + +The 7-model spread appeared only while cost and eco were missing and scoring +neutral 0.5. If a genuinely different balance is wanted, the lever is +`weights` in config.yaml, not more data. ### What the task set actually found -- 2.49.1 From bf16327b22a9ac0890973c05bcdf97b3b0edb60b Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 21:09:20 -0400 Subject: [PATCH 09/20] docs: add README, and tighten the context estimate that real traffic exceeded README.md (from an opencode session routed through the router itself) documents what is actually built, module by module, with the pure/IO split and the four eval scoring kinds. It is accurate: every module it names exists, all four "pure" modules import no I/O, and it correctly files RAG context assembly under limitations rather than features. CLAUDE.md now points at it for "what is built" and at the design doc for "what is planned", and no longer claims 74 tests when there are 156. Also fixes a guardrail that real use walked straight through. CHARS_PER_TOKEN was 4, which holds for prose but not for agent traffic. A 106,158-token opencode prompt was routed to qwen3.6-35b, whose effective window is 94,196 -- overshooting the context hard filter by ~12k tokens. True density was at most 3.55 chars/token; code, JSON and tool schemas pack far denser than prose. It survived only on the 0.75 safety factor's slack against the advertised 131,056 limit. Set to 3, deliberately conservative. The asymmetry is the point: underestimating silently admits a model that cannot hold the prompt, while overestimating merely picks a roomier one. Tests cover the exact failing case. Worth recording separately: the opencode session's chat summary claimed a profiler.py module, 13 modules and 5,920 words. There is no profiler.py, there are 12 modules, and the file is 2,614 words. The artifact was accurate; the model's self-report about the artifact was not. Tests 156 -> 159. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 8 +- README.md | 389 ++++++++++++++++++++++++++++++++++ dispatcher.py | 23 +- tests/test_load_candidates.py | 31 +++ 4 files changed, 441 insertions(+), 10 deletions(-) create mode 100644 README.md diff --git a/CLAUDE.md b/CLAUDE.md index 89efdd9..1d52776 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,7 +1,9 @@ # Local LLM Model Router — project brief -Read `design/local-llm-model-router.md` first for full architecture/rationale. -This file is the working state + immediate next steps. +`README.md` documents what is actually built, module by module. +`design/local-llm-model-router.md` holds the architecture and rationale, +including parts still unbuilt. This file is the working state + immediate +next steps, and is the one to trust on what is currently true. ## What this is @@ -178,7 +180,7 @@ provisional. A single sweep's ranking is one sample of a moving quantity. `code` executes the model's Python against checks, `exact` compares a normalized answer, `tool` inspects the tool call structurally, and only the four prose categories fall back to a `judge`. -- `tests/` — 74 tests, all passing. +- `tests/` — 156 tests across 9 files, all passing. ### Serving class: one base model, many rows diff --git a/README.md b/README.md new file mode 100644 index 0000000..cecc7e4 --- /dev/null +++ b/README.md @@ -0,0 +1,389 @@ +# Local LLM Model Router + +A router that uses a local model (served via Ollama on an RTX 6000, 24GB) to +classify incoming coding/documentation tasks — category, tier, required +context size — and dispatch each task to the cheapest/best-fit open-weight +model on **Neuralwatt Cloud**, weighted by cost, ecological impact +(Neuralwatt exposes real energy-per-request data), and per-category +proficiency. + +Neuralwatt is the only provider. The `provider` column and the +`(model_id, provider)` primary key stay so a second provider can be added +later without a migration. + +## Architecture + +``` + ┌─────────────────────┐ + incoming task ───▶│ Local Classifier │ Ollama (qwen3.5:latest) + │ - task_category │ + │ - task_tier │ ~4s warm, ~120s cold cap + │ - required_context │ temperature: 0, max 1024 tokens + │ - confidence │ + └──────────┬───────────┘ + ▼ + ┌─────────────────┐ + │ Escalation │ low-confidence tier bump + │ (optional) │ threshold configurable + └────────┬─────────┘ + ▼ + ┌─────────────────────┐ + │ Hard Filters │ context window ≥ required + │ (routing.py) │ tier ≥ required + │ │ freshness (active, not stale) + │ │ access_level allowed + │ │ latency_class compatible + └──────────┬───────────┘ + ▼ + ┌─────────────────────┐ + │ Weighted Scoring │ cost + eco + proficiency + │ (scoring.py) │ + └──────────┬───────────┘ + ▼ + ┌─────────────────────┐ + │ Dispatcher │──▶ Neuralwatt Cloud + │ (FastAPI API) │──▶ OpenAI-compatible /v1 endpoints + │ │ Streaming chunk proxy w/ + │ │ SSE comment telemetry scrape + └──────────┬───────────┘ + ▼ + ┌─────────────────────┐ + │ Energy Logging │ per-request cost, kWh, gCO2eq + │ (SQLite) │ seeds median stats for routing + └─────────────────────┘ +``` + +## Tech Stack + +| Layer | Technology | +|---|---| +| **Language** | Python 3 (IO-bound provider APIs; iteration speed matters more than raw speed) | +| **Framework** | FastAPI + uvicorn | +| **Database** | SQLite (`router.db`) — decision table, energy observations, proficiency | +| **Local Classification** | Ollama (OpenAI-compatible at `localhost:11434/v1`) | +| **Local Model** | `qwen3.5:latest` | +| **Cloud Provider** | Neuralwatt only | +| **Config** | `config.yaml` loaded & validated by Pydantic (`config.py`) | +| **OpenAI Client** | `openai==3.0.0` (official SDK) | +| **HTTP** | `requests` for poller, `httpx` (via openai/uvicorn) | +| **Testing** | `pytest` — 156 tests across 9 files | +| **Config Files** | `config.yaml`, `leaderboards.yaml`, `evals/tasks.yaml` | +| **Deployment** | systemd user units (`.service` + `.timer` files in `deploy/`) | +| **Integration** | Pre-configured in `opencode.json` — `cd` into repo uses router by default | + +## Pinned Dependencies (requirements.txt) + +``` +pyyaml==6.0.3 +pydantic==2.13.4 +requests==2.34.2 +fastapi==0.141.1 +uvicorn[standard]==0.52.1 +openai==3.0.0 +python-dotenv==1.2.2 +pytest==9.1.1 +pytest-cov==7.1.0 +``` + +**Pinned intentionally.** Recreating the venv with `>=` constraints silently jumped +`openai 2.53 → 3.0` and `httpx → httpx2` without warning; a service that +restarts on boot shouldn't change its dependency tree underneath itself. + +## Modules + +| Module | Role | I/O? | +|---|---|---| +| **`dispatcher.py`** | FastAPI service: routes, calls providers, logs, streams | Yes (DB, network) | +| **`poller.py`** | Fetches Neuralwatt catalog, normalizes, upserts `models` table | Yes (network, DB) | +| **`scoring.py`** | `normalize_inverted` + `composite_score` weighted formula | Pure | +| **`routing.py`** | Hard filters (`select_candidates`) + ranking (`rank_candidates`) | Pure | +| **`tiering.py`** | Pure tier resolver: 1=cheap, 2=mid, 3=frontier | Pure | +| **`tier.py`** | DB tiering pass: reads models, resolves, writes `tier` column | Yes (DB) | +| **`proficiency.py`** | Score blending: leaderboard + self-eval → weighted composite | Pure | +| **`proficiency_store.py`** | DB access for `proficiency` table; single write path ensuring `blended_score` never drifts | Yes (DB) | +| **`seed_energy.py`** | Reference workload sweep: fixed prompt × N runs per model | Yes (network, DB) | +| **`eval_proficiency.py`** | Self-eval harness: 4 scoring kinds against N models | Yes (network, DB, subprocess) | +| **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) | +| **`config.py`** | YAML loader + Pydantic validators (weights sum to 1, valid tiers, etc.) | Yes (file) | + +## Decision Table Schema (SQLite) + +Three tables, `PRAGMA foreign_keys = ON`: + +### `models` — one row per served model variant + +| Column | Type | Notes | +|---|---|---| +| `model_id` | TEXT | Full catalog id (e.g. `glm-5.2-short-fast-flex`) | +| `provider` | TEXT | `neuralwatt` | +| `base_model_id` | TEXT | Model family (e.g. `glm-5.2`). Proficiency/leaderboard keys here. | +| `display_name` | TEXT | Human-readable name | +| `cost_per_1m_prompt` | REAL | Listed USD per 1M input tokens | +| `cost_per_1m_completion` | REAL | Listed USD per 1M output tokens | +| `cost_per_1m_prompt_cached` | REAL | Cached prefix price (null if no cache discount) | +| `context_window` | INTEGER | Advertised max tokens | +| `effective_context_window` | INTEGER | `advertised × safety_factor − reserve` | +| `max_output_tokens` | INTEGER | | +| `tier` | INTEGER | 1–3, set by `tier.py` pass | +| `supports_tools` | INTEGER | Boolean 0/1 | +| `supports_json_mode` | INTEGER | | +| `supports_vision` | INTEGER | | +| `supports_reasoning` | INTEGER | "API accepts reasoning param" — NOT a quality signal | +| `reasoning_default_enabled` | INTEGER | The actual tier-bearing signal | +| `latency_class` | TEXT | `standard` \| `flex` (`-flex`: discounted async, held during peak) | +| `reasoning_mode` | TEXT | `default` \| `reduced` (`-fast`: reasoning capped) | +| `context_variant` | TEXT | `full` \| `short` (`-short`: 200K pool bounded budget) | +| `access_level` | TEXT | `public` \| `preview` \| `canary` | +| `pricing_tbd` | INTEGER | | +| `deprecated` | INTEGER | | +| `availability` | TEXT | `active` \| `deprecated` \| `stale` | +| `last_updated` | TEXT | ISO8601 | + +**Serving class:** Neuralwatt ships ~6 base models as 19 catalog rows. The id +suffixes are three **orthogonal** dimensions (`glm-5.2-short-fast-flex`), parsed +by `poller.parse_serving_class` into columns. Rows carry identical catalog +pricing, so routing would pick between them arbitrarily without these — the +`latency_tolerance` hard filter resolves it. + +**Access gating:** 6 of 19 rows are prose-gated +("Private preview (grant-gated)", "(Canary)"). `poller.parse_access_level` +parses them into `access_level` and `routing.allowed_access_levels` (default +`[public]`) excludes them, so dispatch won't earn a 403. + +### `proficiency` — one row per (model, provider, category) + +| Column | Type | Notes | +|---|---|---| +| `model_id` | TEXT | | +| `provider` | TEXT | | +| `category` | TEXT | See category list below | +| `leaderboard_score` | REAL | 0–1, from external benchmarks | +| `self_eval_score` | REAL | 0–1, from self-eval harness | +| `self_eval_samples` | INTEGER | Evidence count for blending threshold | +| `blended_score` | REAL | Computed: `w_lb × lb + w_se × se` | +| `source` | TEXT | `blended` \| `self_eval` \| `self_eval_thin` \| `leaderboard` | +| `last_updated` | TEXT | ISO8601 | + +**Category set** (9 categories, defined in `config.yaml`): + +| Category | Example | Scoring type | +|---|---|---| +| `coding_general` | Merge intervals, parse semver, word wrap | Code (execution) | +| `coding_refactor` | Remove repetition, refactor dispatch chain | Code (execution) | +| `debugging` | Fix closure leak, fix binary search, fix regex | Code (execution) | +| `reasoning_math` | Percent trap, rate trap, counting | Exact match | +| `tool_use_agentic` | Right tool / right args / no tool when empty | Structural | +| `docs_writing` | Docstring quality, must-mention gotchas | Judge | +| `summarization` | Root-cause isolation, buried-lede identification | Judge | +| `translation` | Technical register, hedging/informal tone | Judge | +| `general_chat` | Simple explanations, measured pushback | Judge | + +**Scoring kinds** (4 types, objective wherever the category admits it): +- **`code`** — runs model-generated Python in a subprocess, scores fraction of checks passing +- **`exact`** — normalizes & compares a single answer +- **`tool`** — structural: which tool was called, which args matched +- **`judge`** — a strong model scores against a rubric (prose categories only) + +**Blending:** Once `self_eval_samples ≥ self_eval_min_samples` (default 10): +`blended = 0.3 × leaderboard + 0.7 × self_eval`. Before that, falls back +to leaderboard alone. If neither exists, the row scores neutral 0.5. + +Source labels distinguish confidence levels: `self_eval_thin` means +real measurement but below the sample threshold — a caller wanting to +exclude it can. + +**Proficiency inheritance:** `propagate_to_variants` copies evaluated +`scores` to equivalent serving variants (same weights, same reasoning +setting, same context pool), but only when the variant has no measured +data itself. A `-fast` row is **not** equivalent to its `-standard` sibling. + +### `energy_observations` — per-request telemetry + +| Column | Type | Notes | +|---|---|---| +| `id` | INTEGER | Autoincrement | +| `model_id` / `provider` | TEXT | | +| `task_category` | TEXT | | +| `prompt_tokens` / `completion_tokens` | INTEGER | | +| `energy_kwh` | REAL | **Attributed** billed figure (noisy, 20× within-model) | +| `energy_btu` | REAL | `kwh × 3412.14`, dashboard value | +| `avg_power_watts` / `duration_seconds` | REAL | Pre-attribution product, ~1.8× within-model | +| `attribution_ratio` | REAL | Request's share of shared GPU pool (stable quantized: 0.001, 0.25, 0.5, 0.75) | +| `carbon_g_co2eq` | REAL | Reported by provider | +| `grid_carbon_intensity` | REAL | gCO2/kWh at call time | +| `grid_id` | TEXT | e.g. `FI` | +| `carbon_source` | TEXT | `static_fallback` (constant, excluded from routing) or live measurement | +| `cost_usd` | REAL | **Billed** figure, not tokens × list price | +| `allowance_remaining_usd` | REAL | | +| `service_tier` | TEXT | As billed | +| `observed_at` | TEXT | ISO8601 | + +**Attribution noise:** Billed `energy_kwh = avg_power_watts × duration × +attribution_ratio`. The attribution term looks like noise up close (8 identical +calls varied 20×), but ranks 750× between models while within-model spread is +1.8× — it's a stable per-model property reflecting serving concurrency. Routing +scores on the attributed figures with a **median** over all `seed_reference` +rows, so repeated sweeps accumulate into a median-across-time. + +## Weighted Scoring + +Three hard filters are applied **before** scoring (not weighted — outright disqualification): + +1. `effective_context_window ≥ required_context_tokens` +2. `tier ≥ required_tier` (from classifier) +3. Serving class compatible with request's `latency_tolerance`; `access_level` reachable + +``` +composite = w_cost × cost_score (cheapest = 1.0, inverted min-max) + + w_eco × eco_score (cleanest = 1.0, inverted min-max) + + w_prof × proficiency_score (from the task's category) +``` + +| Weight | Value | Notes | +|---|---|---| +| `cost` | 0.4 | From median of billed USD (`seed_reference` category only) | +| `eco` | 0.2 | From median of gCO2eq (`seed_reference` only) | +| `proficiency` | 0.4 | From `proficiency.blended_score` for the task category | + +**Why cost ≠ list price:** Neuralwatt bills flat $8.00/kWh. The catalog's +`input_per_million` / `output_per_million` are **not** what gets charged. +Measured: `cost_usd / energy_kwh = 8.00` across every model. A model listing +at $4/1M can cost 10× more than one at $15/1M on the same prompt. + +**Why cost ≠ eco:** Cost tracks energy (kWh), but carbon is energy × grid +intensity. Grid intensity spans 37 gCO2/kWh (`FI`) to 505 +(`US-MIDA-PJM`) — a 13.6× spread. The models disagree: `glm-5.2-fast` is +2nd cheapest but 6th cleanest; `kimi-k3-flex` draws 3.7× less energy than +`kimi-k2.7-code` while emitting 3.6× more carbon. Collapsing them picks a +side. + +## API Endpoints + +The dispatcher binds `127.0.0.1:8080`. **No auth of its own** — loopback +is the only thing standing between the open internet and your billing +allowance. + +| Method | Path | Description | +|---|---|---| +| `GET` | `/health` | Catalog/reachability status, scoring coverage, warnings | +| `POST` | `/route` | Classify task, rank candidates, return selected model — **no provider call, no cost** | +| `POST` | `/dispatch` | Same as `/route`, plus complete the provider call, stream response, log observation | +| `GET` | `/v1/models` | OpenAI-compatible model list (router virtual models + catalog) | +| `POST` | `/v1/chat/completions` | OpenAI-compatible completions — routes then proxies, **streaming supported** | + +Input to `/route` and `/dispatch` can include `task_category`, `task_tier`, +and `required_context_tokens` overrides — these skip the classifier, useful +for testing routing without the classifier in the loop. + +Two virtual router models: +- `auto` — normal routing, `-flex` rows excluded (interactive) +- `auto:batch` — admits `-flex` rows (overnight/async work) + +Ask for **any real model id** in `/v1/chat/completions` and it dispatches +directly, still logged — routing is transparent, not opaque. + +**Streaming:** Passed through chunk-by-chunk so tokens render as they arrive. +Neuralwatt emits energy and cost as SSE **comment** lines (`: energy {...}`) +before `data: [DONE]` — ordinary clients ignore comments, so the stream flows +untouched while the router scrapes telemetry on the way past. Without this, +streamed calls would log no energy at all. + +## Scheduled Jobs (systemd) + +Five user units. See `deploy/README.md` for full install/operate instructions. + +| Unit | Schedule | What it runs | +|---|---|---| +| `llm-router.service` | Continuous | FastAPI dispatcher | +| `llm-router-poller.timer` | 2 min after boot, then every 2 h | `poller.py` → `tier.py` | +| `llm-router-seed.timer` | Every 6 h | Small `seed_energy.py` sweep | + +**The poller timer is load-bearing, not optional.** `freshness.stale_after_days` +is 3 with `exclude_stale: true` — an unpolled catalog marks every row stale +in 3 days and the router returns zero candidates for everything. + +**The seed timer spans time.** Energy attribution drifts with pool load +across hours (~50× for `deepseek-v4-flash` between sweeps), so a single +sweep measures one moment. The median has to span time — every 6 h sweep +accumulates into a time-weighted median automatically. + +## Self-Eval Harness (`eval_proficiency.py`) + +```bash +python eval_proficiency.py # every routable model × every task +python eval_proficiency.py --models kimi-k3 # subset of models +python eval_proficiency.py --categories coding_general +python eval_proficiency.py --dry-run # plan only +``` + +- Runs every task through the target provider, scores it, writes to `proficiency`. +- Scores **accumulate** (running mean), so repeated runs tighten estimates. +- `propagate_to_variants` copies evaluated scores to equivalent serving variants. +- Safety: code tasks run in a temp directory with a 15s wall-clock timeout — + bounded isolation, not a container. +- Judge tasks skip the model being judged (avoids self-scoring bias). + +## Setup + +```bash +python -m venv .venv && source .venv/bin/activate +pip install -r requirements.txt +sqlite3 router.db < schema.sql +cp .env.example .env # fill in NEURALWATT_API_KEY +python poller.py # populate the catalog +python tier.py # resolve tiers +python seed_energy.py # seed reference energy observations (~5 per model) +python -m uvicorn dispatcher:app --reload +``` + +Ollama must be running locally: +```bash +ollama pull qwen3.5:latest +``` + +## Pointing a Coding Agent at It + +The `/v1` endpoints are OpenAI-compatible. Repo-local `opencode.json` is +already wired up (`cd ~/Sources/6krrt && opencode` routes by the router by +default). For global use, merge `provider.llm-router` into +`~/.config/opencode/opencode.json`. + +Virtual model names: +- `auto` → router picks, interactive mode (flex rows excluded) +- `auto:batch` → router picks, admits flex rows for async work + +## Testing + +```bash +python -m pytest # run all 156 tests +python -m pytest --cov # with coverage +``` + +| Test file | What it covers | +|---|---| +| `test_scoring.py` | `normalize_inverted`, `composite_score`, None-handling | +| `test_routing.py` | Hard filters, candidate selection & ranking across all 9 categories × 3 tiers | +| `test_tiering.py` | Tier resolver precedence: override → reasoning → cost → mid | +| `test_apply_tiering.py` | DB tiering pass, sanity guards | +| `test_poller_parsing.py` | Serving class, base model, access level parsing | +| `test_proficiency.py` | Blending rule, accumulation | +| `test_load_candidates.py` | `load_candidates`: cost/eco/proficiency join | +| `test_eval_scoring.py` | Code, exact, tool, and judge scoring | +| `test_task_set.py` | Reference solutions validating every `code` task's checks & `exact` answers | + +## Known Limitations & Open Items + +- **Leaderboard priors unfilled** — `leaderboards.yaml` ships empty. `python leaderboard.py --check` + names every active family lacking a prior. Must source figures from + published benchmarks; inventing plausible numbers is the same failure as + the provider's `static_fallback` carbon constant. +- **Three models unsettled** — split-half stability at 7 samples: + `kimi-k2.7-code-fast` (29×), `kimi-k3` (14×), `glm-5.2-flex` (2.2×). + More samples needed before their positioning is trustworthy. +- **No escalation feedback** — `apply_escalation` works (low-confidence + tier bumped), but there's no way to flag "this was routed wrong" after + the fact and retry one tier up. +- **No auth** — the service holds a billable API key with no + authentication of its own. Loopback binds only. +- **Context assembly (RAG)** is out of scope — the classifier sees the full + conversation but does not perform document/code retrieval. diff --git a/dispatcher.py b/dispatcher.py index 285c96e..b28d371 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -86,13 +86,22 @@ def gross_energy_kwh(avg_power_watts: float, duration_seconds: float) -> float: """ return avg_power_watts * duration_seconds / 3_600_000 -# Rough chars-per-token. Only used as a FLOOR on the classifier's context -# estimate: a real coding session sends the whole conversation, and the -# classifier — which sees that conversation and is asked to estimate its own -# input — routinely underestimates it by orders of magnitude (10 tokens for a -# 600-token exchange in testing). Underestimating here silently admits models -# that cannot hold the prompt, so the measured size wins when it is larger. -CHARS_PER_TOKEN = 4 +# Rough chars-per-token, used as a FLOOR on the classifier's context estimate: +# a real coding session sends the whole conversation, and the classifier — +# which sees that conversation and is asked to estimate its own input — +# underestimates it by orders of magnitude (10 tokens for a 600-token exchange +# in testing). So the measured size wins when it is larger. +# +# Deliberately conservative at 3, not the ~4 that holds for English prose. +# Measured from real opencode traffic: a 106,158-token prompt was routed to a +# model with a 94,196 effective window, meaning the true density was at most +# 3.55 chars/token — agent traffic is code, JSON and tool schemas, which pack +# far denser than prose. That overshot the guard by ~12k tokens and survived +# only on the 0.75 safety factor's slack. +# +# The asymmetry matters: underestimating silently admits a model that cannot +# hold the prompt, while overestimating merely picks a roomier one. +CHARS_PER_TOKEN = 3 load_dotenv() diff --git a/tests/test_load_candidates.py b/tests/test_load_candidates.py index 555940d..cbbe99c 100644 --- a/tests/test_load_candidates.py +++ b/tests/test_load_candidates.py @@ -139,3 +139,34 @@ def test_unswept_model_has_no_measurements(db): assert row["cost"] is None assert row["eco"] is None assert row["samples"] == 0 + + +# --- context estimation --------------------------------------------------- + +def test_context_estimate_is_conservative_for_agent_traffic(): + # Real measurement: opencode sent a prompt that tokenized to 106,158 while + # the chars/4 estimate put it under 94,196 — so it was routed to a model + # that could not hold it within its effective window. Agent traffic is + # code, JSON and tool schemas, which pack denser than prose. + from dispatcher import estimate_prompt_tokens + + dense = "x" * 376_784 # the char count implied by that failure + # At the old divisor of 4 this estimated 94,196 and passed the filter. + assert estimate_prompt_tokens([{"role": "user", "content": dense}]) > 94_196 + + +def test_context_estimate_counts_every_message(): + from dispatcher import estimate_prompt_tokens + + msgs = [{"role": "user", "content": "a" * 300}, {"role": "assistant", "content": "b" * 300}] + assert estimate_prompt_tokens(msgs) == 200 + + +def test_context_estimate_reads_multimodal_text_parts(): + from dispatcher import estimate_prompt_tokens + + msgs = [{"role": "user", "content": [ + {"type": "text", "text": "c" * 300}, + {"type": "image_url", "image_url": {"url": "data:..."}}, + ]}] + assert estimate_prompt_tokens(msgs) == 100 -- 2.49.1 From c360bde17f92b1ad7ea113dcac99d5743c9f0fb0 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 21:30:15 -0400 Subject: [PATCH 10/20] feat: verify every response structurally, without executing it Free, exact checks on what a model just returned. Extracts fenced blocks and validates them by PARSING: ast.parse for Python, json.loads, yaml.safe_load. Plus finish_reason='length' and unterminated fences, which catch the failure mode that has bitten this project repeatedly -- a truncated answer looks complete to the client, which then acts on a fragment. The router deliberately does NOT execute model output, unlike eval_proficiency.py. There the prompts are ones this project authored, so what comes back is bounded. Here the code is whatever the user asked for and could do anything, so running it as a side effect of routing would be indefensible. Two tests assert non-execution, including one that would delete a file. 'unverifiable' is recorded and is NOT a failure. Most prose lands there, and counting "we could not check this" as "this was wrong" would penalize models for the checker's limits -- the same mistake as scoring a judge malfunction against a model, which this project made once already. Streaming accumulates deltas so a streamed answer gets the same check as a buffered one; otherwise streaming, which is how every agent client talks to the router, would be the unverified path. Non-streaming reports the verdict in an X-Router-Verification header rather than the body, so the response stays a valid OpenAI object and a client that ignores headers is unaffected. Why this is worth running unconditionally: measured on real traffic, a wasted cloud completion costs about 52 local checks at 1,500 tokens and 139 at 4,000. A check that costs nothing at all clears that bar trivially. Tests 159 -> 179. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- dispatcher.py | 82 +++++++++++++++- schema.sql | 22 +++++ tests/test_verification.py | 151 +++++++++++++++++++++++++++++ verification.py | 194 +++++++++++++++++++++++++++++++++++++ 4 files changed, 446 insertions(+), 3 deletions(-) create mode 100644 tests/test_verification.py create mode 100644 verification.py diff --git a/dispatcher.py b/dispatcher.py index b28d371..4f7d962 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -45,12 +45,13 @@ from typing import Any, Literal, Optional import requests from dotenv import load_dotenv from fastapi import FastAPI, HTTPException -from fastapi.responses import StreamingResponse +from fastapi.responses import JSONResponse, StreamingResponse from openai import OpenAI, OpenAIError from pydantic import BaseModel, Field from config import RouterConfig, load_config from routing import BATCH, INTERACTIVE, rank_candidates, select_candidates +from verification import verify_response # Virtual model names that mean "you pick". Anything else is taken as a real # model id and dispatched as asked. @@ -507,6 +508,46 @@ def extract_telemetry(payload: dict) -> Telemetry: ) +def log_verification( + model_id: str, + provider: str, + task_category: Optional[str], + kind: str, + verdict: str, + detail: str, + completion_tokens: Optional[int], +) -> None: + """Record what a check said about one completion. + + Recorded for every response including 'unverifiable', because the rate at + which responses cannot be checked is itself worth knowing — if most + traffic is unverifiable, structural checking is not earning its place. + """ + conn = _db() + try: + conn.execute( + """ + INSERT INTO verifications ( + model_id, provider, task_category, kind, verdict, detail, + completion_tokens, observed_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + model_id, + provider, + task_category, + kind, + verdict, + detail[:300] if detail else None, + completion_tokens, + datetime.now(timezone.utc).isoformat(), + ), + ) + conn.commit() + finally: + conn.close() + + def log_observation( model_id: str, provider: str, @@ -840,21 +881,45 @@ def chat_completions(body: dict[str, Any]): raise HTTPException(resp.status_code, resp.text[:500]) payload = resp.json() usage = payload.get("usage") or {} + completion_tokens = usage.get("completion_tokens") log_observation( target, provider, category, usage.get("prompt_tokens"), - usage.get("completion_tokens"), + completion_tokens, extract_telemetry(payload), ) + + choice = (payload.get("choices") or [{}])[0] + content = (choice.get("message") or {}).get("content") or "" + result = verify_response(content, choice.get("finish_reason")) + log_verification( + target, provider, category, "structural", + result.verdict, result.detail, completion_tokens, + ) + # Report the model actually used, so the client isn't told 'auto'. payload["model"] = target - return payload + # Surfaced as a header rather than in the body: the body must stay a + # valid OpenAI response, and a client that ignores headers is + # unaffected. + return JSONResponse( + content=payload, + headers={ + "X-Router-Model": target, + "X-Router-Verification": result.verdict, + }, + ) def proxy(): collected: dict[str, dict] = {} usage: dict = {} + # Accumulated so a streamed answer gets the same structural check as a + # buffered one. Without this, streaming — which is how opencode and + # every other agent talks to the router — would be the unverified path. + content_parts: list[str] = [] + finish_reason: Optional[str] = None upstream = requests.post( url, headers=headers, json=upstream_body, stream=True, timeout=600 ) @@ -874,6 +939,12 @@ def chat_completions(body: dict[str, Any]): chunk = json.loads(raw[6:]) if chunk.get("usage"): usage = chunk["usage"] + for ch in chunk.get("choices") or []: + piece = (ch.get("delta") or {}).get("content") + if piece: + content_parts.append(piece) + if ch.get("finish_reason"): + finish_reason = ch["finish_reason"] except json.JSONDecodeError: pass yield f"{raw}\n".encode() @@ -888,6 +959,11 @@ def chat_completions(body: dict[str, Any]): usage.get("completion_tokens"), extract_telemetry(collected), ) + result = verify_response("".join(content_parts), finish_reason) + log_verification( + target, provider, category, "structural", + result.verdict, result.detail, usage.get("completion_tokens"), + ) return StreamingResponse(proxy(), media_type="text/event-stream") diff --git a/schema.sql b/schema.sql index 0098d7c..2d36870 100644 --- a/schema.sql +++ b/schema.sql @@ -125,6 +125,28 @@ CREATE TABLE IF NOT EXISTS energy_observations ( observed_at TEXT NOT NULL ); +-- One row per verified completion. Structural checks are free, so every +-- response gets one; the point is to learn which models fail on REAL work +-- rather than only on the fixed 23-task benchmark in evals/tasks.yaml. +-- +-- 'unverifiable' is recorded and is NOT a failure. Most prose lands there, +-- and counting "we could not check this" as "this was wrong" would penalize +-- models for the checker's limits. +CREATE TABLE IF NOT EXISTS verifications ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + model_id TEXT NOT NULL, + provider TEXT NOT NULL, + task_category TEXT, + kind TEXT NOT NULL, -- 'structural' | 'local_llm' + verdict TEXT NOT NULL, -- 'ok' | 'truncated' | 'malformed' | 'unverifiable' + detail TEXT, + completion_tokens INTEGER, -- what a wasted answer cost, for the payoff sum + observed_at TEXT NOT NULL +); + +CREATE INDEX IF NOT EXISTS idx_verifications_model ON verifications (model_id, provider); +CREATE INDEX IF NOT EXISTS idx_verifications_verdict ON verifications (verdict); + CREATE INDEX IF NOT EXISTS idx_models_provider ON models (provider); CREATE INDEX IF NOT EXISTS idx_models_availability ON models (availability); CREATE INDEX IF NOT EXISTS idx_models_routing ON models (access_level, latency_class, tier); diff --git a/tests/test_verification.py b/tests/test_verification.py new file mode 100644 index 0000000..4bae5b5 --- /dev/null +++ b/tests/test_verification.py @@ -0,0 +1,151 @@ +"""Tests for verification.py — structural checks on model responses. + +The load-bearing distinction here is between "this is wrong" and "we could +not check this". Conflating them would penalize a model for the checker's +limits, which is the same mistake as scoring a judge malfunction against a +model — something this project has already done once. + +Nothing here executes model output, and the tests assert that. +""" + +import pytest + +from verification import ( + Block, + check_block, + check_json, + check_python, + check_yaml, + extract_blocks, + verify_response, +) + + +# --- truncation, the failure that looks like success ---------------------- + +def test_finish_reason_length_is_decisive(): + # A truncated answer that happens to parse is the dangerous case: it looks + # complete to the client, which then acts on a fragment. + v = verify_response("```python\nx = 1\n```", finish_reason="length") + assert v.verdict == "truncated" + assert v.failed + + +def test_unterminated_fence_is_truncated_even_if_it_parses(): + # "x = 1" is valid Python; the missing closing fence is the signal + v = verify_response("here you go:\n```python\nx = 1\n") + assert v.verdict == "truncated" + + +def test_complete_response_is_ok(): + v = verify_response("```python\ndef f(x):\n return x * 2\n```") + assert v.verdict == "ok" + assert not v.failed + + +# --- malformed content ---------------------------------------------------- + +def test_broken_python_is_malformed_with_a_line_number(): + v = verify_response("```python\ndef f(x)\n return x\n```") + assert v.verdict == "malformed" + assert "line" in v.detail + + +def test_broken_json_is_malformed(): + assert check_json('{"a": 1,}').verdict == "malformed" + assert check_json('{"a": 1}').verdict == "ok" + + +def test_broken_yaml_is_malformed(): + assert check_yaml("a: [1, 2\nb: 3").verdict == "malformed" + assert check_yaml("a: 1\nb: two").verdict == "ok" + + +def test_prose_left_inside_a_code_fence_is_caught(): + # Models sometimes trail off into explanation without closing the fence + v = verify_response("```python\ndef f():\n return 1\nThis function returns one.\n```") + assert v.verdict == "malformed" + + +# --- unverifiable is not failure ----------------------------------------- + +def test_plain_prose_is_unverifiable_not_failed(): + v = verify_response("A B-tree keeps itself balanced by splitting nodes.") + assert v.verdict == "unverifiable" + assert not v.failed + + +def test_unknown_language_is_unverifiable_not_failed(): + v = verify_response("```rust\nfn main() { let x = ; }\n```") + assert v.verdict == "unverifiable" + assert not v.failed + + +def test_shell_is_deferred_to_the_caller_not_guessed(): + # bash -n is a subprocess, so this module declines rather than pretending + c = check_block(Block(lang="bash", code="echo hi", closed=True)) + assert c.verdict == "unverifiable" + + +def test_empty_block_is_unverifiable(): + assert check_block(Block(lang="python", code=" \n", closed=True)).verdict == "unverifiable" + + +def test_response_with_no_blocks_at_all_is_unverifiable(): + assert verify_response("").verdict == "unverifiable" + + +# --- multiple blocks ------------------------------------------------------ + +def test_one_bad_block_fails_the_response(): + text = "```python\nx = 1\n```\nand\n```json\n{bad\n```" + v = verify_response(text) + assert v.verdict == "malformed" + assert len(v.checks) == 2 + + +def test_truncation_outranks_malformation_in_reporting(): + # If a response is both cut off and broken, the truncation is the cause + text = "```json\n{bad\n```\n```python\nx = (\n" + v = verify_response(text) + assert v.verdict == "truncated" + + +def test_all_good_blocks_pass(): + text = "```python\nx = 1\n```\ntext\n```json\n{\"a\": 1}\n```" + assert verify_response(text).verdict == "ok" + + +def test_checkable_block_alongside_an_uncheckable_one_still_passes(): + text = "```rust\nfn main() {}\n```\n```python\nx = 1\n```" + assert verify_response(text).verdict == "ok" + + +# --- extraction ----------------------------------------------------------- + +def test_extract_records_language_and_closure(): + blocks = extract_blocks("```python\na\n```\n```\nb\n") + assert [(b.lang, b.closed) for b in blocks] == [("python", True), ("", False)] + + +def test_language_tag_is_case_insensitive(): + assert verify_response("```PYTHON\nx = 1\n```").verdict == "ok" + + +# --- safety --------------------------------------------------------------- + +def test_verification_never_executes_the_code_it_checks(tmp_path): + # If verification ran this, the file would exist. It must not. + canary = tmp_path / "canary.txt" + payload = f"```python\nopen({str(canary)!r}, 'w').write('executed')\n```" + v = verify_response(payload) + assert v.verdict == "ok" # it parses fine + assert not canary.exists() # ...and was never run + + +def test_a_syntactically_valid_destructive_snippet_is_only_parsed(tmp_path): + victim = tmp_path / "important.txt" + victim.write_text("still here") + payload = f"```python\nimport os\nos.remove({str(victim)!r})\n```" + verify_response(payload) + assert victim.read_text() == "still here" diff --git a/verification.py b/verification.py new file mode 100644 index 0000000..d48d48b --- /dev/null +++ b/verification.py @@ -0,0 +1,194 @@ +"""Pure verification of model responses. + +Free, exact checks on what a model just returned, before the answer is +accepted. Like ``scoring.py`` and ``routing.py`` this module does no I/O and +makes no network calls, so it is testable in milliseconds and costs nothing +to run on every request. + +**This module never executes model output.** ``eval_proficiency.py`` does run +generated code, but there the prompts are ones this project authored, so what +comes back is bounded. Here the code is whatever the user asked a model to +write — it could delete files, make requests, anything — and running it as a +side effect of *routing* would be indefensible. So the checks are structural: +parse it, don't run it. ``ast.parse`` builds a tree without evaluating, +``json.loads`` and ``yaml.safe_load`` never construct arbitrary objects, and +shell is checked with ``bash -n`` (parse-only) by the caller. + +What this catches is the failure mode that has actually bitten this project +repeatedly: truncated and malformed output. A response cut off at the token +limit looks like a normal answer to a client and is silently broken. + +Economics, measured on real traffic: a wasted cloud completion costs about 52 +local checks at 1,500 tokens and 139 at 4,000, so a check that costs nothing +at all is worth running unconditionally. +""" + +from __future__ import annotations + +import ast +import json +import re +from dataclasses import dataclass, field +from typing import Literal, Optional + +Verdict = Literal["ok", "truncated", "malformed", "unverifiable"] + +# Languages whose syntax can be checked in-process without executing anything. +PYTHON_LANGS = frozenset({"python", "py", "python3"}) +JSON_LANGS = frozenset({"json", "jsonc"}) +YAML_LANGS = frozenset({"yaml", "yml"}) +SHELL_LANGS = frozenset({"bash", "sh", "shell", "zsh"}) + +FENCE_RE = re.compile( + r"```[ \t]*([A-Za-z0-9_+-]*)[ \t]*\r?\n(.*?)(?:```|\Z)", re.DOTALL +) + + +@dataclass +class Block: + """One fenced block from a response.""" + + lang: str + code: str + closed: bool + + +@dataclass +class Check: + """The outcome of validating one block.""" + + lang: str + verdict: Verdict + detail: str = "" + + +@dataclass +class Verification: + """What we can say about a whole response.""" + + verdict: Verdict + checks: list[Check] = field(default_factory=list) + detail: str = "" + + @property + def failed(self) -> bool: + return self.verdict in ("truncated", "malformed") + + +def extract_blocks(text: str) -> list[Block]: + """Pull fenced blocks out of a response. + + An unterminated fence is kept and flagged rather than dropped — a response + that runs out of tokens mid-block leaves exactly that, and it is the + signal worth catching. + """ + blocks = [] + for match in FENCE_RE.finditer(text or ""): + lang = (match.group(1) or "").lower() + body = match.group(2) + closed = match.group(0).rstrip().endswith("```") + blocks.append(Block(lang=lang, code=body, closed=closed)) + return blocks + + +def check_python(code: str) -> Check: + """Parse Python without running it. + + ``ast.parse`` builds a syntax tree and evaluates nothing, so this is safe + on arbitrary output. It catches truncation, unbalanced brackets, and the + stray prose that models sometimes leave inside a fence. + """ + try: + ast.parse(code) + except SyntaxError as e: + return Check("python", "malformed", f"line {e.lineno}: {e.msg}") + except (ValueError, MemoryError, RecursionError) as e: + return Check("python", "malformed", f"{type(e).__name__}: {e}") + return Check("python", "ok") + + +def check_json(code: str) -> Check: + try: + json.loads(code) + except json.JSONDecodeError as e: + return Check("json", "malformed", f"line {e.lineno}: {e.msg}") + return Check("json", "ok") + + +def check_yaml(code: str) -> Check: + # Imported lazily so the module stays dependency-free for callers that + # only need the Python and JSON checks. + try: + import yaml + except ImportError: # pragma: no cover + return Check("yaml", "unverifiable", "pyyaml not installed") + try: + yaml.safe_load(code) + except yaml.YAMLError as e: + return Check("yaml", "malformed", str(e).splitlines()[0][:120]) + return Check("yaml", "ok") + + +def check_block(block: Block) -> Check: + """Validate one block, or report that nothing structural applies. + + An unclosed fence is reported as truncated regardless of language: the + content may happen to parse, but the response was cut off mid-thought and + the client is about to act on a fragment. + """ + if not block.closed: + return Check(block.lang or "?", "truncated", "unterminated code fence") + if not block.code.strip(): + return Check(block.lang or "?", "unverifiable", "empty block") + + if block.lang in PYTHON_LANGS: + return check_python(block.code) + if block.lang in JSON_LANGS: + return check_json(block.code) + if block.lang in YAML_LANGS: + return check_yaml(block.code) + if block.lang in SHELL_LANGS: + # Shell needs `bash -n`, which is a subprocess and therefore the + # caller's job; this module stays I/O-free. + return Check(block.lang, "unverifiable", "shell needs an external parse") + return Check(block.lang or "?", "unverifiable", "no structural check for this language") + + +def verify_response( + text: str, finish_reason: Optional[str] = None +) -> Verification: + """Structurally verify a completion. + + ``finish_reason == 'length'`` is decisive on its own: the model ran out of + budget mid-answer, so whatever came back is a fragment even if it happens + to parse. That is checked first because a truncated response that parses + is the most dangerous case — it looks fine. + + A response with nothing checkable returns ``unverifiable``, which is NOT a + failure. Most prose answers land here, and treating "we could not check + this" as "this is wrong" would penalize models for the checker's limits — + the same mistake as scoring a judge malfunction against a model. + """ + if finish_reason == "length": + return Verification("truncated", detail="finish_reason=length") + + blocks = extract_blocks(text) + if not blocks: + return Verification("unverifiable", detail="no fenced blocks in response") + + checks = [check_block(b) for b in blocks] + + for verdict in ("truncated", "malformed"): + bad = [c for c in checks if c.verdict == verdict] + if bad: + return Verification( + verdict, + checks=checks, + detail=f"{len(bad)}/{len(checks)} blocks: {bad[0].lang}: {bad[0].detail}", + ) + + if any(c.verdict == "ok" for c in checks): + return Verification("ok", checks=checks, detail=f"{len(checks)} block(s) parsed") + return Verification( + "unverifiable", checks=checks, detail="no block had a structural check" + ) -- 2.49.1 From cf9f5c6be9783ecd3de73351dfb983921cef4ca8 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 21:43:57 -0400 Subject: [PATCH 11/20] feat: local LLM verification for responses nothing structural can check Runs on the local model AFTER the response has gone back to the client, so its ~6s never lands on anyone's latency. It exists to learn which models fail on real work, not to gate answers. Gated on answer size, which is economics rather than taste: a local check costs ~15% of a median 193-token answer, so it would only pay if such answers failed more than ~15% of the time. At 1,500 completion tokens the break-even failure rate drops to ~1.9%. Below the threshold a check costs more than the risk it removes, so small answers are left alone. Three defects were found and fixed by running it, each of which would have recorded false failures against models: 1. Head-truncating the answer for the checker made a complete 6,637-character response end mid-sentence, and the checker duly reported "cut off mid-thought". The harness manufactured the defect it then detected. Now the MIDDLE is elided and the cut is labelled, so the ending being judged is the real ending. 2. The checker then flagged the elision marker itself as a defect, so the system prompt now explains that the marker is the harness, not the model. 3. With thinking enabled the local model never emitted a verdict at all: 2,048 tokens produced 7,880 characters of reasoning and EMPTY content, and raising the cap to 8,192 simply bought more reasoning. Verification does not need chain-of-thought. Switched to Ollama's native endpoint, the only one that exposes `think`, and with think=False it answers in 25 tokens. An empty response is now settled by an if-statement rather than a model: asked about one, the local checker returned ok=true with the reason "Answer is too short" -- self-contradictory. Never ask a model what code can decide. After those fixes the checker returned the same verdict on 3/3 repeats of four labelled cases (complete, truncated, refusal, wrong-question). A malfunctioning checker still records NO sample rather than a failure. Tests 179 -> 193. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- config.py | 8 +++ config.yaml | 18 ++++++ dispatcher.py | 101 ++++++++++++++++++++++++++++-- tests/test_verification.py | 109 ++++++++++++++++++++++++++++++++- verification.py | 122 +++++++++++++++++++++++++++++++++++++ 5 files changed, 352 insertions(+), 6 deletions(-) diff --git a/config.py b/config.py index e1a4e7a..fb27f14 100644 --- a/config.py +++ b/config.py @@ -113,6 +113,13 @@ class RoutingConfig(BaseModel): return v +class VerificationConfig(BaseModel): + local_llm_enabled: bool = True + min_completion_tokens: int = 600 + timeout_seconds: int = 60 + max_output_tokens: int = 1024 + + class EscalationConfig(BaseModel): enabled: bool max_tier: int @@ -159,6 +166,7 @@ class RouterConfig(BaseModel): tiering: TieringConfig proficiency: ProficiencyConfig routing: RoutingConfig + verification: VerificationConfig = VerificationConfig() escalation: EscalationConfig freshness: FreshnessConfig database: DatabaseConfig diff --git a/config.yaml b/config.yaml index 3724616..58d7f9f 100644 --- a/config.yaml +++ b/config.yaml @@ -72,6 +72,24 @@ routing: # cost actually billed for the reference workload (see seed_energy.py), and # a flex row's measured cost already is its flex cost. +verification: + # Structural checks (parse the code, never run it) are free and always on. + # This section governs the LOCAL LLM check, which is not free. + local_llm_enabled: true + + # Only check answers this large. Measured on real traffic: a local check + # costs ~15% of a median 193-token answer, so it would only pay if such + # answers failed more than ~15% of the time. At 1,500 completion tokens the + # break-even failure rate drops to ~1.9%, which is plausible. Below the + # threshold a check costs more than the risk it removes. + min_completion_tokens: 600 + + # The check runs AFTER the response has gone back to the client, so it never + # adds its ~6s to anyone's latency. It exists to learn which models fail on + # real work, not to gate answers. + timeout_seconds: 60 + max_output_tokens: 1024 + freshness: stale_after_days: 3 # Router refuses to route to a model whose row is stale/deprecated, diff --git a/dispatcher.py b/dispatcher.py index 4f7d962..57dce64 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -44,14 +44,21 @@ from typing import Any, Literal, Optional import requests from dotenv import load_dotenv -from fastapi import FastAPI, HTTPException +from fastapi import BackgroundTasks, FastAPI, HTTPException from fastapi.responses import JSONResponse, StreamingResponse from openai import OpenAI, OpenAIError from pydantic import BaseModel, Field from config import RouterConfig, load_config from routing import BATCH, INTERACTIVE, rank_candidates, select_candidates -from verification import verify_response +from verification import ( + LOCAL_VERIFY_SYSTEM, + build_local_verify_prompt, + interpret_local_verdict, + parse_verdict_json, + verify_response, + worth_local_check, +) # Virtual model names that mean "you pick". Anything else is taken as a real # model id and dispatched as asked. @@ -548,6 +555,72 @@ def log_verification( conn.close() +def run_local_verification( + model_id: str, + provider: str, + task_category: Optional[str], + request_text: str, + answer: str, + completion_tokens: Optional[int], +) -> None: + """Second-opinion check on an answer nothing structural could judge. + + Runs on the LOCAL model, after the response has already gone back to the + client, so its ~6s never lands on anyone's latency. Failures here are + swallowed: a checker that cannot answer must not take down a request that + already succeeded, and must not record a verdict either — no sample beats + a false one. + """ + # Ollama's NATIVE endpoint, not the OpenAI-compatible one, because only it + # exposes `think`. That matters more than the inconsistency of using two + # APIs: with thinking on, this model spent its ENTIRE budget reasoning and + # never emitted the verdict — 2,048 tokens produced 7,880 characters of + # thought and empty content, and raising the cap to 8,192 just bought more + # thought. With think=False it answers in 25 tokens, and returned the same + # verdict on 3/3 repeats of four labelled cases. + url = cfg.classifier.base_url.rstrip("/") + if url.endswith("/v1"): + url = url[: -len("/v1")] + try: + resp = requests.post( + f"{url}/api/chat", + json={ + "model": cfg.classifier.model, + "think": False, + "stream": False, + "format": "json", + "options": { + "temperature": 0, + "num_predict": cfg.verification.max_output_tokens, + }, + "messages": [ + {"role": "system", "content": LOCAL_VERIFY_SYSTEM}, + { + "role": "user", + "content": build_local_verify_prompt(request_text, answer), + }, + ], + }, + timeout=cfg.verification.timeout_seconds, + ) + resp.raise_for_status() + raw = ((resp.json() or {}).get("message") or {}).get("content") or "" + except (requests.RequestException, ValueError) as e: + print(f"local verification unavailable ({type(e).__name__})", file=sys.stderr) + return + + check = interpret_local_verdict(parse_verdict_json(raw)) + if check is None: + print("local verification returned nothing usable; no sample recorded", + file=sys.stderr) + return + + log_verification( + model_id, provider, task_category, "local_llm", + check.verdict, check.detail, completion_tokens, + ) + + def log_observation( model_id: str, provider: str, @@ -804,7 +877,7 @@ def _sniff_telemetry_line(line: str) -> Optional[tuple[str, dict]]: @app.post("/v1/chat/completions") -def chat_completions(body: dict[str, Any]): +def chat_completions(body: dict[str, Any], background: BackgroundTasks): """OpenAI-compatible completions, routed then proxied. Streaming is passed through chunk by chunk rather than buffered, so a @@ -898,6 +971,13 @@ def chat_completions(body: dict[str, Any]): target, provider, category, "structural", result.verdict, result.detail, completion_tokens, ) + if cfg.verification.local_llm_enabled and worth_local_check( + result.verdict, completion_tokens, cfg.verification.min_completion_tokens + ): + background.add_task( + run_local_verification, target, provider, category, + _last_user_text(messages), content, completion_tokens, + ) # Report the model actually used, so the client isn't told 'auto'. payload["model"] = target @@ -959,11 +1039,22 @@ def chat_completions(body: dict[str, Any]): usage.get("completion_tokens"), extract_telemetry(collected), ) - result = verify_response("".join(content_parts), finish_reason) + streamed = "".join(content_parts) + result = verify_response(streamed, finish_reason) + ct = usage.get("completion_tokens") log_verification( target, provider, category, "structural", - result.verdict, result.detail, usage.get("completion_tokens"), + result.verdict, result.detail, ct, ) + # Inline rather than a background task: the generator has already + # finished streaming, so the client is not waiting on this. + if cfg.verification.local_llm_enabled and worth_local_check( + result.verdict, ct, cfg.verification.min_completion_tokens + ): + run_local_verification( + target, provider, category, + _last_user_text(messages), streamed, ct, + ) return StreamingResponse(proxy(), media_type="text/event-stream") diff --git a/tests/test_verification.py b/tests/test_verification.py index 4bae5b5..d4b63c9 100644 --- a/tests/test_verification.py +++ b/tests/test_verification.py @@ -92,7 +92,8 @@ def test_empty_block_is_unverifiable(): def test_response_with_no_blocks_at_all_is_unverifiable(): - assert verify_response("").verdict == "unverifiable" + # Prose, not empty — an empty response is a failure, decided separately + assert verify_response("A B-tree splits nodes to stay balanced.").verdict == "unverifiable" # --- multiple blocks ------------------------------------------------------ @@ -149,3 +150,109 @@ def test_a_syntactically_valid_destructive_snippet_is_only_parsed(tmp_path): payload = f"```python\nimport os\nos.remove({str(victim)!r})\n```" verify_response(payload) assert victim.read_text() == "still here" + + +# --- local LLM verification ----------------------------------------------- + +def test_local_verdict_survives_a_thinking_preamble(): + from verification import parse_verdict_json + raw = 'Let me consider the answer...\n\n{"ok": false, "reason": "cut off mid-sentence"}' + assert parse_verdict_json(raw)["ok"] is False + + +def test_unusable_local_output_records_no_verdict(): + # A checker that malfunctions must not produce a failure verdict — that + # charges the model for the checker's problem + from verification import interpret_local_verdict, parse_verdict_json + assert interpret_local_verdict(parse_verdict_json("I think it's fine")) is None + assert interpret_local_verdict(None) is None + + +def test_non_boolean_ok_is_rejected(): + from verification import interpret_local_verdict + assert interpret_local_verdict({"ok": "yes"}) is None + assert interpret_local_verdict({"reason": "no verdict"}) is None + + +def test_local_verdict_maps_to_a_check(): + from verification import interpret_local_verdict + assert interpret_local_verdict({"ok": True, "reason": "fine"}).verdict == "ok" + assert interpret_local_verdict({"ok": False, "reason": "empty"}).verdict == "malformed" + + +# --- the size gate is economics, not taste -------------------------------- + +def test_small_answers_are_not_worth_checking(): + # A local check costs ~15% of a median 193-token answer, so it only pays + # if such answers fail more than ~15% of the time + from verification import worth_local_check + assert worth_local_check("unverifiable", 193, 600) is False + assert worth_local_check("unverifiable", 599, 600) is False + + +def test_large_answers_are_worth_checking(): + from verification import worth_local_check + assert worth_local_check("unverifiable", 600, 600) is True + assert worth_local_check("unverifiable", 4000, 600) is True + + +def test_structurally_decided_responses_skip_the_fuzzy_check(): + # If the code already parsed (or failed to), that verdict is exact and + # free; a fuzzy second opinion adds nothing and costs real time + from verification import worth_local_check + for verdict in ("ok", "malformed", "truncated"): + assert worth_local_check(verdict, 5000, 600) is False + + +def test_missing_token_count_does_not_trigger_a_check(): + from verification import worth_local_check + assert worth_local_check("unverifiable", None, 600) is False + + +# --- excerpting must not manufacture truncation --------------------------- + +def test_excerpt_leaves_short_text_alone(): + from verification import excerpt + assert excerpt("short", 4000) == "short" + + +def test_excerpt_preserves_the_real_ending(): + # The bug this guards: head-truncating a complete 6,637-char answer to + # 4,000 made it end mid-sentence, and the local checker correctly reported + # "cut off mid-thought" — a false failure created by the harness. + from verification import excerpt + text = "START" + ("x" * 10_000) + "THE ACTUAL ENDING." + out = excerpt(text, 4000) + assert out.startswith("START") + assert out.endswith("THE ACTUAL ENDING.") + assert "elided" in out + + +def test_excerpt_labels_the_cut_so_the_checker_knows(): + from verification import excerpt + out = excerpt("a" * 10_000, 1000) + assert "characters elided" in out + # Budget is respected apart from the marker itself + assert len(out) < 1000 + 100 + + +def test_prompt_carries_both_sides_with_the_real_ending(): + from verification import build_local_verify_prompt + answer = "A" * 9000 + "FINAL SENTENCE." + prompt = build_local_verify_prompt("do a thing", answer, limit=2000) + assert "FINAL SENTENCE." in prompt + assert "do a thing" in prompt + + +def test_empty_response_is_decided_in_code_not_by_a_model(): + # Asking the local 9.7B checker about an empty answer returned ok=true + # with the reason "Answer is too short" — a self-contradiction. This is + # settled by an if-statement instead. + assert verify_response("").verdict == "malformed" + assert verify_response(" \n\t ").verdict == "malformed" + assert verify_response("").failed + + +def test_whitespace_only_beats_the_no_blocks_path(): + # It must not fall through to 'unverifiable' just for lacking code fences + assert verify_response("\n\n").verdict != "unverifiable" diff --git a/verification.py b/verification.py index d48d48b..4bf4dad 100644 --- a/verification.py +++ b/verification.py @@ -172,6 +172,13 @@ def verify_response( if finish_reason == "length": return Verification("truncated", detail="finish_reason=length") + # Decided in code, not by a model. An empty answer is unambiguously a + # failure, and asking a 9.7B model about it produced a verdict of "ok" + # with the reason "Answer is too short" — self-contradictory. Never ask a + # model what an if-statement can settle. + if not (text or "").strip(): + return Verification("malformed", detail="empty response") + blocks = extract_blocks(text) if not blocks: return Verification("unverifiable", detail="no fenced blocks in response") @@ -192,3 +199,118 @@ def verify_response( return Verification( "unverifiable", checks=checks, detail="no block had a structural check" ) + + +# --- local LLM verification (for responses nothing structural can check) --- + +VERDICT_JSON_RE = re.compile(r"\{.*\}", re.DOTALL) + +LOCAL_VERIFY_SYSTEM = ( + "You check whether an assistant's answer actually addresses the user's " + "request. Reply with ONLY a JSON object: " + '{"ok": true|false, "reason": ""}. ' + "Answer false ONLY for a clear failure: the answer is empty, refuses " + "without cause, ENDS mid-sentence, contradicts itself, or responds to a " + "different question. Style, brevity and debatable choices are NOT " + "failures. When unsure, answer true.\n" + "IMPORTANT: a marker reading '[... N characters elided from the middle " + "...]' is this harness shortening a long answer so you can read it. It is " + "NOT a defect. Ignore it and judge only the beginning and the ending you " + "were given." +) + + +ELISION = "\n\n[... {n} characters elided from the middle ...]\n\n" + + +def excerpt(text: str, limit: int) -> str: + """Shorten text for the checker WITHOUT making it look truncated. + + Naive head-truncation is not safe here. A complete 6,637-character answer + cut to 4,000 reaches the checker ending mid-sentence, and it duly reports + "cut off mid-thought" — a false failure caused entirely by the harness. + That happened on the first live run, and had it been wired into + proficiency it would have recorded every long answer as a failure. + + So the middle is dropped instead of the tail, and the cut is labelled, so + the ending the checker judges is the real ending. + """ + if not text or len(text) <= limit: + return text + head = limit // 2 + tail = limit - head + dropped = len(text) - limit + return text[:head] + ELISION.format(n=dropped) + text[-tail:] + + +def build_local_verify_prompt(request_text: str, answer: str, limit: int = 4000) -> str: + """Frame one answer for the local checker. + + Both sides are shortened — the local model is small and slow, and a check + that reads 100k tokens costs more than the answer it guards — but via + ``excerpt``, so an elision is never mistaken for a truncated answer. + """ + return ( + f"USER'S REQUEST:\n{excerpt(request_text, limit)}\n\n" + f"ASSISTANT'S ANSWER:\n{excerpt(answer, limit)}" + ) + + +def parse_verdict_json(raw: str) -> Optional[dict]: + """Pull the verdict object out of a local model's reply. + + Same defence as the eval judge: local models are reasoning models and leak + their thinking into the content despite response_format, so the object + usually arrives wrapped in prose. Returns None when nothing usable came + back, so the caller records no sample rather than a false verdict. + """ + if not raw: + return None + candidates = [raw] + match = VERDICT_JSON_RE.search(raw) + if match: + candidates.append(match.group(0)) + for candidate in candidates: + try: + parsed = json.loads(candidate) + except json.JSONDecodeError: + continue + if isinstance(parsed, dict) and "ok" in parsed: + return parsed + return None + + +def interpret_local_verdict(parsed: Optional[dict]) -> Optional[Check]: + """Turn a parsed local verdict into a Check, or None if unusable. + + A malfunctioning checker must never produce a failure verdict: that would + charge the model for the checker's problem, which is the mistake this + project already made once with the eval judge. + """ + if parsed is None: + return None + ok = parsed.get("ok") + if not isinstance(ok, bool): + return None + reason = str(parsed.get("reason", ""))[:120] + return Check("local_llm", "ok" if ok else "malformed", reason) + + +def worth_local_check( + verdict: Verdict, completion_tokens: Optional[int], min_tokens: int +) -> bool: + """Whether a local LLM check earns its cost on this response. + + Only for responses nothing structural could judge — if the code already + parsed, or failed to, that verdict is exact and free and a fuzzy second + opinion adds nothing. + + The size gate is economics, measured on real traffic: a local check costs + about 15% of a median 193-token answer, so it pays only if such answers + fail more than ~15% of the time. On a 1,500-token answer the break-even + failure rate drops to ~1.9%, which is plausible. Small answers are simply + not worth checking. + """ + if verdict != "unverifiable": + return False + return bool(completion_tokens and completion_tokens >= min_tokens) -- 2.49.1 From 6c6a8a0edc6943ed368f3f541e607db4ebadeb49 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 21:47:30 -0400 Subject: [PATCH 12/20] feat: fold observed verification failures back into proficiency The eval harness measures models on a fixed 23-task benchmark. feedback.py measures them on real traffic, which is more predictive of routing quality and accumulates for free as you work. Proven end to end: a truncated response dropped qwen3.6-35b's coding_general from 1.00 to 0.75 while leaving coding_refactor untouched. Only FAILURES are folded in, deliberately. A structural 'ok' means the code parsed, not that it was correct -- a model emitting syntactically valid nonsense would score 1.0. Recording passes would flood self_eval_score with 1.0 samples and wash out the benchmark's discrimination; the coding categories already sit at 1.00 for every model and this would spread that flatness everywhere. A model that never fails keeps its benchmark score untouched; one that does is penalized in proportion to how often. Testing it exposed a flaw in the design. The failure I forced was caused by MY OWN max_tokens=40, not by the model -- and without a fix, any agent setting a tight cap would systematically drag down whatever model it routed to. Now a truncation under a client-supplied max_tokens is recorded (the response really was unusable) but marked model_attributable=0 and excluded from feedback. The score that penalty wrongly cost has been restored. applied_at makes application idempotent, so re-running cannot punish a model repeatedly for the same bad response. Tests 193 -> 207. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 34 +++++++- dispatcher.py | 10 ++- feedback.py | 147 ++++++++++++++++++++++++++++++++ schema.sql | 11 ++- tests/test_feedback.py | 188 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 384 insertions(+), 6 deletions(-) create mode 100644 feedback.py create mode 100644 tests/test_feedback.py diff --git a/CLAUDE.md b/CLAUDE.md index 1d52776..7cd59a7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -301,6 +301,32 @@ fails. It immediately caught a check where the expected value was simply wrong — which would have docked every model on a task and been indistinguishable from genuine difficulty. +## Verification: what local compute is actually good for + +Local inference is a poor substitute for cloud completions here — 7-82x the +energy, ~670x the carbon on Michigan's grid, and slower (6.2s vs 1.4-2.0s). +But it is very good at stopping a cloud completion from being wasted, and +completions are where all the money is: fitted on real traffic, a completion +token costs **201x** a prompt token. + +| check | cost | what it catches | +|---|---|---| +| structural (`verification.py`) | **free** | truncation, malformed code/JSON/YAML, empty answers | +| local LLM (Ollama) | ~$6.9e-05, ~6s | refusals, wrong-question answers, incoherence | + +Structural checks run on every response and never execute the code — they +parse it. The local LLM check runs only on answers above +`verification.min_completion_tokens`, in the background after the client has +its response, because it costs ~15% of a median 193-token answer and only +pays above ~600 tokens. + +`feedback.py` folds observed failures into `proficiency`, so routing learns +from your traffic rather than only the 23-task benchmark. Only **failures** +are folded in: a structural 'ok' means the code parsed, not that it was +correct, and recording those as 1.0 would flatten every score toward the +ceiling. Failures the model did not cause — a client's own tight `max_tokens` +truncating the answer — are recorded but excluded. + ## The classifier is the latency floor Every routed request pays a full local classification round-trip before a @@ -359,9 +385,11 @@ that request was two orders of magnitude low. `seed_energy.py --models ... --samples 21` for those before trusting where they land. -3. **Escalation feedback loop** — low-confidence tier bumping works - (`apply_escalation`), but there's no way for a human or downstream agent to - flag "this was routed wrong" after the fact and retry one tier up. +3. **Acting on verification.** Every response is now checked and failures + feed back into proficiency (`verification.py`, `feedback.py`), but nothing + retries yet — the loop observes and learns, it does not intervene. Whether + to auto-escalate on a failed check should be decided from the observed + failure rate, which is now being collected. ## Known open questions diff --git a/dispatcher.py b/dispatcher.py index 57dce64..356cab2 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -523,6 +523,7 @@ def log_verification( verdict: str, detail: str, completion_tokens: Optional[int], + model_attributable: bool = True, ) -> None: """Record what a check said about one completion. @@ -536,8 +537,8 @@ def log_verification( """ INSERT INTO verifications ( model_id, provider, task_category, kind, verdict, detail, - completion_tokens, observed_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + completion_tokens, observed_at, model_attributable + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( model_id, @@ -548,6 +549,7 @@ def log_verification( detail[:300] if detail else None, completion_tokens, datetime.now(timezone.utc).isoformat(), + int(model_attributable), ), ) conn.commit() @@ -938,6 +940,8 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): if not api_key: raise HTTPException(503, f"{settings.api_key_env} is not set.") + # A truncated answer under a cap the CLIENT chose is not the model failing. + client_capped = body.get("max_tokens") is not None upstream_body = {**body, "model": target} streaming = bool(body.get("stream")) if streaming: @@ -970,6 +974,7 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): log_verification( target, provider, category, "structural", result.verdict, result.detail, completion_tokens, + model_attributable=not client_capped, ) if cfg.verification.local_llm_enabled and worth_local_check( result.verdict, completion_tokens, cfg.verification.min_completion_tokens @@ -1045,6 +1050,7 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): log_verification( target, provider, category, "structural", result.verdict, result.detail, ct, + model_attributable=not client_capped, ) # Inline rather than a background task: the generator has already # finished streaming, so the client is not waiting on this. diff --git a/feedback.py b/feedback.py new file mode 100644 index 0000000..8012e0c --- /dev/null +++ b/feedback.py @@ -0,0 +1,147 @@ +#!/usr/bin/env python3 +"""Fold observed verification failures back into proficiency. + +The eval harness measures models on a fixed 23-task benchmark. This measures +them on YOUR traffic, which is more predictive of routing quality and +accumulates for free as you work. + + python feedback.py --dry-run # show what would change + python feedback.py # apply + +**Only failures are folded in, deliberately.** A verification pass is weak +evidence: structural 'ok' means the code parsed, not that it was correct, and +a model emitting syntactically valid nonsense would score 1.0. Recording those +passes would flood `self_eval_score` with 1.0 samples and wash out the +benchmark's hard-won discrimination — the coding categories already sit at +1.00 for every model, and this would spread that flatness everywhere. + +A failure is the opposite: a truncated, malformed or empty response is +definitive, and its cause does not matter for routing. So a model that never +fails keeps its benchmark score untouched, and a model that does fail is +penalized in proportion to how often. That asymmetry is the point. + +Each failure is applied once. `applied_at` marks consumed rows so re-running +cannot penalize a model repeatedly for the same bad response. +""" + +from __future__ import annotations + +import argparse +import sqlite3 +import sys +from collections import defaultdict + +from config import load_config +from proficiency_store import add_self_eval + +# Verdicts that count as observed failures. 'unverifiable' is excluded: it +# means the checker had nothing to say, which is not evidence about the model. +FAILURE_VERDICTS = ("truncated", "malformed") + +# Failures the model did not cause are excluded. The obvious case: a client +# that sets max_tokens=40 and gets a truncated answer caused that itself, and +# counting it would let any agent with a tight cap drag down whatever model it +# happened to route to. Found by forcing exactly that during testing. + + +def unapplied_failures(conn: sqlite3.Connection) -> list[sqlite3.Row]: + conn.row_factory = sqlite3.Row + placeholders = ",".join("?" * len(FAILURE_VERDICTS)) + return conn.execute( + f""" + SELECT id, model_id, provider, task_category, kind, verdict, detail + FROM verifications + WHERE verdict IN ({placeholders}) + AND applied_at IS NULL + AND task_category IS NOT NULL + AND model_attributable = 1 + ORDER BY id + """, + FAILURE_VERDICTS, + ).fetchall() + + +def summarize(rows: list[sqlite3.Row]) -> dict[tuple[str, str, str], list[int]]: + """Group failures by (model, provider, category).""" + grouped: dict[tuple[str, str, str], list[int]] = defaultdict(list) + for r in rows: + grouped[(r["model_id"], r["provider"], r["task_category"])].append(r["id"]) + return grouped + + +def apply_failures(conn: sqlite3.Connection, cfg, grouped, dry_run: bool) -> int: + applied = 0 + for (model_id, provider, category), ids in sorted(grouped.items()): + print(f" {model_id:24s} {category:18s} {len(ids)} failure(s)") + if dry_run: + continue + # One 0.0 sample per observed failure, folded into the running mean by + # proficiency.accumulate — so the penalty scales with failure rate + # rather than replacing the benchmark outright. + add_self_eval(conn, cfg, model_id, provider, category, [0.0] * len(ids)) + conn.executemany( + "UPDATE verifications SET applied_at = datetime('now') WHERE id = ?", + [(i,) for i in ids], + ) + applied += len(ids) + if not dry_run: + conn.commit() + return applied + + +def coverage(conn: sqlite3.Connection) -> None: + """Report what verification has seen, so its usefulness stays visible.""" + conn.row_factory = sqlite3.Row + rows = conn.execute( + """ + SELECT kind, verdict, COUNT(*) n FROM verifications + GROUP BY kind, verdict ORDER BY kind, verdict + """ + ).fetchall() + if not rows: + print(" no verifications recorded yet") + return + total = sum(r["n"] for r in rows) + print(f" {'kind':12s}{'verdict':16s}{'n':>6}{'share':>9}") + for r in rows: + print(f" {r['kind']:12s}{r['verdict']:16s}{r['n']:>6}{r['n']/total*100:>8.1f}%") + unver = sum(r["n"] for r in rows if r["verdict"] == "unverifiable") + if total and unver / total > 0.8: + print( + f"\n NOTE: {unver/total*100:.0f}% of responses were unverifiable. " + "Structural checking is not earning much here —\n" + " most traffic is prose. The local LLM check covers that, but only " + "above the size threshold." + ) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--dry-run", action="store_true") + args = ap.parse_args() + + cfg = load_config("config.yaml") + conn = sqlite3.connect(cfg.database.path) + + print("verification coverage so far:") + coverage(conn) + print() + + rows = unapplied_failures(conn) + if not rows: + print("no unapplied failures — nothing to fold in") + conn.close() + return 0 + + grouped = summarize(rows) + print(f"{'would apply' if args.dry_run else 'applying'} " + f"{len(rows)} failure(s) across {len(grouped)} (model, category) pair(s):") + applied = apply_failures(conn, cfg, grouped, args.dry_run) + conn.close() + if not args.dry_run: + print(f"\napplied {applied} failure sample(s)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/schema.sql b/schema.sql index 2d36870..25197e2 100644 --- a/schema.sql +++ b/schema.sql @@ -141,7 +141,16 @@ CREATE TABLE IF NOT EXISTS verifications ( verdict TEXT NOT NULL, -- 'ok' | 'truncated' | 'malformed' | 'unverifiable' detail TEXT, completion_tokens INTEGER, -- what a wasted answer cost, for the payoff sum - observed_at TEXT NOT NULL + observed_at TEXT NOT NULL, + -- Set once feedback.py has folded this row into proficiency, so re-running + -- cannot penalize a model repeatedly for the same bad response. + applied_at TEXT, + -- Whether this failure is the MODEL's fault. A client that sets a tight + -- max_tokens and gets a truncated answer caused that itself; counting it + -- against the model would let any agent with a small cap systematically + -- drag down whatever it routed to. Still recorded -- the response really + -- was unusable -- but excluded from proficiency feedback. + model_attributable INTEGER DEFAULT 1 ); CREATE INDEX IF NOT EXISTS idx_verifications_model ON verifications (model_id, provider); diff --git a/tests/test_feedback.py b/tests/test_feedback.py new file mode 100644 index 0000000..aafc58c --- /dev/null +++ b/tests/test_feedback.py @@ -0,0 +1,188 @@ +"""Tests for feedback.py — folding observed failures into proficiency. + +The design decision under test is the asymmetry: failures are recorded, +passes are not. A structural 'ok' only means the code parsed, so treating it +as a quality sample would flood self_eval_score with 1.0s and wash out the +benchmark's discrimination — the coding categories already sit at 1.00 for +every model, and this would spread that flatness everywhere. +""" + +import sqlite3 +from pathlib import Path + +import pytest + +from config import load_config +from feedback import FAILURE_VERDICTS, apply_failures, summarize, unapplied_failures + +ROOT = Path(__file__).resolve().parent.parent +SCHEMA_SQL = (ROOT / "schema.sql").read_text() +CFG = load_config(ROOT / "config.yaml") + + +@pytest.fixture +def db(tmp_path): + conn = sqlite3.connect(tmp_path / "t.db") + conn.row_factory = sqlite3.Row + conn.executescript(SCHEMA_SQL) + conn.execute( + """ + INSERT INTO models (model_id, provider, base_model_id, availability, last_updated) + VALUES ('m', 'nw', 'm', 'active', '2026-08-17T00:00:00+00:00') + """ + ) + conn.commit() + yield conn + conn.close() + + +def _verify(conn, verdict, category="coding_general", kind="structural"): + conn.execute( + """ + INSERT INTO verifications (model_id, provider, task_category, kind, verdict, observed_at) + VALUES ('m', 'nw', ?, ?, ?, '2026-08-17T00:00:00+00:00') + """, + (category, kind, verdict), + ) + conn.commit() + + +def _prof(conn): + return conn.execute( + "SELECT self_eval_score s, self_eval_samples n FROM proficiency WHERE model_id='m'" + ).fetchone() + + +# --- only failures are folded in ------------------------------------------ + +def test_passes_are_not_recorded_as_samples(db): + # A structural 'ok' means "it parsed", not "it was correct". Recording it + # as a 1.0 would inflate every model toward the ceiling. + for _ in range(5): + _verify(db, "ok") + assert unapplied_failures(db) == [] + + +def test_unverifiable_is_not_evidence(db): + # The checker had nothing to say; that says nothing about the model + for _ in range(5): + _verify(db, "unverifiable") + assert unapplied_failures(db) == [] + + +@pytest.mark.parametrize("verdict", FAILURE_VERDICTS) +def test_failures_are_collected(db, verdict): + _verify(db, verdict) + assert len(unapplied_failures(db)) == 1 + + +def test_a_failure_drags_the_score_down_proportionally(db): + from proficiency_store import add_self_eval + + # Given: a benchmark score of 0.9 over 9 samples + add_self_eval(db, CFG, "m", "nw", "coding_general", [0.9] * 9) + _verify(db, "malformed") + + apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False) + row = _prof(db) + # One 0.0 folded into the running mean: (0.9*9 + 0)/10 + assert row["n"] == 10 + assert row["s"] == pytest.approx(0.81) + + +def test_a_model_that_never_fails_keeps_its_benchmark_score(db): + from proficiency_store import add_self_eval + + add_self_eval(db, CFG, "m", "nw", "coding_general", [0.9] * 9) + for _ in range(20): + _verify(db, "ok") + apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False) + row = _prof(db) + assert (row["n"], row["s"]) == (9, pytest.approx(0.9)) + + +# --- idempotence ---------------------------------------------------------- + +def test_a_failure_is_applied_only_once(db): + _verify(db, "malformed") + apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False) + first = _prof(db)["n"] + # Re-running must not punish the model again for the same bad response + assert unapplied_failures(db) == [] + apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False) + assert _prof(db)["n"] == first + + +def test_dry_run_changes_nothing(db): + _verify(db, "truncated") + apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=True) + assert _prof(db) is None + assert len(unapplied_failures(db)) == 1 + + +# --- grouping ------------------------------------------------------------- + +def test_failures_group_by_model_and_category(db): + _verify(db, "malformed", category="coding_general") + _verify(db, "malformed", category="coding_general") + _verify(db, "truncated", category="debugging") + grouped = summarize(unapplied_failures(db)) + assert {k[2]: len(v) for k, v in grouped.items()} == { + "coding_general": 2, + "debugging": 1, + } + + +def test_failures_without_a_category_are_skipped(db): + # Nothing to attribute them to — proficiency is per-category + db.execute( + """ + INSERT INTO verifications (model_id, provider, task_category, kind, verdict, observed_at) + VALUES ('m', 'nw', NULL, 'structural', 'malformed', '2026-08-17T00:00:00+00:00') + """ + ) + db.commit() + assert unapplied_failures(db) == [] + + +def test_both_check_kinds_count(db): + _verify(db, "malformed", kind="structural") + _verify(db, "malformed", kind="local_llm") + assert len(unapplied_failures(db)) == 2 + + +# --- attribution: not every failure is the model's fault ------------------- + +def _verify_capped(conn, verdict="truncated", category="coding_general"): + conn.execute( + """ + INSERT INTO verifications (model_id, provider, task_category, kind, verdict, + observed_at, model_attributable) + VALUES ('m', 'nw', ?, 'structural', ?, '2026-08-17T00:00:00+00:00', 0) + """, + (category, verdict), + ) + conn.commit() + + +def test_client_capped_truncation_is_not_the_models_fault(db): + # Found by forcing it: a request with max_tokens=40 truncates, and without + # this any agent using a tight cap would systematically drag down whatever + # model it routed to. + _verify_capped(db) + assert unapplied_failures(db) == [] + + +def test_capped_failures_are_still_recorded_for_visibility(db): + # The response really was unusable — it just says nothing about the model + _verify_capped(db) + n = db.execute("SELECT COUNT(*) c FROM verifications WHERE verdict='truncated'").fetchone()["c"] + assert n == 1 + + +def test_attributable_and_capped_failures_are_separated(db): + _verify(db, "malformed") # model's fault + _verify_capped(db) # client's cap + rows = unapplied_failures(db) + assert len(rows) == 1 + assert rows[0]["verdict"] == "malformed" -- 2.49.1 From dac40c8fb226b25fed3f8e6f8145d6b8acd79126 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 21:50:43 -0400 Subject: [PATCH 13/20] docs: refresh module list and test counts (207 across 11 files) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 2 +- README.md | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7cd59a7..f28c5fe 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -180,7 +180,7 @@ provisional. A single sweep's ranking is one sample of a moving quantity. `code` executes the model's Python against checks, `exact` compares a normalized answer, `tool` inspects the tool call structurally, and only the four prose categories fall back to a `judge`. -- `tests/` — 156 tests across 9 files, all passing. +- `tests/` — 207 tests across 11 files, all passing. ### Serving class: one base model, many rows diff --git a/README.md b/README.md index cecc7e4..9a683f4 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ later without a migration. | **Config** | `config.yaml` loaded & validated by Pydantic (`config.py`) | | **OpenAI Client** | `openai==3.0.0` (official SDK) | | **HTTP** | `requests` for poller, `httpx` (via openai/uvicorn) | -| **Testing** | `pytest` — 156 tests across 9 files | +| **Testing** | `pytest` — 207 tests across 11 files | | **Config Files** | `config.yaml`, `leaderboards.yaml`, `evals/tasks.yaml` | | **Deployment** | systemd user units (`.service` + `.timer` files in `deploy/`) | | **Integration** | Pre-configured in `opencode.json` — `cd` into repo uses router by default | @@ -103,6 +103,8 @@ restarts on boot shouldn't change its dependency tree underneath itself. | **`proficiency_store.py`** | DB access for `proficiency` table; single write path ensuring `blended_score` never drifts | Yes (DB) | | **`seed_energy.py`** | Reference workload sweep: fixed prompt × N runs per model | Yes (network, DB) | | **`eval_proficiency.py`** | Self-eval harness: 4 scoring kinds against N models | Yes (network, DB, subprocess) | +| **`verification.py`** | Structural checks on responses: parses code/JSON/YAML, detects truncation. Never executes model output | Pure | +| **`feedback.py`** | Folds observed verification failures into `proficiency` so routing learns from real traffic | Yes (DB) | | **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) | | **`config.py`** | YAML loader + Pydantic validators (weights sum to 1, valid tiers, etc.) | Yes (file) | @@ -355,7 +357,7 @@ Virtual model names: ## Testing ```bash -python -m pytest # run all 156 tests +python -m pytest # run all 207 tests python -m pytest --cov # with coverage ``` -- 2.49.1 From cdb186348a932c8d739754226eb4c9bb453a9703 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 22:00:03 -0400 Subject: [PATCH 14/20] docs: correct a stale limitation and document the single-model routing outcome Two gaps in the README, found by checking it against the running system. The "no escalation feedback" limitation undersold what shipped. Flagging a bad response DOES exist now -- verification.py detects failures and feedback.py folds them into proficiency automatically. What is missing is only the action: nothing retries or escalates. That is deliberate, since whether auto-escalation pays should come from the observed failure rate now being collected. More usefully, the README said nothing about the fact that all 27 routing decisions currently return qwen3.6-35b. A reader would reasonably expect traffic to spread across models and go hunting for a misconfiguration. It is Pareto-dominant on this catalog: cheapest AND cleanest, within 0.15 of the best proficiency. On summarization it costs 7x less and emits 65x less carbon than kimi-k3, which beats it 1.00 to 0.85 -- no defensible weighting picks kimi-k3 there. Also records what is easy to miss: proficiency still reorders the runners-up by category, and the field widens once the leader stops being eligible -- past ~94,196 tokens qwen3.6-35b is filtered out and glm-5.2-fast takes over, past ~790K nothing qualifies and the request 422s. On a large codebase the model will change mid-session, and that is the context filter working. Every figure verified against the live database before writing. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- README.md | 297 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 254 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index 9a683f4..044227e 100644 --- a/README.md +++ b/README.md @@ -11,21 +11,120 @@ Neuralwatt is the only provider. The `provider` column and the `(model_id, provider)` primary key stay so a second provider can be added later without a migration. +## At a Glance + +| Dimension | Detail | +|---|---| +| **Cost model** | Per-kWh, not per-token. Flat $8.00/kWh measured across the catalog. | +| **Latency** | ~4 s warm classification, up to ~120 s on cold start. Dispatcher response is unaffected — verification runs async. | +| **Quality** | Three-axis scoring: cost (0.4) + eco (0.2) + proficiency per category (0.4). Proficiency blends external benchmarks with a self-run eval harness. | +| **Verification** | Every response structurally checked (free) + local LLM spot-check on long prose answers (~6 s, never blocks). Failures fold back into proficiency via `feedback.py`. | +| **Fault tolerance** | Classifier failure degrades to a mid-tier fallback rather than 502/503. Retry count is zero to prevent silent 3× time-outs. | +| **API surface** | OpenAI-compatible `/v1` endpoints, streaming chunk proxy with SSE telemetry scraping. | + +## Verification Pipeline + +Every completion passes through a two-layer check **before** routing learns from +it and **without** adding to client-facing latency for the async layer. + +### Structural Verification — `verification.py` + +Free, exact checks on what a model just returned — no code execution. +Extracts fenced code blocks and validates them by **parsing**, not running: + +| Language | Check | Safeguard | +|---|---|---| +| Python (`python`, `py`, `python3`) | `ast.parse` | Syntax tree, no evaluation | +| JSON (`json`, `jsonc`) | `json.loads` | Strict parse | +| YAML (`yaml`, `yml`) | `yaml.safe_load` | No arbitrary objects | +| Shell (`bash`, `sh`, `zsh`) | `bash -n` (caller's job) | Parse-only, never runs | +| Other | `unverifiable` (not a failure) | Prose answers land here | + +Plus two universal checks: +- **`finish_reason == "length"`** — decisive truncation, even if parsed content + looks fine (the most dangerous case: a fragment that parses) +- **Unterminated code fences** — the response ran out of tokens mid-block + +**Key design decisions:** +- Never executes model output. The eval harness (`eval_proficiency.py`) does + run generated code, but there the prompts are ones this project authored, + so what comes back is bounded. Here the code is whatever the user asked for + and could do anything. +- `unverifiable` is **not** a failure. Most prose answers land there (correctly + — there's nothing structural to check), and counting it as wrong would + penalize models for the checker's limits. +- Measured on real traffic: a wasted cloud completion costs about 52 local + checks at 1,500 tokens and 139 at 4,000. A zero-cost check pays trivially. + +### Local LLM Verification — `verification.py` (cont.) + +For answers nothing structural can judge (prose, reasoning, refusals), a local +model spot-checks the response **after** it has gone back to the client, so its +~6 s never lands on anyone's latency. + +The prompt frames the local model as a judge: "Does this answer actually +address the user's request?" and returns `{"ok": true|false, "reason": "..."}`. + +Gating and safeguards: +- **Size threshold** — only responses ≥ 600 completion tokens trigger a local + check (configurable via `verification.min_completion_tokens`). A check on + a 193-token answer costs ~15% of the answer; at 1,500 tokens the break-even + failure rate drops to ~1.9%. +- **Middle elision** — if the answer is longer than the limit (default 4,000), + the **middle** is elided, not the head, so the checker judges the real ending. + An elision marker tells the checker not to flag it as a defect. +- **Thinking disabled** — the local model is a reasoning model and will otherwise + emit unbounded chain-of-thought. Verification uses Ollama's native endpoint + with `think=False`, so it answers in ~25 tokens instead of burning through + the budget on reasoning. +- **Malfunction safety** — an empty or unparseable verdict produces no sample + rather than a false failure. An empty response is settled by an if-statement, + never asked of a model what code can decide. + +### Feedback Loop — `feedback.py` + +Observation → learning. `feedback.py` folds verification failures into +`proficiency` so routing improves on **your traffic**, not just the fixed +23-task benchmark: + +```bash +python feedback.py --dry-run # preview what would change +python feedback.py # apply +``` + +Key behaviors: +- **Only failures** are folded in. A structural 'ok' means the code parsed, + not that it was correct — scoring a parse-success as a proficiency 1.0 would + flatten every score toward the ceiling. The coding categories already sit + at 1.00 for every model; this would spread that flatness everywhere. +- **One 0.0 sample per failure**, added to the running mean so the penalty + scales with failure rate rather than replacing the benchmark score outright. +- **Idempotent** — `applied_at` marks consumed rows so re-running cannot + penalize a model repeatedly for the same response. +- **Not the model's fault** — a client that sets `max_tokens=40` and gets a + truncated answer caused that itself. Those failures are recorded in the + verifications table but excluded from proficiency feedback. + ## Architecture ``` ┌─────────────────────┐ incoming task ───▶│ Local Classifier │ Ollama (qwen3.5:latest) - │ - task_category │ - │ - task_tier │ ~4s warm, ~120s cold cap - │ - required_context │ temperature: 0, max 1024 tokens + │ - task_category │ ~4s warm, ~120s cold cap + │ - task_tier │ temperature: 0, max 1024 tokens + │ - required_context │ max_retries: 0 (silent 3× cap guard) │ - confidence │ └──────────┬───────────┘ ▼ - ┌─────────────────┐ - │ Escalation │ low-confidence tier bump - │ (optional) │ threshold configurable - └────────┬─────────┘ + ┌─────────────────────┐ + │ Classifier Fallback │ tier 2 / general_chat on failure + │ (graceful degrade) │ — not a 502/503 + └──────────┬───────────┘ + ▼ + ┌─────────────────┐ + │ Escalation │ low-confidence tier bump + │ (optional) │ threshold: 0.6 confidence + └────────┬─────────┘ ▼ ┌─────────────────────┐ │ Hard Filters │ context window ≥ required @@ -40,17 +139,23 @@ later without a migration. │ (scoring.py) │ └──────────┬───────────┘ ▼ - ┌─────────────────────┐ - │ Dispatcher │──▶ Neuralwatt Cloud - │ (FastAPI API) │──▶ OpenAI-compatible /v1 endpoints - │ │ Streaming chunk proxy w/ - │ │ SSE comment telemetry scrape - └──────────┬───────────┘ - ▼ - ┌─────────────────────┐ - │ Energy Logging │ per-request cost, kWh, gCO2eq - │ (SQLite) │ seeds median stats for routing - └─────────────────────┘ + ┌──────────────────────┐ + │ Dispatcher / Provider │──▶ Neuralwatt Cloud + │ (FastAPI API) │──▶ OpenAI-compatible /v1 + │ │ Streaming SSE + energy scrape + └───────┬──────────────┘ + ▼ + ┌─────────────────────────┐ + │ Verification Pipeline │ + │ Layer 1: Structural │ ast.parse, json.load, yaml.safe_load + │ Layer 2: Local LLM │ async, ~6s, >600 token gate + └──────┬──────────────────┘ + │ + ▼ + ┌─────────────────────┐ + │ feedback.py │ failures → proficiency updates + │ (on-demand agent) │ idempotent, attributed-only + └─────────────────────┘ ``` ## Tech Stack @@ -59,7 +164,7 @@ later without a migration. |---|---| | **Language** | Python 3 (IO-bound provider APIs; iteration speed matters more than raw speed) | | **Framework** | FastAPI + uvicorn | -| **Database** | SQLite (`router.db`) — decision table, energy observations, proficiency | +| **Database** | SQLite (`router.db`) — decision table, energy observations, proficiency, verifications | | **Local Classification** | Ollama (OpenAI-compatible at `localhost:11434/v1`) | | **Local Model** | `qwen3.5:latest` | | **Cloud Provider** | Neuralwatt only | @@ -103,14 +208,14 @@ restarts on boot shouldn't change its dependency tree underneath itself. | **`proficiency_store.py`** | DB access for `proficiency` table; single write path ensuring `blended_score` never drifts | Yes (DB) | | **`seed_energy.py`** | Reference workload sweep: fixed prompt × N runs per model | Yes (network, DB) | | **`eval_proficiency.py`** | Self-eval harness: 4 scoring kinds against N models | Yes (network, DB, subprocess) | -| **`verification.py`** | Structural checks on responses: parses code/JSON/YAML, detects truncation. Never executes model output | Pure | -| **`feedback.py`** | Folds observed verification failures into `proficiency` so routing learns from real traffic | Yes (DB) | +| **`verification.py`** | Two-layer check: structural parse (always) + local LLM spot-check (async, size-gated). Never executes model output | Pure | +| **`feedback.py`** | Folds observed verification failures into `proficiency`; routing learns from real traffic | Yes (DB) | | **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) | | **`config.py`** | YAML loader + Pydantic validators (weights sum to 1, valid tiers, etc.) | Yes (file) | ## Decision Table Schema (SQLite) -Three tables, `PRAGMA foreign_keys = ON`: +Three data tables plus one observability table, `PRAGMA foreign_keys = ON`: ### `models` — one row per served model variant @@ -227,6 +332,26 @@ calls varied 20×), but ranks 750× between models while within-model spread is scores on the attributed figures with a **median** over all `seed_reference` rows, so repeated sweeps accumulate into a median-across-time. +### `verifications` — response quality observations + +| Column | Type | Notes | +|---|---|---| +| `id` | INTEGER | Autoincrement | +| `model_id` / `provider` | TEXT | | +| `task_category` | TEXT | Optional (prose answers may lack a category) | +| `kind` | TEXT | `structural` \| `local_llm` | +| `verdict` | TEXT | `ok` \| `truncated` \| `malformed` \| `unverifiable` | +| `detail` | TEXT | Human-readable reason | +| `completion_tokens` | INTEGER | Wasted answer cost, for payoff sum | +| `observed_at` | TEXT | ISO8601 | +| `applied_at` | TEXT | Set by `feedback.py` when folded into proficiency | +| `model_attributable` | INTEGER | 1 = model's fault; 0 = client caused (e.g. tight cap) | + +Verifications drive the feedback loop: `feedback.py` reads unanswered failures, +applies a 0.0 sample per failure to `proficiency`, and marks them +`applied_at` for idempotency. `unverifiable` is recorded but not treated as a +failure — it means the checker had nothing to say, not that the model failed. + ## Weighted Scoring Three hard filters are applied **before** scoring (not weighted — outright disqualification): @@ -259,6 +384,34 @@ intensity. Grid intensity spans 37 gCO2/kWh (`FI`) to 505 `kimi-k2.7-code` while emitting 3.6× more carbon. Collapsing them picks a side. +### What routing actually does today — expect one model + +Run `/route` across all 9 categories and 3 tiers and every one of those 27 +decisions currently returns **`qwen3.6-35b`**. That is not a bug and not a +misconfiguration, so it is worth stating plainly before you go looking for +one. + +On this catalog `qwen3.6-35b` is **Pareto-dominant**: cheapest *and* cleanest +in the routable set, while scoring within 0.15 of the best model on +proficiency. On `summarization` it costs 7× less and emits 65× less carbon +than `kimi-k3`, which beats it 1.00 to 0.85 on quality — no defensible +weighting picks `kimi-k3` there. One model winning is the correct answer to +the question the weights ask. + +Proficiency still does work underneath: runner-up ordering reorders by +category (`kimi-k3-fast` climbs to 3rd on `docs_writing` and `summarization` +where it scores 1.00, and leaves the top four elsewhere). The axis is live; +it simply cannot overturn a leader that wins on two axes at once. + +The spread widens when the leader stops being eligible. Past ~94,196 tokens +of context `qwen3.6-35b` is filtered out and `glm-5.2-fast` takes over; past +~790K nothing qualifies and `/v1/chat/completions` returns 422 naming the +constraint rather than truncating. So on a large codebase you will see the +model change mid-session — that is the context filter working. + +If a different balance is wanted, the lever is `weights` in `config.yaml`, +not more data. + ## API Endpoints The dispatcher binds `127.0.0.1:8080`. **No auth of its own** — loopback @@ -284,12 +437,42 @@ Two virtual router models: Ask for **any real model id** in `/v1/chat/completions` and it dispatches directly, still logged — routing is transparent, not opaque. -**Streaming:** Passed through chunk-by-chunk so tokens render as they arrive. -Neuralwatt emits energy and cost as SSE **comment** lines (`: energy {...}`) -before `data: [DONE]` — ordinary clients ignore comments, so the stream flows +**Streaming** (chunk-by-chunk proxy): Tokens render as they arrive. Neuralwatt +emits energy and cost as SSE **comment** lines (`: energy {...}`) before +`data: [DONE]` — ordinary clients ignore comments, so the stream flows untouched while the router scrapes telemetry on the way past. Without this, streamed calls would log no energy at all. +**Verification headers** (non-streaming): When streaming is not used, the +structural verification verdict surfaces in the `X-Router-Verification` header +so a client can inspect it without parsing the response body. Valid values: +`ok`, `truncated`, `malformed`, `unverifiable`, `none`. + +## Quick Usage + +```bash +# Pick a model without spending anything +curl -s -X POST localhost:8080/route -H 'content-type: application/json' \ + -d '{"task":"Refactor this 800-line Django view into service objects."}' + +# With explicit category/tier overrides (skips classifier) +curl -s -X POST localhost:8080/route -H 'content-type: application/json' \ + -d '{"task":"Refactor this 800-line Django view","task_category":"coding_refactor","task_tier":2}' + +# Actually call the winner and log energy/cost +curl -s -X POST localhost:8080/dispatch -H 'content-type: application/json' \ + -d '{"task":"What is a Python context manager?"}' + +# OpenAI-compatible — works with any SDK or agent client +curl -s localhost:8080/v1/models +curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \ + -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}' + +# Admit flex rows for overnight/async work +curl -s -X POST localhost:8080/route -H 'content-type: application/json' \ + -d '{"task":"nightly code review","latency_tolerance":"batch"}' +``` + ## Scheduled Jobs (systemd) Five user units. See `deploy/README.md` for full install/operate instructions. @@ -325,23 +508,27 @@ python eval_proficiency.py --dry-run # plan only bounded isolation, not a container. - Judge tasks skip the model being judged (avoids self-scoring bias). -## Setup +## Classifier Reliability Notes -```bash -python -m venv .venv && source .venv/bin/activate -pip install -r requirements.txt -sqlite3 router.db < schema.sql -cp .env.example .env # fill in NEURALWATT_API_KEY -python poller.py # populate the catalog -python tier.py # resolve tiers -python seed_energy.py # seed reference energy observations (~5 per model) -python -m uvicorn dispatcher:app --reload -``` +Several tuning decisions keep the classifier from cascading failures: -Ollama must be running locally: -```bash -ollama pull qwen3.5:latest -``` +- **`max_retries: 0`** on the classifier client. The OpenAI SDK retries twice + by default, so `timeout_seconds` silently became a 3× wall-clock bound — a + request could hang past 250s on a 120s setting with no log. +- **`max_output_tokens: 1024`** bounds a reasoning model's chain of thought. + Without it, `qwen3.5` emits unbounded traces — Ollama keeps generating after + the client gives up AND serializes per model, so one runaway request queues + every later request behind it. 256 was tried first and was too tight; the + trace would consume the whole budget and the model would be truncated before + emitting JSON. +- **`temperature: 0`** prevents non-reproducible classifications. At the + default, the same prompt classified tier 2 then tier 1 on consecutive calls, + routing to two different models. +- **Graceful fallback** routes to `fallback_tier: 2` / `fallback_category: + general_chat` when the classifier times out, errors, or returns + unparseable output. A coding agent would rather have a mid-tier answer than + an error. Escalation deliberately skips fallbacks so an unavailable local + model doesn't silently promote every request to the frontier tier. ## Pointing a Coding Agent at It @@ -372,6 +559,26 @@ python -m pytest --cov # with coverage | `test_load_candidates.py` | `load_candidates`: cost/eco/proficiency join | | `test_eval_scoring.py` | Code, exact, tool, and judge scoring | | `test_task_set.py` | Reference solutions validating every `code` task's checks & `exact` answers | +| `test_verification.py` | Structural checks on every language, verifier edge cases, local-LLM gating | +| `test_feedback.py` | Failure identification, idempotency, attribution filtering | + +## Setup + +```bash +python -m venv .venv && source .venv/bin/activate +pip install -r requirements.txt +sqlite3 router.db < schema.sql +cp .env.example .env # fill in NEURALWATT_API_KEY +python poller.py # populate the catalog +python tier.py # resolve tiers +python seed_energy.py # seed reference energy observations (~5 per model) +python -m uvicorn dispatcher:app --reload +``` + +Ollama must be running locally: +```bash +ollama pull qwen3.5:latest +``` ## Known Limitations & Open Items @@ -382,9 +589,13 @@ python -m pytest --cov # with coverage - **Three models unsettled** — split-half stability at 7 samples: `kimi-k2.7-code-fast` (29×), `kimi-k3` (14×), `glm-5.2-flex` (2.2×). More samples needed before their positioning is trustworthy. -- **No escalation feedback** — `apply_escalation` works (low-confidence - tier bumped), but there's no way to flag "this was routed wrong" after - the fact and retry one tier up. +- **Verification observes but never intervenes** — failures are detected and + fed back into `proficiency` automatically (`verification.py` → + `feedback.py`), so the router does learn which models fail on real work. + What is missing is action: nothing retries or escalates on a failed check. + That is deliberate — whether auto-escalation pays should be decided from + the observed failure rate, which is now being collected rather than + guessed at. - **No auth** — the service holds a billable API key with no authentication of its own. Loopback binds only. - **Context assembly (RAG)** is out of scope — the classifier sees the full -- 2.49.1 From 42b71b9bbcc399b796b408d9dcb6abd10d28368e Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 22:19:25 -0400 Subject: [PATCH 15/20] feat: quality is the objective, energy is the constraint, eco is out Replaces the three-way weighted blend with a rule that states the dual mandate honestly: maximize quality, subject to a per-request energy ceiling, tie-broken by cost. Eco is dropped from the objective (still logged per request). It was 20% of every decision optimizing a goal that is handled outside this router. Cost stops being a weight. Under the blend, 60% of each decision adjudicated fractions of a cent -- all real traffic to date totals $0.07 -- and min-max normalization made "expensive" relative to whoever else happened to be a candidate, so a model could lose for being 2x a very cheap one when both round to nothing. Measurement retired the blend outright: turning the cost weight from 0.4 to ZERO changed the winner in only 2 of 6 categories, so it was never steering on quality. The ceiling is denominated in kWh, not dollars, because the plan is a subscription with a 6.25 kWh quota. Dollars accrue and can be reconciled later; a quota is a wall you hit mid-task. /health now reports burn against the allowance and warns past 80%, with the caveat that it counts only what the router saw. quality_tolerance (0.10) is measurement noise rather than preference: proficiency rests on 2-3 samples per category, so smaller gaps are sampling variation and paying more for them buys noise. Narrow it as samples accumulate. Also corrects a figure the docs asserted as a constant. Grid intensity is not 37 gCO2/kWh: observations show FI at 49 and 50 at different times, FR at 54, US-MIDA-PJM at 442, and the provider's own 24h blended figure was 145.4. Tests 207 -> 211. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 4 +- README.md | 31 +++++---- config.py | 37 +++++++---- config.yaml | 37 +++++++++-- dispatcher.py | 58 ++++++++++++++++- routing.py | 98 +++++++++++++++++++--------- tests/test_routing.py | 148 ++++++++++++++++++++---------------------- 7 files changed, 272 insertions(+), 141 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f28c5fe..2a6f167 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -71,9 +71,9 @@ grid intensity, and models run in different regions: | grid | gCO2/kWh | models | |---|---|---| -| `FI` | 37 | most of the catalog | +| `FI` | ~49-50 | most of the catalog; varies by time | | `FI` (reported) | 475 | `glm-5.2-fast`, `glm-5.2-flex` | -| `US-MIDA-PJM` | 505 | the `kimi-k3` family | +| `US-MIDA-PJM` | ~442 | the `kimi-k3` family | A 13.6x spread, so the two axes disagree: `glm-5.2-fast` is the 2nd cheapest model and only the 6th cleanest; `kimi-k3-flex` draws 3.7x *less* energy than diff --git a/README.md b/README.md index 044227e..f59f35e 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ later without a migration. |---|---| | **Cost model** | Per-kWh, not per-token. Flat $8.00/kWh measured across the catalog. | | **Latency** | ~4 s warm classification, up to ~120 s on cold start. Dispatcher response is unaffected — verification runs async. | -| **Quality** | Three-axis scoring: cost (0.4) + eco (0.2) + proficiency per category (0.4). Proficiency blends external benchmarks with a self-run eval harness. | +| **Quality** | Quality is the objective; cost is a per-request kWh ceiling plus a tiebreak. Eco is logged but no longer optimized. Proficiency blends external benchmarks with a self-run eval harness. | | **Verification** | Every response structurally checked (free) + local LLM spot-check on long prose answers (~6 s, never blocks). Failures fold back into proficiency via `feedback.py`. | | **Fault tolerance** | Classifier failure degrades to a mid-tier fallback rather than 502/503. Retry count is zero to prevent silent 3× time-outs. | | **API surface** | OpenAI-compatible `/v1` endpoints, streaming chunk proxy with SSE telemetry scraping. | @@ -135,8 +135,9 @@ Key behaviors: └──────────┬───────────┘ ▼ ┌─────────────────────┐ - │ Weighted Scoring │ cost + eco + proficiency - │ (scoring.py) │ + │ Quality-first select │ max proficiency, cheapest + │ (routing.py) │ among equals, under a + │ │ per-request kWh ceiling └──────────┬───────────┘ ▼ ┌──────────────────────┐ @@ -361,16 +362,22 @@ Three hard filters are applied **before** scoring (not weighted — outright dis 3. Serving class compatible with request's `latency_tolerance`; `access_level` reachable ``` -composite = w_cost × cost_score (cheapest = 1.0, inverted min-max) - + w_eco × eco_score (cleanest = 1.0, inverted min-max) - + w_prof × proficiency_score (from the task's category) +1. drop candidates whose measured energy exceeds objective.max_energy_per_request +2. rank by proficiency for the task's category +3. treat differences smaller than objective.quality_tolerance as equal +4. among equals, pick the cheapest ``` -| Weight | Value | Notes | +| Setting | Value | Notes | |---|---|---| -| `cost` | 0.4 | From median of billed USD (`seed_reference` category only) | -| `eco` | 0.2 | From median of gCO2eq (`seed_reference` only) | -| `proficiency` | 0.4 | From `proficiency.blended_score` for the task category | +| `quality_tolerance` | 0.10 | Measurement noise, not preference: scores rest on 2-3 samples, so smaller gaps are sampling variation | +| `max_energy_per_request` | null | Per-request kWh ceiling. The plan is a 6.25 kWh quota — a wall, not a bill | +| `plan_kwh_per_period` | 6.25 | Reported in `/health` as burn against the allowance | + +This replaced a weighted blend (cost 0.4 / eco 0.2 / proficiency 0.4). +Measurement retired it: turning the cost weight from 0.4 to **zero** changed +the winner in only 2 of 6 categories, so the blend was never steering on +quality — while 60% of every decision adjudicated fractions of a cent. **Why cost ≠ list price:** Neuralwatt bills flat $8.00/kWh. The catalog's `input_per_million` / `output_per_million` are **not** what gets charged. @@ -378,8 +385,8 @@ Measured: `cost_usd / energy_kwh = 8.00` across every model. A model listing at $4/1M can cost 10× more than one at $15/1M on the same prompt. **Why cost ≠ eco:** Cost tracks energy (kWh), but carbon is energy × grid -intensity. Grid intensity spans 37 gCO2/kWh (`FI`) to 505 -(`US-MIDA-PJM`) — a 13.6× spread. The models disagree: `glm-5.2-fast` is +intensity. Grid intensity spans ~49 gCO2/kWh (`FI`) to ~442 (`US-MIDA-PJM`) — roughly a 9× spread, +and it moves: the provider's own 24h blended figure was 145.4. The models disagree: `glm-5.2-fast` is 2nd cheapest but 6th cleanest; `kimi-k3-flex` draws 3.7× less energy than `kimi-k2.7-code` while emitting 3.6× more carbon. Collapsing them picks a side. diff --git a/config.py b/config.py index fb27f14..114eb3d 100644 --- a/config.py +++ b/config.py @@ -16,20 +16,33 @@ import yaml from pydantic import BaseModel, field_validator, model_validator -class Weights(BaseModel): - cost: float - eco: float - proficiency: float +class Objective(BaseModel): + """What the router optimizes: quality, bounded by cost. - @model_validator(mode="after") - def must_sum_to_one(self) -> "Weights": - total = round(self.cost + self.eco + self.proficiency, 6) - if total != 1.0: + Replaced a three-way weighted blend. See config.yaml for why — briefly, + the cost weight was measured to be nearly inert while consuming 40% of + every decision. + """ + + quality_tolerance: float = 0.10 + max_energy_per_request: Optional[float] = None + plan_kwh_per_period: Optional[float] = None + + @field_validator("quality_tolerance") + @classmethod + def tolerance_in_range(cls, v: float) -> float: + if not (0.0 <= v < 1.0): + raise ValueError("objective.quality_tolerance must be in [0, 1)") + return v + + @field_validator("max_energy_per_request") + @classmethod + def ceiling_positive(cls, v: Optional[float]) -> Optional[float]: + if v is not None and v <= 0: raise ValueError( - f"weights.cost + weights.eco + weights.proficiency must sum to " - f"1.0, got {total}" + "objective.max_energy_per_request must be > 0 kWh, or null to disable" ) - return self + return v class ContextConfig(BaseModel): @@ -160,7 +173,7 @@ class LoggingConfig(BaseModel): class RouterConfig(BaseModel): - weights: Weights + objective: Objective context: ContextConfig tiers: dict[int, str] tiering: TieringConfig diff --git a/config.yaml b/config.yaml index 58d7f9f..b9d4533 100644 --- a/config.yaml +++ b/config.yaml @@ -2,11 +2,38 @@ # All weights, thresholds, and provider settings live here so they can be # tuned without touching code. Loaded/validated by config.py. -weights: - cost: 0.4 - eco: 0.2 - proficiency: 0.4 - # Must sum to 1.0 — config.py will raise on load if they don't. +objective: + # Quality is the objective. Cost is a constraint and a tiebreak. Eco is + # logged per request but is NOT optimized here — that judgement is made + # outside this router. + # + # This replaced a weighted blend (cost 0.4 / eco 0.2 / proficiency 0.4). + # Measurement killed it: turning the cost weight from 0.4 to ZERO changed + # the winner in only 2 of 6 categories, so the blend was never steering on + # quality — while 60% of every decision adjudicated fractions of a cent + # (all real traffic to date totals $0.07). + + # Proficiency differences smaller than this are treated as equal and the + # cheaper model wins. This is measurement noise, not preference: scores + # currently rest on 2-3 samples per category, so a 0.05 gap is + # indistinguishable from sampling variation and paying for it buys noise. + # Narrow it as samples accumulate. + quality_tolerance: 0.10 + + # Per-request ceiling on measured ENERGY, in kWh. null disables it. + # + # Denominated in kWh rather than dollars because the plan is a subscription + # with a 6.25 kWh quota, not pay-per-request. Dollars accrue; a quota is a + # wall you hit mid-task. So this is the cost mandate stated as a guarantee. + # For scale: the reference task runs ~5e-06 kWh on the cheapest model and + # ~2.2e-04 on the most expensive. + max_energy_per_request: null + + # The subscription's kWh allowance per billing period, for reporting burn in + # /health. Set to match your plan; null disables the report. NeuralWatt also + # returns allowance_remaining_usd per request, which is logged, but that is a + # dollar figure while the plan is denominated in energy. + plan_kwh_per_period: 6.25 context: safety_factor: 0.75 # fraction of advertised context treated as usable diff --git a/dispatcher.py b/dispatcher.py index 356cab2..5dc939f 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -162,11 +162,13 @@ class Candidate(BaseModel): # gCO2eq over the reference workload. `list_price_per_1m` rides along # because it is NOT what gets billed, and the gap is worth seeing. cost: Optional[float] = None + energy: Optional[float] = None + # Still reported — carbon is logged and worth seeing — but it is no longer + # part of the decision. eco: Optional[float] = None list_price_per_1m: Optional[float] = None composite: float cost_score: float - eco_score: float proficiency_score: float @@ -378,9 +380,10 @@ def load_candidates(conn: sqlite3.Connection, category: str) -> list[dict]: # the catalog is 19 rows, so clarity beats a window-function expression. costs: dict[tuple, list[float]] = {} carbons: dict[tuple, list[float]] = {} + energies: dict[tuple, list[float]] = {} for o in conn.execute( """ - SELECT model_id, provider, cost_usd, carbon_g_co2eq, carbon_source + SELECT model_id, provider, cost_usd, energy_kwh, carbon_g_co2eq, carbon_source FROM energy_observations WHERE task_category = ? """, (SEED_CATEGORY,), @@ -388,6 +391,10 @@ def load_candidates(conn: sqlite3.Connection, category: str) -> list[dict]: key = (o["model_id"], o["provider"]) if o["cost_usd"] is not None: costs.setdefault(key, []).append(o["cost_usd"]) + # Energy, not just its dollar equivalent: the plan is a fixed kWh + # quota, so energy is the resource the ceiling has to bound. + if o["energy_kwh"] is not None: + energies.setdefault(key, []).append(o["energy_kwh"]) if ( o["carbon_g_co2eq"] is not None and o["carbon_source"] != FALLBACK_CARBON_SOURCE @@ -399,6 +406,7 @@ def load_candidates(conn: sqlite3.Connection, category: str) -> list[dict]: row = dict(r) key = (row["model_id"], row["provider"]) row["cost"] = median(costs[key]) if key in costs else None + row["energy"] = median(energies[key]) if key in energies else None row["eco"] = median(carbons[key]) if key in carbons else None row["samples"] = len(costs.get(key, [])) out.append(row) @@ -452,7 +460,8 @@ def route(req: TaskRequest) -> RouteResponse: ) ranked = rank_candidates( eligible, - weights=(cfg.weights.cost, cfg.weights.eco, cfg.weights.proficiency), + quality_tolerance=cfg.objective.quality_tolerance, + max_energy_per_request=cfg.objective.max_energy_per_request, ) return RouteResponse( @@ -761,6 +770,13 @@ def scoring_coverage() -> dict: missing_proficiency = [m for m, p in routable if (m, p) not in with_proficiency] warnings = [] + quota = quota_burn() + if quota and quota["metered_fraction_of_plan"] > 0.8: + warnings.append( + f"metered usage is {quota['metered_fraction_of_plan']*100:.0f}% of the " + f"{quota['plan_kwh']} kWh plan allowance. A quota is a wall, not a bill — " + "requests fail rather than costing more." + ) if missing_energy: warnings.append( f"{len(missing_energy)}/{total} routable models have no reference-workload " @@ -778,10 +794,46 @@ def scoring_coverage() -> dict: "routable_models": total, "with_energy_data": total - len(missing_energy), "with_proficiency_data": total - len(missing_proficiency), + "quota": quota, "warnings": warnings, } +def quota_burn() -> Optional[dict]: + """Energy this router has metered, against the plan's allowance. + + Reported because the plan is a fixed kWh quota rather than a bill: you do + not overspend it, you hit it, mid-task. + + This counts only what the ROUTER saw. Traffic that bypasses it — the eval + harness, ad-hoc scripts, a client pointed straight at the provider — is + invisible here, so treat this as a floor on real consumption. Measured + against the provider's own 24h figure it accounted for ~86% of energy. + """ + if not cfg.objective.plan_kwh_per_period: + return None + conn = _db() + try: + row = conn.execute( + """ + SELECT COALESCE(SUM(energy_kwh), 0) kwh, COUNT(*) n + FROM energy_observations + WHERE observed_at > datetime('now', '-30 days') + """ + ).fetchone() + finally: + conn.close() + + plan = cfg.objective.plan_kwh_per_period + return { + "plan_kwh": plan, + "metered_kwh_30d": round(row["kwh"], 5), + "metered_fraction_of_plan": round(row["kwh"] / plan, 4), + "metered_calls_30d": row["n"], + "note": "router-metered only; traffic bypassing the router is not counted", + } + + @app.post("/route", response_model=RouteResponse) def route_endpoint(req: TaskRequest): """Classify and pick a model without calling it.""" diff --git a/routing.py b/routing.py index cc3207b..97f0733 100644 --- a/routing.py +++ b/routing.py @@ -25,7 +25,7 @@ from __future__ import annotations from collections.abc import Sequence -from scoring import composite_score, cost_score, eco_score, proficiency_score +from scoring import cost_score, proficiency_score INTERACTIVE = "interactive" BATCH = "batch" @@ -97,49 +97,87 @@ def select_candidates( ] +def within_budget(row: dict, max_energy_kwh: float | None) -> bool: + """Whether a candidate's measured energy is inside the per-request ceiling. + + Energy rather than dollars because the plan is a subscription with a fixed + kWh quota. Dollars accrue and can be reasoned about after the fact; a quota + is a wall you hit in the middle of a task. + + A model with no energy measurement is admitted. Excluding the unmeasured + would mean a newly listed model could never be picked and so could never + acquire a measurement — the same trap the neutral-0.5 default avoids + elsewhere. + """ + if max_energy_kwh is None: + return True + energy = row.get("energy") + return energy is None or energy <= max_energy_kwh + + def rank_candidates( rows: Sequence[dict], *, - weights: tuple[float, float, float], + quality_tolerance: float = 0.1, + max_energy_per_request: float | None = None, ) -> list[dict]: - """Score and sort candidates best-first. + """Order candidates: best quality first, cheapest among equals. - Each row carries an optional ``cost`` (mean USD actually billed for the - reference workload), an optional ``eco`` (mean gCO2eq for the same), and - an optional ``proficiency`` (blended score for the task's category). - Any of them missing yields the neutral 0.5 from ``scoring``, so a model - is never penalized merely for being new to the table. + This replaced a weighted blend of cost, eco and proficiency, for reasons + measurement forced: - ``cost`` is the billed figure rather than the catalog's list price, which - NeuralWatt does not charge — see ``dispatcher.load_candidates``. That - also removes the need to guess a flex discount: a flex row's measured - cost already is its flex cost. + - **Eco is no longer an objective.** Carbon is still logged per request, + but it is not something this router optimizes; that judgement is made + outside it. Leaving it in meant 20% of every decision optimized an + unstated goal. + - **Cost is a constraint and a tiebreak, not a weight.** Under the blend, + 60% of the decision adjudicated differences of fractions of a cent — + all real traffic to date totals $0.07 — and min-max normalization made + "expensive" relative to whoever else happened to be a candidate, so a + model could lose for being 2x a very cheap model when both round to + nothing. A ceiling states the cost mandate as a guarantee instead. + - **Quality is the objective**, which is what the blend obscured: turning + the cost weight from 0.4 all the way to zero changed the winner in only + 2 of 6 categories, because the blend was never really steering on + quality at all. - Returns a list of dicts: the original row plus ``composite`` and the - three sub-scores, so a caller can log exactly why a model won. + ``quality_tolerance`` is the band inside which two proficiency scores are + treated as equal. It is not a preference — it reflects measurement noise. + Proficiency currently rests on 2-3 samples per category, so a gap of 0.05 + is indistinguishable from sampling variation, and paying more for it would + be buying noise. Widen it as confidence falls, narrow it as samples + accumulate. + + Returns each row plus ``proficiency_score``, ``cost_score`` (reported for + visibility only, no longer part of the decision) and ``composite``, which + is now simply the effective quality after the tolerance band. """ - cost_scores = cost_score([r.get("cost") for r in rows]) - eco_scores = eco_score([r.get("eco") for r in rows]) + affordable = [r for r in rows if within_budget(r, max_energy_per_request)] + + # cost_score is retained purely so callers can still see the spread; it + # does not enter the ordering. + cost_scores = cost_score([r.get("cost") for r in affordable]) ranked = [] - for row, c_s, e_s in zip(rows, cost_scores, eco_scores): + for row, c_s in zip(affordable, cost_scores): p_s = proficiency_score(row.get("proficiency")) - ranked.append( - { - **row, - "cost_score": c_s, - "eco_score": e_s, - "proficiency_score": p_s, - "composite": composite_score(c_s, e_s, p_s, weights), - } - ) + ranked.append({**row, "cost_score": c_s, "proficiency_score": p_s, + "composite": p_s}) + + if not ranked: + return [] + + # Quality first, but only differences larger than the tolerance count. + # Bucketing by band means a 0.01 edge cannot outrank a 10x cost saving, + # while a real gap (tool_use_agentic spans 0.67) still decides outright. + best = max(r["proficiency_score"] for r in ranked) + + def band(r: dict) -> int: + return int((best - r["proficiency_score"]) / quality_tolerance) - # Sort by composite, then by cost ascending, then model_id, so that rows - # tying on every scored dimension still resolve deterministically instead - # of returning whatever order the DB happened to hand back. ranked.sort( key=lambda r: ( - -r["composite"], + band(r), r["cost"] if r.get("cost") is not None else float("inf"), r["model_id"], ) diff --git a/tests/test_routing.py b/tests/test_routing.py index 92d0472..43398ff 100644 --- a/tests/test_routing.py +++ b/tests/test_routing.py @@ -8,7 +8,6 @@ import pytest from routing import is_eligible, rank_candidates, select_candidates -WEIGHTS = (0.4, 0.2, 0.4) def _row(**overrides) -> dict: @@ -18,6 +17,7 @@ def _row(**overrides) -> dict: "provider": "neuralwatt", "tier": 2, "cost": 1.0, + "energy": 1.0e-5, "effective_context_window": 100_000, "availability": "active", "deprecated": 0, @@ -145,97 +145,91 @@ def test_select_candidates_filters_and_preserves_order(): # --- measured cost -------------------------------------------------------- -def test_unknown_cost_is_neutral_not_worst(): - # Given: a model never covered by the reference sweep. It must not be - # ranked last for lacking data, or a newly listed model could never be - # picked and so could never acquire data. - ranked = rank_candidates([_row(model_id="unswept", cost=None)], weights=WEIGHTS) - assert ranked[0]["cost_score"] == 0.5 +# --- ranking: quality first, cheapest among equals ------------------------- + +def test_a_real_quality_gap_decides_outright(): + # tool_use_agentic spans 0.67 across the catalog; a gap that size must + # beat any cost saving + rows = [_row(model_id="cheap-bad", cost=1e-6, proficiency=0.33), + _row(model_id="dear-good", cost=1e-3, proficiency=1.00)] + assert rank_candidates(rows)[0]["model_id"] == "dear-good" -# --- ranking -------------------------------------------------------------- - -def test_cheaper_model_wins_when_only_cost_differs(): - rows = [_row(model_id="pricey", cost=15.0), - _row(model_id="cheap", cost=0.28)] - ranked = rank_candidates(rows, weights=WEIGHTS) - assert ranked[0]["model_id"] == "cheap" - assert ranked[0]["composite"] > ranked[1]["composite"] +def test_within_tolerance_the_cheaper_model_wins(): + # 0.02 apart on 2-3 samples is sampling noise, not a quality difference. + # Paying 100x for it would be buying noise. + rows = [_row(model_id="cheap", cost=1e-5, proficiency=0.98), + _row(model_id="dear", cost=1e-3, proficiency=1.00)] + assert rank_candidates(rows, quality_tolerance=0.1)[0]["model_id"] == "cheap" -def test_opposed_full_spreads_tie_at_equal_weight(): - # Given: two candidates at opposite extremes of both cost and proficiency. - # Min-max normalization puts each at 0.0/1.0, and cost and proficiency - # carry the same 0.4 weight, so the composites land exactly equal and the - # cost tiebreak decides. Worth pinning: with only two candidates the - # normalization always produces a full spread, which makes the default - # weights much tie-ier than they look. - rows = [ - _row(model_id="cheap-bad", cost=0.1, proficiency=0.0), - _row(model_id="dear-good", cost=0.2, proficiency=1.0), - ] - ranked = rank_candidates(rows, weights=WEIGHTS) - assert ranked[0]["composite"] == ranked[1]["composite"] - assert ranked[0]["model_id"] == "cheap-bad" +def test_narrowing_the_tolerance_makes_small_gaps_count(): + # As samples accumulate and confidence rises, the band should shrink + rows = [_row(model_id="cheap", cost=1e-5, proficiency=0.98), + _row(model_id="dear", cost=1e-3, proficiency=1.00)] + assert rank_candidates(rows, quality_tolerance=0.001)[0]["model_id"] == "dear" -def test_proficiency_wins_when_weighted_above_cost(): - rows = [ - _row(model_id="cheap-bad", cost=0.1, proficiency=0.0), - _row(model_id="dear-good", cost=0.2, proficiency=1.0), - ] - ranked = rank_candidates(rows, weights=(0.2, 0.2, 0.6)) - assert ranked[0]["model_id"] == "dear-good" +def test_eco_no_longer_influences_the_decision(): + # Carbon is still logged; it is simply not what this router optimizes + rows = [_row(model_id="clean", cost=1e-3, proficiency=0.5, eco=1e-6), + _row(model_id="dirty", cost=1e-5, proficiency=0.5, eco=1e9)] + assert rank_candidates(rows)[0]["model_id"] == "dirty" -def test_missing_proficiency_and_eco_are_neutral_not_penalized(): - ranked = rank_candidates([_row()], weights=WEIGHTS) +def test_missing_proficiency_is_neutral_not_penalized(): + ranked = rank_candidates([_row()]) assert ranked[0]["proficiency_score"] == 0.5 - assert ranked[0]["eco_score"] == 0.5 -def test_ties_break_deterministically_by_cost_then_model_id(): - # Given: the real problem — sibling rows identical on every scored - # dimension. Order must not depend on what the DB happened to return. +def test_unknown_cost_does_not_disqualify(db_free=None): + # A model never swept must stay pickable, or it can never acquire a + # measurement — the same trap the neutral-0.5 default avoids + ranked = rank_candidates([_row(model_id="unswept", cost=None)]) + assert ranked[0]["model_id"] == "unswept" + + +def test_ties_break_deterministically(): rows = [_row(model_id="glm-b"), _row(model_id="glm-a"), _row(model_id="glm-c")] - first = rank_candidates(rows, weights=WEIGHTS) - second = rank_candidates(list(reversed(rows)), weights=WEIGHTS) - assert [r["model_id"] for r in first] == ["glm-a", "glm-b", "glm-c"] - assert [r["model_id"] for r in first] == [r["model_id"] for r in second] - - -def test_ranking_reports_the_sub_scores_that_produced_the_winner(): - ranked = rank_candidates([_row(proficiency=0.8)], weights=WEIGHTS) - top = ranked[0] - assert top["composite"] == pytest.approx( - 0.4 * top["cost_score"] + 0.2 * top["eco_score"] + 0.4 * top["proficiency_score"] - ) + first = [r["model_id"] for r in rank_candidates(rows)] + second = [r["model_id"] for r in rank_candidates(list(reversed(rows)))] + assert first == ["glm-a", "glm-b", "glm-c"] == second def test_empty_candidate_set_ranks_to_empty(): - assert rank_candidates([], weights=WEIGHTS) == [] + assert rank_candidates([]) == [] -def test_cost_and_eco_can_rank_models_oppositely(): - # Given: the real catalog shape. NeuralWatt bills per kWh, so cost tracks - # energy — but carbon is energy x the serving region's grid intensity, and - # that spans 37 gCO2/kWh (FI) to 505 (US-MIDA-PJM). glm-5.2-fast draws - # little energy on a dirty grid; kimi-k2.7-code draws a lot on a clean one. - rows = [ - _row(model_id="glm-5.2-fast", cost=6.72e-05, eco=3.973e-03), - _row(model_id="kimi-k2.7-code", cost=1.18e-03, eco=5.459e-03), - ] - # Then: whichever axis carries the weight decides, and they disagree — - # which is why these must stay two axes rather than being collapsed. - by_cost = rank_candidates(rows, weights=(1.0, 0.0, 0.0)) - by_eco = rank_candidates(rows, weights=(0.0, 1.0, 0.0)) - assert by_cost[0]["model_id"] == "glm-5.2-fast" - assert by_eco[0]["model_id"] == "glm-5.2-fast" +# --- the budget ceiling: the quota mandate as a guarantee ------------------ - # And with a genuinely inverted pair the orders flip outright. - rows = [ - _row(model_id="low-cost-dirty", cost=1.0, eco=100.0), - _row(model_id="high-cost-clean", cost=10.0, eco=1.0), - ] - assert rank_candidates(rows, weights=(1.0, 0.0, 0.0))[0]["model_id"] == "low-cost-dirty" - assert rank_candidates(rows, weights=(0.0, 1.0, 0.0))[0]["model_id"] == "high-cost-clean" +def test_ceiling_excludes_models_over_budget(): + # Denominated in kWh: the plan is a fixed quota, and a quota is a wall you + # hit mid-task rather than a bill that accrues + rows = [_row(model_id="affordable", energy=1e-6, proficiency=0.5), + _row(model_id="expensive", energy=1e-3, proficiency=1.0)] + ranked = rank_candidates(rows, max_energy_per_request=1e-4) + assert [r["model_id"] for r in ranked] == ["affordable"] + + +def test_ceiling_binds_even_against_the_best_model(): + # This is the point of a constraint rather than a weight: no amount of + # quality buys past the ceiling + rows = [_row(model_id="expensive", energy=1e-3, proficiency=1.0)] + assert rank_candidates(rows, max_energy_per_request=1e-6) == [] + + +def test_no_ceiling_admits_everything(): + rows = [_row(model_id="expensive", energy=1e9, proficiency=1.0)] + assert len(rank_candidates(rows, max_energy_per_request=None)) == 1 + + +def test_unmeasured_cost_is_admitted_under_a_ceiling(): + # Excluding the unmeasured would mean a new model could never be picked + # and so could never acquire a measurement + rows = [_row(model_id="unswept", energy=None, proficiency=1.0)] + assert len(rank_candidates(rows, max_energy_per_request=1e-9)) == 1 + + +def test_ceiling_at_exactly_the_cost_admits(): + rows = [_row(model_id="borderline", energy=1e-6, proficiency=0.5)] + assert len(rank_candidates(rows, max_energy_per_request=1e-6)) == 1 -- 2.49.1 From e1fd65f3e4639fb40dc54cef6863f610be00f9bf Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 22:25:54 -0400 Subject: [PATCH 16/20] feat: tier is an iteration budget, spent on evidence rather than on a hunch A tier used to mean only a capability floor. It now also buys corrective attempts after a verification failure: tier 1 gets one shot, tier 2 one retry, tier 3 two. Interactive requests are capped below their tier regardless, because every retry doubles time-to-answer and in interactive use latency IS a quality loss. escalation.preemptive_on_low_confidence now defaults to FALSE. Bumping the tier because the classifier was unsure of its own call pays frontier prices before anything has gone wrong. Spending after a check has actually failed is better on both mandates: the cheap attempt usually succeeds and costs nothing extra, and when it fails there is evidence rather than a hunch. Retries are matched to the failure, because the causes differ. Truncation raises the token budget on the same model -- a different one would run out too. Malformed output escalates to the next-ranked candidate, since more tokens will not make unparseable output parse. 'ok' and 'unverifiable' buy nothing; retrying unverifiable would burn quota across the majority of prose traffic for no signal at all. Testing it live exposed that the truncation branch was UNREACHABLE as first written. Through /v1 the token cap is either the client's, which is not ours to override, or absent -- and when absent the model hit its own ceiling, so doubling changes nothing. It now escalates to a candidate with a larger output ceiling, which is the actionable move, and declines when no such candidate exists rather than wasting an attempt against a fixed kWh quota. Known limit, recorded in CLAUDE.md: retry does not reach the streaming path. Once bytes have gone to the client there is nothing to take back, and buffering to allow correction would cost streaming itself. opencode streams, so the main workflow gets verification and feedback but not correction. Tests 211 -> 227. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 41 ++++++++++-- config.py | 24 +++++++ config.yaml | 29 +++++++- dispatcher.py | 125 ++++++++++++++++++++++++++++------- iteration.py | 142 ++++++++++++++++++++++++++++++++++++++++ tests/test_iteration.py | 113 ++++++++++++++++++++++++++++++++ 6 files changed, 442 insertions(+), 32 deletions(-) create mode 100644 iteration.py create mode 100644 tests/test_iteration.py diff --git a/CLAUDE.md b/CLAUDE.md index 2a6f167..027f49d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -301,6 +301,36 @@ fails. It immediately caught a check where the expected value was simply wrong — which would have docked every model on a task and been indistinguishable from genuine difficulty. +## Tier is an iteration budget, not just a floor + +A tier used to mean only "do not route below this". It now also buys +corrective attempts after a verification failure: + +| tier | batch | interactive | +|---|---|---| +| 1 | 0 retries | 0 | +| 2 | 1 | 1 | +| 3 | 2 | 1 | + +Interactive is capped below its tier because every retry doubles +time-to-answer, and in interactive use latency **is** a quality loss. + +Retries are matched to the failure, since the causes differ: + +- **truncated** — raise the token budget on the same model; a different one + would run out too. If there is no cap to raise, the model's own output + ceiling is the wall, so escalate to a candidate that can emit more. +- **malformed** — more tokens will not make unparseable output parse, so + escalate to the next-ranked candidate. +- **ok / unverifiable** — buy nothing. Retrying `unverifiable` would burn + quota across the majority of prose traffic for no signal. + +`escalation.preemptive_on_low_confidence` is now **off by default**. Bumping +the tier because the classifier was unsure pays frontier prices before +anything has gone wrong; spending after a check has actually failed is better +on both mandates — the cheap attempt usually succeeds, and when it fails you +have evidence rather than a hunch. + ## Verification: what local compute is actually good for Local inference is a poor substitute for cloud completions here — 7-82x the @@ -385,11 +415,12 @@ that request was two orders of magnitude low. `seed_energy.py --models ... --samples 21` for those before trusting where they land. -3. **Acting on verification.** Every response is now checked and failures - feed back into proficiency (`verification.py`, `feedback.py`), but nothing - retries yet — the loop observes and learns, it does not intervene. Whether - to auto-escalate on a failed check should be decided from the observed - failure rate, which is now being collected. +3. **Retry does not reach streaming.** The iteration budget (`iteration.py`) + retries after a failed check, but only on the non-streaming path — once + bytes have gone to the client there is nothing to take back. opencode + streams, so the main workflow gets verification and feedback but not + correction. Buffering to fix that would cost streaming itself, which is a + worse trade for interactive work. ## Known open questions diff --git a/config.py b/config.py index 114eb3d..5d6cb8d 100644 --- a/config.py +++ b/config.py @@ -137,6 +137,29 @@ class EscalationConfig(BaseModel): enabled: bool max_tier: int min_confidence_before_bump: float + # Off by default: the iteration budget escalates on evidence instead. + preemptive_on_low_confidence: bool = False + + +class IterationConfig(BaseModel): + """A tier's budget for corrective attempts after a verification failure.""" + + enabled: bool = True + attempts_by_tier: dict[int, int] = {1: 0, 2: 1, 3: 2} + max_attempts_interactive: int = 1 + + @field_validator("attempts_by_tier") + @classmethod + def attempts_sane(cls, v: dict[int, int]) -> dict[int, int]: + for tier, attempts in v.items(): + if attempts < 0: + raise ValueError(f"iteration.attempts_by_tier[{tier}] must be >= 0") + if attempts > 5: + raise ValueError( + f"iteration.attempts_by_tier[{tier}]={attempts} is implausibly " + "high; each attempt spends energy against a fixed quota" + ) + return v class FreshnessConfig(BaseModel): @@ -181,6 +204,7 @@ class RouterConfig(BaseModel): routing: RoutingConfig verification: VerificationConfig = VerificationConfig() escalation: EscalationConfig + iteration: IterationConfig = IterationConfig() freshness: FreshnessConfig database: DatabaseConfig classifier: ClassifierConfig diff --git a/config.yaml b/config.yaml index b9d4533..8d2eb93 100644 --- a/config.yaml +++ b/config.yaml @@ -78,10 +78,35 @@ proficiency: escalation: enabled: true max_tier: 3 - # If the local model's classification confidence is below this, bump - # required_tier by one as a precaution rather than trusting a shaky call. + + # Bump the tier when the classifier is unsure of its own call. DEFAULT OFF: + # this pays frontier prices on a hunch, before anything has gone wrong. The + # iteration budget below spends after a check has actually failed, which is + # strictly better on both mandates — the cheap attempt usually succeeds and + # costs nothing extra, and when it fails you have evidence. + preemptive_on_low_confidence: false min_confidence_before_bump: 0.6 +iteration: + # A tier is not only a capability floor, it is a budget for getting the + # answer right. These are corrective attempts AFTER a verification failure, + # not speculative retries. + # + # Retries are matched to the failure: a truncated answer gets a bigger token + # budget on the SAME model (a different one would also run out), while a + # malformed answer escalates to the next-best candidate (more tokens will + # not make unparseable output parse). + enabled: true + attempts_by_tier: + 1: 0 # cheap/simple — one shot; iterating costs more than it is worth + 2: 1 + 3: 2 + + # Interactive requests are capped below their tier's budget regardless of + # tier: every retry doubles time-to-answer, and in interactive use latency + # IS a quality loss. Batch work does not care. + max_attempts_interactive: 1 + routing: # Access gating is prose-only in the NeuralWatt catalog ("Private preview # (grant-gated)", "(Canary)"), so the poller parses it into access_level and diff --git a/dispatcher.py b/dispatcher.py index 5dc939f..c210cfb 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -51,6 +51,7 @@ from pydantic import BaseModel, Field from config import RouterConfig, load_config from routing import BATCH, INTERACTIVE, rank_candidates, select_candidates +from iteration import attempts_allowed, plan_retry from verification import ( LOCAL_VERIFY_SYSTEM, build_local_verify_prompt, @@ -303,6 +304,11 @@ def apply_escalation(c: Classification) -> Classification: # local model is unavailable, which is the expensive failure mode. if c.source == "fallback": return c + # Off by default. Bumping on uncertainty pays more before anything has + # gone wrong; the iteration budget spends after a check has actually + # failed, which is better on both mandates. + if not cfg.escalation.preemptive_on_low_confidence: + return c if c.confidence >= cfg.escalation.min_confidence_before_bump: return c bumped = min(c.task_tier + 1, cfg.escalation.max_tier) @@ -413,6 +419,23 @@ def load_candidates(conn: sqlite3.Connection, category: str) -> list[dict]: return out +def model_output_ceiling(model_id: str, provider: str) -> Optional[int]: + """The model's own advertised output limit, if the catalog states one. + + Bounds a truncation retry: doubling the budget past what the model accepts + would be rejected outright, wasting the attempt. + """ + conn = _db() + try: + row = conn.execute( + "SELECT max_output_tokens FROM models WHERE model_id = ? AND provider = ?", + (model_id, provider), + ).fetchone() + finally: + conn.close() + return row["max_output_tokens"] if row else None + + def _to_candidate(row: dict) -> Candidate: fields = {k: row[k] for k in Candidate.model_fields if k in row} fields["list_price_per_1m"] = row.get("cost_per_1m_completion") @@ -1005,47 +1028,99 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): headers = {"authorization": f"Bearer {api_key}"} if not streaming: - resp = requests.post(url, headers=headers, json=upstream_body, timeout=600) - if resp.status_code >= 400: - raise HTTPException(resp.status_code, resp.text[:500]) - payload = resp.json() - usage = payload.get("usage") or {} - completion_tokens = usage.get("completion_tokens") - log_observation( - target, - provider, - category, - usage.get("prompt_tokens"), - completion_tokens, - extract_telemetry(payload), + # A tier's iteration budget: corrective attempts AFTER a verification + # failure, never speculative ones. Retries are matched to the failure — + # truncation gets a bigger budget on the same model, malformed output + # escalates to the next candidate. See iteration.py. + budget = ( + attempts_allowed( + decision.classification.task_tier, + decision.latency_tolerance, + cfg.iteration.attempts_by_tier, + cfg.iteration.max_attempts_interactive, + ) + if cfg.iteration.enabled and wants_routing + else 0 ) + alternatives = [ + (c.model_id, model_output_ceiling(c.model_id, provider)) + for c in decision.runners_up + ] + current_model = target + current_max_tokens = body.get("max_tokens") + attempts_used = 0 + retry_trail: list[str] = [] + + while True: + attempt_body = {**body, "model": current_model} + if current_max_tokens is not None: + attempt_body["max_tokens"] = current_max_tokens + resp = requests.post(url, headers=headers, json=attempt_body, timeout=600) + if resp.status_code >= 400: + raise HTTPException(resp.status_code, resp.text[:500]) + + payload = resp.json() + usage = payload.get("usage") or {} + completion_tokens = usage.get("completion_tokens") + log_observation( + current_model, provider, category, + usage.get("prompt_tokens"), completion_tokens, + extract_telemetry(payload), + ) + + choice = (payload.get("choices") or [{}])[0] + content = (choice.get("message") or {}).get("content") or "" + result = verify_response(content, choice.get("finish_reason")) + log_verification( + current_model, provider, category, "structural", + result.verdict, result.detail, completion_tokens, + model_attributable=not client_capped, + ) + + if not result.failed or attempts_used >= budget: + break + + plan = plan_retry( + result.verdict, + current_model, + current_max_tokens, + model_output_ceiling(current_model, provider), + alternatives, + client_capped, + ) + if plan is None: + # Declining is a real outcome: a retry that cannot help still + # spends energy against a fixed quota. + break + + retry_trail.append(f"{result.verdict}:{plan.detail}") + print(f"iteration: {plan.detail} (was {result.verdict})", file=sys.stderr) + if plan.reason == "malformed": + alternatives = alternatives[1:] + current_model = plan.model_id + current_max_tokens = plan.max_tokens + attempts_used += 1 - choice = (payload.get("choices") or [{}])[0] - content = (choice.get("message") or {}).get("content") or "" - result = verify_response(content, choice.get("finish_reason")) - log_verification( - target, provider, category, "structural", - result.verdict, result.detail, completion_tokens, - model_attributable=not client_capped, - ) if cfg.verification.local_llm_enabled and worth_local_check( result.verdict, completion_tokens, cfg.verification.min_completion_tokens ): background.add_task( - run_local_verification, target, provider, category, + run_local_verification, current_model, provider, category, _last_user_text(messages), content, completion_tokens, ) # Report the model actually used, so the client isn't told 'auto'. - payload["model"] = target - # Surfaced as a header rather than in the body: the body must stay a + payload["model"] = current_model + # Surfaced as headers rather than in the body: the body must stay a # valid OpenAI response, and a client that ignores headers is # unaffected. return JSONResponse( content=payload, headers={ - "X-Router-Model": target, + "X-Router-Model": current_model, "X-Router-Verification": result.verdict, + "X-Router-Attempts": str(attempts_used + 1), + **({"X-Router-Retries": "; ".join(retry_trail)} if retry_trail else {}), }, ) diff --git a/iteration.py b/iteration.py new file mode 100644 index 0000000..1879dd1 --- /dev/null +++ b/iteration.py @@ -0,0 +1,142 @@ +"""Pure logic for spending a tier's iteration budget. + +Reframes what a tier means. It used to be only a capability floor — +"don't route below this". It is now also a budget for getting the answer +right: tier 1 buys one attempt, higher tiers buy corrective attempts after a +verification failure. + +Why this is better than what it replaces. `apply_escalation` bumps the tier +*preemptively* when the classifier is unsure of its own call, so an uncertain +guess costs frontier prices before anything has gone wrong. Spending after a +check has actually failed is strictly better on both mandates: the cheap +attempt usually succeeds and costs nothing extra, and when it fails you have +evidence rather than a hunch. + +The retry strategy depends on the failure, because the failures have different +causes: + +- **truncated** — the answer ran out of budget. If there is a cap to raise, + raise it on the same model. If there is not, the model's own output ceiling + is the wall, so escalate to a candidate that can emit more. (Without that + second case the branch was unreachable in practice: through /v1 the cap is + either the client's, which is not ours to override, or absent.) +- **malformed** — the model produced something that would not parse. More + tokens will not help, so this escalates to the next-best candidate. + +Latency is treated as a cost. Every retry doubles time-to-answer, which in +interactive use *is* a quality loss, so interactive work gets a lower cap than +batch work regardless of tier. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Literal, Optional + +INTERACTIVE = "interactive" +BATCH = "batch" + +# Multiplier applied to the token budget when retrying a truncated answer. +TRUNCATION_BUDGET_MULTIPLIER = 2 + + +@dataclass +class RetryPlan: + """How to spend one attempt from the budget.""" + + reason: Literal["truncated", "malformed"] + model_id: str + max_tokens: Optional[int] + detail: str + + +def attempts_allowed( + tier: int, + latency_tolerance: str, + attempts_by_tier: dict[int, int], + max_attempts_interactive: int, +) -> int: + """How many corrective attempts this request has earned. + + Interactive requests are capped below their tier's budget: a user waiting + on an answer pays for every retry in latency, and a slow correct answer can + be worth less than a fast one they can judge themselves. + """ + budget = attempts_by_tier.get(tier, 0) + if latency_tolerance == INTERACTIVE: + return min(budget, max_attempts_interactive) + return budget + + +def plan_retry( + verdict: str, + current_model: str, + current_max_tokens: Optional[int], + model_ceiling: Optional[int], + runners_up: list[tuple[str, Optional[int]]], + client_capped: bool, +) -> Optional[RetryPlan]: + """Decide how to retry a failed response, or decline to. + + ``runners_up`` is (model_id, output_ceiling) in ranked order, because a + truncation retry needs to know which alternatives can actually hold a + longer answer. + + Returns None when retrying cannot help, which is as important as returning + a plan — a wasted attempt costs energy against a fixed quota. + """ + if verdict == "truncated": + # A client that set its own max_tokens chose this outcome. Overriding + # it would ignore an explicit instruction, and the caller may well want + # a short answer. + if client_capped: + return None + + if current_max_tokens is not None: + bigger = current_max_tokens * TRUNCATION_BUDGET_MULTIPLIER + if model_ceiling: + bigger = min(bigger, model_ceiling) + if bigger > current_max_tokens: + return RetryPlan( + "truncated", + current_model, + bigger, + f"retrying {current_model} with {bigger} tokens", + ) + + # No cap to raise, or already at this model's limit: the model's own + # output ceiling is the wall. Retrying it changes nothing, but a + # candidate that can emit MORE might finish the answer. + roomier = next( + ( + (m, ceiling) + for m, ceiling in runners_up + if ceiling and (not model_ceiling or ceiling > model_ceiling) + ), + None, + ) + if roomier is None: + return None + return RetryPlan( + "truncated", + roomier[0], + current_max_tokens, + f"escalating to {roomier[0]} ({roomier[1]} output tokens)", + ) + + if verdict == "malformed": + # More tokens will not make unparseable output parse. Try the next + # candidate the ranking preferred. + if not runners_up: + return None + return RetryPlan( + "malformed", + runners_up[0][0], + current_max_tokens, + f"escalating to {runners_up[0][0]}", + ) + + # 'ok' and 'unverifiable' are not failures and buy no retry. Spending an + # attempt on 'unverifiable' would burn quota on the majority of prose + # traffic for no signal at all. + return None diff --git a/tests/test_iteration.py b/tests/test_iteration.py new file mode 100644 index 0000000..d8be648 --- /dev/null +++ b/tests/test_iteration.py @@ -0,0 +1,113 @@ +"""Tests for iteration.py — spending a tier's retry budget. + +Two ideas under test. First, that a retry is matched to the failure: a +truncated answer needs a bigger budget, not a different model, and a malformed +one needs a different model, not a bigger budget. Second, that declining to +retry is a real outcome — every wasted attempt burns energy against a fixed +kWh quota. +""" + +import pytest + +from iteration import attempts_allowed, plan_retry + +BY_TIER = {1: 0, 2: 1, 3: 2} + + +# --- how much iteration a tier buys --------------------------------------- + +def test_tier_one_buys_no_retries(): + # Cheap/simple work: one shot. Iterating on it costs more than it is worth. + assert attempts_allowed(1, "batch", BY_TIER, 1) == 0 + + +def test_higher_tiers_buy_more_attempts(): + assert attempts_allowed(2, "batch", BY_TIER, 5) == 1 + assert attempts_allowed(3, "batch", BY_TIER, 5) == 2 + + +def test_interactive_work_is_capped_below_its_tier_budget(): + # Every retry doubles time-to-answer, and in interactive use latency IS a + # quality loss — a slow correct answer can be worth less than a fast one + # the user can judge themselves. + assert attempts_allowed(3, "interactive", BY_TIER, 1) == 1 + assert attempts_allowed(3, "batch", BY_TIER, 1) == 2 + + +def test_an_unknown_tier_buys_nothing(): + assert attempts_allowed(9, "batch", BY_TIER, 5) == 0 + + +# --- truncation: more budget, same model ---------------------------------- + +def test_truncation_retries_the_same_model_with_more_tokens(): + # A different model would also run out. The budget is the problem. + plan = plan_retry("truncated", "qwen3.6-35b", 1000, 16384, [("kimi-k3", 65536)], False) + assert plan.model_id == "qwen3.6-35b" + assert plan.max_tokens == 2000 + + +def test_truncation_retry_respects_the_model_ceiling(): + plan = plan_retry("truncated", "gemma-4-31b", 10000, 16384, [], False) + assert plan.max_tokens == 16384 + + +def test_no_retry_when_already_at_the_model_ceiling(): + # Doubling would change nothing, and the attempt costs quota + assert plan_retry("truncated", "gemma-4-31b", 16384, 16384, [], False) is None + + +def test_a_client_chosen_cap_is_not_overridden(): + # The caller explicitly asked for a short answer. Ignoring that would + # override an instruction, and they may well want it short. + assert plan_retry("truncated", "qwen3.6-35b", 40, 16384, [("kimi-k3", 65536)], True) is None + + +def test_no_cap_set_escalates_to_a_model_that_can_emit_more(): + # The model's own output ceiling is the wall, so retrying IT changes + # nothing — but a roomier candidate might finish the answer. Without this + # the truncation branch was unreachable in practice: via /v1 the cap is + # either the client's (not ours to override) or absent. + plan = plan_retry("truncated", "gemma-4-31b", None, 16384, [("kimi-k3", 65536)], False) + assert plan.model_id == "kimi-k3" + + +def test_no_roomier_candidate_means_no_retry(): + plan = plan_retry("truncated", "deepseek-v4-flash", None, 65536, + [("gemma-4-31b", 16384)], False) + assert plan is None + + +def test_candidates_with_unknown_ceilings_are_not_assumed_roomier(): + # 11 of 19 catalog rows report no max_output_tokens; guessing they are + # bigger would waste the attempt + assert plan_retry("truncated", "gemma-4-31b", None, 16384, + [("qwen3.6-35b", None)], False) is None + + +# --- malformed: different model, same budget ------------------------------ + +def test_malformed_escalates_to_the_next_candidate(): + # More tokens will not make unparseable output parse + plan = plan_retry("malformed", "qwen3.6-35b", 1000, 16384, [("kimi-k3", 65536), ("gemma-4-31b", 16384)], False) + assert plan.model_id == "kimi-k3" + assert plan.max_tokens == 1000 + + +def test_malformed_with_no_alternative_declines(): + assert plan_retry("malformed", "qwen3.6-35b", 1000, 16384, [], False) is None + + +def test_malformed_ignores_the_client_cap_question(): + # The cap is irrelevant here — the answer did not parse, it was not cut off + plan = plan_retry("malformed", "qwen3.6-35b", 40, 16384, [("kimi-k3", 65536)], True) + assert plan.model_id == "kimi-k3" + + +# --- what must NOT buy a retry -------------------------------------------- + +@pytest.mark.parametrize("verdict", ["ok", "unverifiable"]) +def test_non_failures_buy_no_retry(verdict): + # 'unverifiable' is most prose traffic. Retrying it would burn quota + # across the majority of requests for no signal at all. + assert plan_retry(verdict, "qwen3.6-35b", 1000, 16384, [("kimi-k3", 65536)], False) is None -- 2.49.1 From e28443cb368c47f8cadf8a0c5a63526d1c59db0b Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 22:32:04 -0400 Subject: [PATCH 17/20] =?UTF-8?q?feat:=20POST=20/outcome=20=E2=80=94=20the?= =?UTF-8?q?=20only=20ground=20truth=20the=20router=20can=20get?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Everything else this router records is a proxy. Structural checks know whether code parses. The local checker guesses whether prose looks right. Neither knows whether the answer did the job. The client does: it ran the tests, or used the answer, or watched it fail. Clients report against the provider's completion id, which they already receive in the response body and on every stream chunk. energy_observations and verifications now store that id so a report has something to join on. Two properties make this the highest-value signal available. It is the only quality signal that survives streaming. A retry cannot reach a streamed response -- the bytes are already gone -- but a report arrives afterwards and works identically either way. Every agent client streams, so without this the main workflow had verification and feedback but no route from outcome back into routing. And its successes count. feedback.py folds client outcomes in BOTH directions, unlike checks where only failures do. That asymmetry is deliberate: a parser reporting 'ok' means the code parsed, which is weak evidence that would inflate every score toward the ceiling, while a client reporting 'succeeded' means the work worked. An unknown request_id returns 404 rather than being quietly accepted. A client whose reports go nowhere should find out rather than train nothing. Verified end to end on both paths, including a streamed completion reported as failed after the fact. Tests 227 -> 232. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- CLAUDE.md | 39 ++++++++++++-- dispatcher.py | 116 ++++++++++++++++++++++++++++++++++++----- feedback.py | 81 ++++++++++++++++++---------- schema.sql | 14 ++++- tests/test_feedback.py | 52 ++++++++++++++++++ 5 files changed, 255 insertions(+), 47 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 027f49d..74ab3ff 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -331,6 +331,32 @@ anything has gone wrong; spending after a check has actually failed is better on both mandates — the cheap attempt usually succeeds, and when it fails you have evidence rather than a hunch. +## The only ground truth: `POST /outcome` + +Everything else the router records is a proxy. Structural checks know whether +code *parses*. The local checker guesses whether prose *looks* right. Neither +knows whether the answer did the job — the client does, because it ran the +tests. + +```bash +# id comes from the completion body, or any stream chunk +curl -s localhost:8080/outcome -H 'content-type: application/json' \ + -d '{"request_id":"chatcmpl-...","ok":false,"detail":"tests failed"}' +``` + +Two things make this the highest-value signal available: + +- **It is the only quality signal that survives streaming.** A retry cannot + reach a streamed response because the bytes are already gone; a report + arrives afterwards and works either way. Every agent client streams. +- **Its successes count.** `feedback.py` folds client outcomes in BOTH + directions, unlike checks, where only failures count. A parser's 'ok' means + the code parsed and would inflate scores; a client's 'succeeded' means the + work worked. + +An unknown `request_id` returns 404 rather than being quietly accepted — a +client whose reports go nowhere should find out. + ## Verification: what local compute is actually good for Local inference is a poor substitute for cloud completions here — 7-82x the @@ -417,10 +443,15 @@ that request was two orders of magnitude low. 3. **Retry does not reach streaming.** The iteration budget (`iteration.py`) retries after a failed check, but only on the non-streaming path — once - bytes have gone to the client there is nothing to take back. opencode - streams, so the main workflow gets verification and feedback but not - correction. Buffering to fix that would cost streaming itself, which is a - worse trade for interactive work. + bytes have gone to the client there is nothing to take back. Buffering to + fix that would cost streaming itself, a worse trade for interactive work. + `POST /outcome` is the answer for streamed traffic: it arrives afterwards, + so it works identically either way. + +4. **Nothing calls `/outcome` yet.** The endpoint exists and round-trips, but + a client has to be taught to use it. For opencode that means reporting + after it runs the tests it already runs — which is where the ground truth + is. ## Known open questions diff --git a/dispatcher.py b/dispatcher.py index c210cfb..eca1752 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -551,10 +551,12 @@ def log_verification( model_id: str, provider: str, task_category: Optional[str], + request_id: Optional[str] = None, + *, kind: str, verdict: str, detail: str, - completion_tokens: Optional[int], + completion_tokens: Optional[int] = None, model_attributable: bool = True, ) -> None: """Record what a check said about one completion. @@ -568,13 +570,14 @@ def log_verification( conn.execute( """ INSERT INTO verifications ( - model_id, provider, task_category, kind, verdict, detail, + model_id, provider, request_id, task_category, kind, verdict, detail, completion_tokens, observed_at, model_attributable - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( model_id, provider, + request_id, task_category, kind, verdict, @@ -596,6 +599,7 @@ def run_local_verification( request_text: str, answer: str, completion_tokens: Optional[int], + request_id: Optional[str] = None, ) -> None: """Second-opinion check on an answer nothing structural could judge. @@ -650,8 +654,9 @@ def run_local_verification( return log_verification( - model_id, provider, task_category, "local_llm", - check.verdict, check.detail, completion_tokens, + model_id, provider, task_category, request_id, + kind="local_llm", verdict=check.verdict, detail=check.detail, + completion_tokens=completion_tokens, ) @@ -659,6 +664,7 @@ def log_observation( model_id: str, provider: str, task_category: str, + request_id: Optional[str], prompt_tokens: Optional[int], completion_tokens: Optional[int], telemetry: Telemetry, @@ -670,15 +676,16 @@ def log_observation( conn.execute( """ INSERT INTO energy_observations ( - model_id, provider, task_category, prompt_tokens, completion_tokens, + model_id, provider, request_id, task_category, prompt_tokens, completion_tokens, energy_kwh, energy_btu, avg_power_watts, duration_seconds, attribution_ratio, carbon_g_co2eq, grid_carbon_intensity, grid_id, carbon_source, cost_usd, allowance_remaining_usd, service_tier, observed_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( model_id, provider, + request_id, task_category, prompt_tokens, completion_tokens, @@ -857,6 +864,80 @@ def quota_burn() -> Optional[dict]: } +class OutcomeReport(BaseModel): + """A client telling the router whether an answer actually worked.""" + + request_id: str = Field( + ..., + description=( + "The completion id the router returned (`id` in the response body, " + "and present on every stream chunk)." + ), + ) + ok: bool = Field(..., description="Did the answer actually do the job?") + detail: Optional[str] = Field( + None, description="Optional short note — an error message, what broke." + ) + + +@app.post("/outcome") +def report_outcome(report: OutcomeReport): + """Record whether a completion actually worked. + + This is the only ground truth the router can get. Everything else it + records is a proxy: structural checks know whether code *parses*, the + local checker guesses whether prose *looks* right, and neither knows + whether the answer did the job. The client does — it ran the tests, or + used the answer, or watched it fail. + + It is also the only quality signal that survives streaming. Retries cannot + reach a streamed response because the bytes are already gone, but a report + arrives afterwards and works the same either way — which matters because + every agent client streams. + + Unlike a structural pass, a client-reported SUCCESS is worth recording. + 'ok' from a parser means the code parsed, which is weak evidence and would + inflate scores if counted. 'succeeded' from a client means the work + worked. + """ + conn = _db() + try: + row = conn.execute( + """ + SELECT model_id, provider, task_category FROM energy_observations + WHERE request_id = ? ORDER BY id DESC LIMIT 1 + """, + (report.request_id,), + ).fetchone() + finally: + conn.close() + + if row is None: + # Deliberately a 404 rather than a silent accept: a client whose + # reports go nowhere should find out, not quietly train nothing. + raise HTTPException( + 404, + f"No routed completion found for request_id {report.request_id!r}. " + "Only completions this router dispatched can be reported on.", + ) + + log_verification( + row["model_id"], + row["provider"], + row["task_category"], + report.request_id, + kind="client_outcome", + verdict="succeeded" if report.ok else "failed", + detail=(report.detail or "")[:300], + ) + return { + "recorded": True, + "model_id": row["model_id"], + "task_category": row["task_category"], + "verdict": "succeeded" if report.ok else "failed", + } + + @app.post("/route", response_model=RouteResponse) def route_endpoint(req: TaskRequest): """Classify and pick a model without calling it.""" @@ -1062,8 +1143,9 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): payload = resp.json() usage = payload.get("usage") or {} completion_tokens = usage.get("completion_tokens") + request_id = payload.get("id") log_observation( - current_model, provider, category, + current_model, provider, category, request_id, usage.get("prompt_tokens"), completion_tokens, extract_telemetry(payload), ) @@ -1072,8 +1154,9 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): content = (choice.get("message") or {}).get("content") or "" result = verify_response(content, choice.get("finish_reason")) log_verification( - current_model, provider, category, "structural", - result.verdict, result.detail, completion_tokens, + current_model, provider, category, request_id, + kind="structural", verdict=result.verdict, detail=result.detail, + completion_tokens=completion_tokens, model_attributable=not client_capped, ) @@ -1107,6 +1190,7 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): background.add_task( run_local_verification, current_model, provider, category, _last_user_text(messages), content, completion_tokens, + request_id, ) # Report the model actually used, so the client isn't told 'auto'. @@ -1132,6 +1216,7 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): # every other agent talks to the router — would be the unverified path. content_parts: list[str] = [] finish_reason: Optional[str] = None + stream_request_id: Optional[str] = None upstream = requests.post( url, headers=headers, json=upstream_body, stream=True, timeout=600 ) @@ -1149,6 +1234,8 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): elif raw.startswith("data: ") and raw.strip() != "data: [DONE]": try: chunk = json.loads(raw[6:]) + if chunk.get("id"): + stream_request_id = chunk["id"] if chunk.get("usage"): usage = chunk["usage"] for ch in chunk.get("choices") or []: @@ -1167,6 +1254,7 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): target, provider, category, + stream_request_id, usage.get("prompt_tokens"), usage.get("completion_tokens"), extract_telemetry(collected), @@ -1175,9 +1263,9 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): result = verify_response(streamed, finish_reason) ct = usage.get("completion_tokens") log_verification( - target, provider, category, "structural", - result.verdict, result.detail, ct, - model_attributable=not client_capped, + target, provider, category, stream_request_id, + kind="structural", verdict=result.verdict, detail=result.detail, + completion_tokens=ct, model_attributable=not client_capped, ) # Inline rather than a background task: the generator has already # finished streaming, so the client is not waiting on this. @@ -1187,6 +1275,7 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): run_local_verification( target, provider, category, _last_user_text(messages), streamed, ct, + stream_request_id, ) return StreamingResponse(proxy(), media_type="text/event-stream") @@ -1230,6 +1319,7 @@ def dispatch_endpoint(req: TaskRequest): selected.model_id, selected.provider, decision.classification.task_category, + payload.get("id"), prompt_tokens, completion_tokens, telemetry, diff --git a/feedback.py b/feedback.py index 8012e0c..f22200f 100644 --- a/feedback.py +++ b/feedback.py @@ -8,17 +8,22 @@ accumulates for free as you work. python feedback.py --dry-run # show what would change python feedback.py # apply -**Only failures are folded in, deliberately.** A verification pass is weak -evidence: structural 'ok' means the code parsed, not that it was correct, and -a model emitting syntactically valid nonsense would score 1.0. Recording those -passes would flood `self_eval_score` with 1.0 samples and wash out the -benchmark's hard-won discrimination — the coding categories already sit at -1.00 for every model, and this would spread that flatness everywhere. +Two sources, treated differently, because they are not the same kind of +evidence. -A failure is the opposite: a truncated, malformed or empty response is -definitive, and its cause does not matter for routing. So a model that never -fails keeps its benchmark score untouched, and a model that does fail is -penalized in proportion to how often. That asymmetry is the point. +**Checks (`structural`, `local_llm`) contribute failures only.** A pass there +is weak: structural 'ok' means the code parsed, not that it was correct, and a +model emitting syntactically valid nonsense would score 1.0. Recording those +passes would flood `self_eval_score` with 1.0 samples and wash out the +benchmark's discrimination — the coding categories already sit at 1.00 for +every model, and this would spread that flatness everywhere. A failure is the +opposite: truncated or malformed output is definitive. + +**Client outcomes contribute BOTH ways.** A client reporting success is not +the same claim as a parser reporting success — it ran the tests, or used the +answer, and knows the work worked. That is the only ground truth available +here, so it counts in both directions. It is also the only signal that +survives streaming, where a retry cannot reach. Each failure is applied once. `applied_at` marks consumed rows so re-running cannot penalize a model repeatedly for the same bad response. @@ -36,7 +41,11 @@ from proficiency_store import add_self_eval # Verdicts that count as observed failures. 'unverifiable' is excluded: it # means the checker had nothing to say, which is not evidence about the model. -FAILURE_VERDICTS = ("truncated", "malformed") +FAILURE_VERDICTS = ("truncated", "malformed", "failed") + +# The one verdict that counts as a positive sample. A parser's 'ok' does not +# qualify — it means the code parsed. This means the client ran it. +SUCCESS_VERDICTS = ("succeeded",) # Failures the model did not cause are excluded. The obvious case: a client # that sets max_tokens=40 and gets a truncated answer caused that itself, and @@ -45,8 +54,14 @@ FAILURE_VERDICTS = ("truncated", "malformed") def unapplied_failures(conn: sqlite3.Connection) -> list[sqlite3.Row]: + """Rows that should move a model's score, with the sample each contributes. + + Named for what it mostly is. Successes only enter via client outcomes; a + check passing is not evidence the answer was right. + """ conn.row_factory = sqlite3.Row - placeholders = ",".join("?" * len(FAILURE_VERDICTS)) + scored = FAILURE_VERDICTS + SUCCESS_VERDICTS + placeholders = ",".join("?" * len(scored)) return conn.execute( f""" SELECT id, model_id, provider, task_category, kind, verdict, detail @@ -57,28 +72,38 @@ def unapplied_failures(conn: sqlite3.Connection) -> list[sqlite3.Row]: AND model_attributable = 1 ORDER BY id """, - FAILURE_VERDICTS, + scored, ).fetchall() -def summarize(rows: list[sqlite3.Row]) -> dict[tuple[str, str, str], list[int]]: - """Group failures by (model, provider, category).""" - grouped: dict[tuple[str, str, str], list[int]] = defaultdict(list) +def summarize( + rows: list[sqlite3.Row], +) -> dict[tuple[str, str, str], list[tuple[int, float]]]: + """Group by (model, provider, category) as (row id, sample score) pairs.""" + grouped: dict[tuple[str, str, str], list[tuple[int, float]]] = defaultdict(list) for r in rows: - grouped[(r["model_id"], r["provider"], r["task_category"])].append(r["id"]) + score = 1.0 if r["verdict"] in SUCCESS_VERDICTS else 0.0 + grouped[(r["model_id"], r["provider"], r["task_category"])].append( + (r["id"], score) + ) return grouped def apply_failures(conn: sqlite3.Connection, cfg, grouped, dry_run: bool) -> int: applied = 0 - for (model_id, provider, category), ids in sorted(grouped.items()): - print(f" {model_id:24s} {category:18s} {len(ids)} failure(s)") + for (model_id, provider, category), pairs in sorted(grouped.items()): + ids = [i for i, _ in pairs] + scores = [sc for _, sc in pairs] + wins = sum(1 for sc in scores if sc == 1.0) + print( + f" {model_id:24s} {category:18s} " + f"{len(scores) - wins} failure(s), {wins} success(es)" + ) if dry_run: continue - # One 0.0 sample per observed failure, folded into the running mean by - # proficiency.accumulate — so the penalty scales with failure rate - # rather than replacing the benchmark outright. - add_self_eval(conn, cfg, model_id, provider, category, [0.0] * len(ids)) + # Folded into the running mean by proficiency.accumulate, so the effect + # scales with the observed rate rather than replacing the benchmark. + add_self_eval(conn, cfg, model_id, provider, category, scores) conn.executemany( "UPDATE verifications SET applied_at = datetime('now') WHERE id = ?", [(i,) for i in ids], @@ -102,9 +127,9 @@ def coverage(conn: sqlite3.Connection) -> None: print(" no verifications recorded yet") return total = sum(r["n"] for r in rows) - print(f" {'kind':12s}{'verdict':16s}{'n':>6}{'share':>9}") + print(f" {'kind':16s}{'verdict':16s}{'n':>6}{'share':>9}") for r in rows: - print(f" {r['kind']:12s}{r['verdict']:16s}{r['n']:>6}{r['n']/total*100:>8.1f}%") + print(f" {r['kind']:16s}{r['verdict']:16s}{r['n']:>6}{r['n']/total*100:>8.1f}%") unver = sum(r["n"] for r in rows if r["verdict"] == "unverifiable") if total and unver / total > 0.8: print( @@ -129,17 +154,17 @@ def main() -> int: rows = unapplied_failures(conn) if not rows: - print("no unapplied failures — nothing to fold in") + print("no unapplied signals — nothing to fold in") conn.close() return 0 grouped = summarize(rows) print(f"{'would apply' if args.dry_run else 'applying'} " - f"{len(rows)} failure(s) across {len(grouped)} (model, category) pair(s):") + f"{len(rows)} sample(s) across {len(grouped)} (model, category) pair(s):") applied = apply_failures(conn, cfg, grouped, args.dry_run) conn.close() if not args.dry_run: - print(f"\napplied {applied} failure sample(s)") + print(f"\napplied {applied} sample(s)") return 0 diff --git a/schema.sql b/schema.sql index 25197e2..4343dc6 100644 --- a/schema.sql +++ b/schema.sql @@ -78,6 +78,10 @@ CREATE TABLE IF NOT EXISTS energy_observations ( id INTEGER PRIMARY KEY AUTOINCREMENT, model_id TEXT NOT NULL, provider TEXT NOT NULL, + -- The provider's completion id (chatcmpl-...). The client receives this in + -- the response body and in every stream chunk, so it is the join key that + -- lets a client report back later whether the answer actually worked. + request_id TEXT, task_category TEXT, prompt_tokens INTEGER, completion_tokens INTEGER, @@ -136,9 +140,14 @@ CREATE TABLE IF NOT EXISTS verifications ( id INTEGER PRIMARY KEY AUTOINCREMENT, model_id TEXT NOT NULL, provider TEXT NOT NULL, + request_id TEXT, -- provider completion id, for client reports task_category TEXT, - kind TEXT NOT NULL, -- 'structural' | 'local_llm' - verdict TEXT NOT NULL, -- 'ok' | 'truncated' | 'malformed' | 'unverifiable' + kind TEXT NOT NULL, -- 'structural' | 'local_llm' | 'client_outcome' + -- 'succeeded'/'failed' come only from client_outcome and are ground truth: + -- the client ran the code, or used the answer, and knows. Every other + -- verdict is a proxy for that. + verdict TEXT NOT NULL, -- ok | truncated | malformed | unverifiable + -- | succeeded | failed detail TEXT, completion_tokens INTEGER, -- what a wasted answer cost, for the payoff sum observed_at TEXT NOT NULL, @@ -155,6 +164,7 @@ CREATE TABLE IF NOT EXISTS verifications ( CREATE INDEX IF NOT EXISTS idx_verifications_model ON verifications (model_id, provider); CREATE INDEX IF NOT EXISTS idx_verifications_verdict ON verifications (verdict); +CREATE INDEX IF NOT EXISTS idx_observations_request ON energy_observations (request_id); CREATE INDEX IF NOT EXISTS idx_models_provider ON models (provider); CREATE INDEX IF NOT EXISTS idx_models_availability ON models (availability); diff --git a/tests/test_feedback.py b/tests/test_feedback.py index aafc58c..cce643b 100644 --- a/tests/test_feedback.py +++ b/tests/test_feedback.py @@ -186,3 +186,55 @@ def test_attributable_and_capped_failures_are_separated(db): rows = unapplied_failures(db) assert len(rows) == 1 assert rows[0]["verdict"] == "malformed" + + +# --- client outcomes: the only two-way evidence ---------------------------- + +def _outcome(conn, verdict, category="coding_general"): + conn.execute( + """ + INSERT INTO verifications (model_id, provider, task_category, kind, verdict, observed_at) + VALUES ('m', 'nw', ?, 'client_outcome', ?, '2026-08-17T00:00:00+00:00') + """, + (category, verdict), + ) + conn.commit() + + +def test_a_client_reported_success_counts_as_a_positive_sample(db): + # Unlike a parser's 'ok' — which means the code parsed — this means the + # client ran it and the work worked. That is the only ground truth here. + from proficiency_store import add_self_eval + + add_self_eval(db, CFG, "m", "nw", "coding_general", [0.5] * 3) + _outcome(db, "succeeded") + apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False) + row = _prof(db) + assert row["n"] == 4 + assert row["s"] == pytest.approx((0.5 * 3 + 1.0) / 4) + + +def test_a_client_reported_failure_counts_against(db): + from proficiency_store import add_self_eval + + add_self_eval(db, CFG, "m", "nw", "coding_general", [1.0] * 3) + _outcome(db, "failed") + apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False) + assert _prof(db)["s"] == pytest.approx(0.75) + + +def test_a_structural_pass_still_contributes_nothing(db): + # The asymmetry that matters: 'ok' from a parser is weak evidence and + # would inflate every score toward the ceiling + for _ in range(10): + _verify(db, "ok") + assert unapplied_failures(db) == [] + + +def test_successes_and_failures_mix_into_one_rate(db): + for _ in range(3): + _outcome(db, "succeeded") + _outcome(db, "failed") + apply_failures(db, CFG, summarize(unapplied_failures(db)), dry_run=False) + row = _prof(db) + assert (row["n"], row["s"]) == (4, pytest.approx(0.75)) -- 2.49.1 From 6e729ad670ef63b0acd399e57da9aabe6c69cc15 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Mon, 17 Aug 2026 23:51:14 -0400 Subject: [PATCH 18/20] feat: parallel-safe outcome attribution, and an opencode plugin to feed it Closes the ground-truth loop. deploy/opencode-plugin/router-outcome.js hooks tool.execute.after, watches for test and build commands, and reports pass/fail to /outcome. opencode already runs your tests; this is what makes the result reach routing. Command detection is deliberately narrow -- pytest, npm test, cargo, go, ruff, mypy, tsc and friends. A failing `ls` says nothing about model quality, and a false signal is worse than none because it trains the router on noise. Verdict comes from exit status plus text signatures for tools that exit 0 while reporting failures, with "0 failed" and "no errors" guarded against. The router being unreachable never breaks a session. Attribution is the hard part, and two assumptions failed under test. The first fingerprint design keyed on the system prompt. One real opencode run produced TWO distinct keys, because it runs several agents with different prompts -- so that fingerprint identifies AGENTS, not sessions, and would have refused every single run forever. A permanent false positive dressed as safety. The second assumption was that opencode states its project root up front. Capturing a real request showed it does not. Directory is now derived from the file paths an agent touches across the whole conversation, counting every ancestor so the shared project root wins over any one subdirectory, and stripping trailing filenames so a file is never mistaken for a directory. A single mention is not enough; corroboration is required. When a report cannot be matched by directory and more than one conversation was active in the window, /outcome answers 409 and records nothing. Refusing beats guessing: a misattributed failure penalizes a model for work it never did, and this project has already recorded false failures twice from harness bugs that took measurement to catch. The window is 120 seconds, not 30 minutes. At 30 it swept in traffic from earlier in the same work session and refused a legitimate report -- observed directly, not theorised. Tests 232 -> 243. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa --- config.py | 2 + config.yaml | 8 + deploy/opencode-plugin/router-outcome.js | 108 ++++++++++++ dispatcher.py | 212 ++++++++++++++++++++--- schema.sql | 10 ++ tests/test_session_identity.py | 99 +++++++++++ 6 files changed, 412 insertions(+), 27 deletions(-) create mode 100644 deploy/opencode-plugin/router-outcome.js create mode 100644 tests/test_session_identity.py diff --git a/config.py b/config.py index 5d6cb8d..090f018 100644 --- a/config.py +++ b/config.py @@ -131,6 +131,7 @@ class VerificationConfig(BaseModel): min_completion_tokens: int = 600 timeout_seconds: int = 60 max_output_tokens: int = 1024 + outcome_attribution_window_seconds: int = 120 class EscalationConfig(BaseModel): @@ -179,6 +180,7 @@ class ClassifierConfig(BaseModel): timeout_seconds: int temperature: float = 0.0 max_output_tokens: int = 1024 + outcome_attribution_window_seconds: int = 120 fallback_tier: int = 2 fallback_category: str = "general_chat" response_format: str diff --git a/config.yaml b/config.yaml index 8d2eb93..4d8ec11 100644 --- a/config.yaml +++ b/config.yaml @@ -142,6 +142,14 @@ verification: timeout_seconds: 60 max_output_tokens: 1024 + # How far back /outcome looks when a report carries no request_id and no + # matching directory. A test run follows the completion that caused it within + # seconds, so this is deliberately short: a wide window sweeps in sessions + # that finished long ago and makes every report look ambiguous. If more than + # one conversation was active inside it, the report is refused rather than + # guessed at. + outcome_attribution_window_seconds: 120 + freshness: stale_after_days: 3 # Router refuses to route to a model whose row is stale/deprecated, diff --git a/deploy/opencode-plugin/router-outcome.js b/deploy/opencode-plugin/router-outcome.js new file mode 100644 index 0000000..3e258e3 --- /dev/null +++ b/deploy/opencode-plugin/router-outcome.js @@ -0,0 +1,108 @@ +/** + * Report test outcomes back to the local LLM router. + * + * This closes the only loop the router cannot close by itself. Everything it + * records on its own is a proxy: structural checks know whether code *parses*, + * the local checker guesses whether prose *looks* right. Neither knows whether + * the answer did the job. opencode does — it runs your tests. + * + * Hooks `tool.execute.after`, watches for test/build commands, and POSTs the + * pass/fail to the router's /outcome endpoint. The router folds client + * outcomes into proficiency in BOTH directions (a client's "succeeded" means + * the work worked, unlike a parser's "ok" which only means it parsed), so this + * is what eventually makes routing discriminate on quality. + * + * Install: + * mkdir -p ~/.config/opencode/plugins + * cp router-outcome.js ~/.config/opencode/plugins/ + * + * Or per-project, in .opencode/plugins/. + * + * PARALLEL SESSIONS: the report carries this session's directory, which the + * router matches against the working directory it sees in the conversation. + * That is exact even with several sessions running. If it cannot match, and + * more than one conversation has been routed recently, the router answers 409 + * and records nothing rather than guessing — a misattributed failure would + * penalize a model for work it never did. + */ + +const ROUTER = process.env.LLM_ROUTER_URL || "http://127.0.0.1:8080"; + +// Commands whose exit status is a real verdict on the work. Deliberately +// narrow: a failing `ls` says nothing about model quality, and a false signal +// is worse than no signal — it trains the router on noise. +const TEST_COMMAND = new RegExp( + [ + "\\bpytest\\b", + "\\bunittest\\b", + "\\bnpm\\s+(run\\s+)?test\\b", + "\\bpnpm\\s+(run\\s+)?test\\b", + "\\byarn\\s+test\\b", + "\\bvitest\\b", + "\\bjest\\b", + "\\bcargo\\s+(test|check|build)\\b", + "\\bgo\\s+(test|build|vet)\\b", + "\\bmake\\s+(test|check)\\b", + "\\bmvn\\s+test\\b", + "\\bgradle\\s+test\\b", + "\\btsc\\b", + "\\bruff\\b", + "\\bmypy\\b", + "\\beslint\\b", + ].join("|"), +); + +// Failure signatures, for tools that exit 0 while reporting failures. +const FAILURE_TEXT = + /\b(\d+\s+failed|FAILED|FAIL\b|Traceback \(most recent call last\)|error(s)?:|panic:|AssertionError|✗|✖)/; + +function looksFailed(output) { + const exit = output?.exitCode ?? output?.exit_code; + if (typeof exit === "number" && exit !== 0) return true; + const text = `${output?.stdout ?? ""}\n${output?.stderr ?? ""}\n${ + typeof output?.output === "string" ? output.output : "" + }`; + // "0 failed" and "no errors" must not trip the failure regex. + if (/\b0 failed\b|\bno errors?\b/i.test(text)) return false; + return FAILURE_TEXT.test(text); +} + +function commandOf(input) { + const args = input?.args ?? input?.arguments ?? {}; + return args.command ?? args.cmd ?? args.script ?? ""; +} + +export const RouterOutcome = async ({ directory }) => { + return { + "tool.execute.after": async (input, output) => { + // Only shell-ish tools carry a command whose exit status is a verdict. + const command = commandOf(input); + if (!command || !TEST_COMMAND.test(command)) return; + + const ok = !looksFailed(output); + const detail = `${command.slice(0, 120)}${ok ? " — passed" : " — failed"}`; + + try { + const res = await fetch(`${ROUTER}/outcome`, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ ok, detail, source: directory }), + // The router being down must never break the user's session. + signal: AbortSignal.timeout(3000), + }); + if (res.status === 409) { + // Several sessions active and the directory did not match one the + // router had seen. Dropping the sample is the correct outcome. + console.error( + "[router-outcome] ambiguous session; outcome not recorded", + ); + } else if (!res.ok && res.status !== 404) { + console.error(`[router-outcome] ${res.status} reporting outcome`); + } + } catch (err) { + // Swallowed on purpose: a reporting failure is not the user's problem. + console.error(`[router-outcome] could not reach ${ROUTER}: ${err.message}`); + } + }, + }; +}; diff --git a/dispatcher.py b/dispatcher.py index eca1752..1715885 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -36,6 +36,8 @@ from __future__ import annotations import json import os +import hashlib +import re import sqlite3 import sys from datetime import datetime, timezone @@ -665,6 +667,9 @@ def log_observation( provider: str, task_category: str, request_id: Optional[str], + session_key: Optional[str] = None, + session_dir: Optional[str] = None, + *, prompt_tokens: Optional[int], completion_tokens: Optional[int], telemetry: Telemetry, @@ -676,16 +681,19 @@ def log_observation( conn.execute( """ INSERT INTO energy_observations ( - model_id, provider, request_id, task_category, prompt_tokens, completion_tokens, + model_id, provider, request_id, session_key, session_dir, + task_category, prompt_tokens, completion_tokens, energy_kwh, energy_btu, avg_power_watts, duration_seconds, attribution_ratio, carbon_g_co2eq, grid_carbon_intensity, grid_id, carbon_source, cost_usd, allowance_remaining_usd, service_tier, observed_at - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( model_id, provider, request_id, + session_key, + session_dir, task_category, prompt_tokens, completion_tokens, @@ -867,17 +875,63 @@ def quota_burn() -> Optional[dict]: class OutcomeReport(BaseModel): """A client telling the router whether an answer actually worked.""" - request_id: str = Field( - ..., + request_id: Optional[str] = Field( + None, description=( "The completion id the router returned (`id` in the response body, " - "and present on every stream chunk)." + "and on every stream chunk). Omit it and the report attaches to the " + "most recent completion this router served — see the endpoint docs " + "for when that is safe." ), ) ok: bool = Field(..., description="Did the answer actually do the job?") detail: Optional[str] = Field( None, description="Optional short note — an error message, what broke." ) + source: Optional[str] = Field( + None, description="Where the report came from, e.g. the project directory." + ) + + +AMBIGUOUS = object() + + +def _most_recent_if_unambiguous(conn: sqlite3.Connection): + """The latest completion, but only when one conversation is active. + + Guessing here is how quality data gets corrupted: attribute a failing test + run to the wrong session and a model is penalized for work it never did. + Corrupt evidence is worse than missing evidence — this project has already + recorded false failures twice from harness bugs, and both took measurement + to catch. + + So when two conversations have been served inside the window, this returns + AMBIGUOUS and the caller refuses rather than picks. + + The window is short on purpose. A test run follows the completion that + caused it within seconds; a wide window sweeps in sessions that finished + long ago and makes every report look ambiguous. Observed directly: at 30 + minutes, a real opencode run was refused because of test traffic from + earlier in the same session of work. + """ + window = cfg.verification.outcome_attribution_window_seconds + recent = conn.execute( + f""" + SELECT id, request_id, model_id, provider, task_category, session_key + FROM energy_observations + WHERE request_id IS NOT NULL + AND task_category != ? + AND observed_at > datetime('now', '-{int(window)} seconds') + ORDER BY id DESC LIMIT 50 + """, + (SEED_CATEGORY,), + ).fetchall() + if not recent: + return None + keys = {r["session_key"] for r in recent if r["session_key"]} + if len(keys) > 1: + return AMBIGUOUS + return recent[0] @app.post("/outcome") @@ -902,36 +956,70 @@ def report_outcome(report: OutcomeReport): """ conn = _db() try: - row = conn.execute( - """ - SELECT model_id, provider, task_category FROM energy_observations - WHERE request_id = ? ORDER BY id DESC LIMIT 1 - """, - (report.request_id,), - ).fetchone() + if report.request_id: + row = conn.execute( + """ + SELECT id, request_id, model_id, provider, task_category + FROM energy_observations + WHERE request_id = ? ORDER BY id DESC LIMIT 1 + """, + (report.request_id,), + ).fetchone() + elif report.source: + # The client told us where it is. If any completion came from a + # conversation naming that directory, this is exact even with + # several sessions running. + row = conn.execute( + """ + SELECT id, request_id, model_id, provider, task_category + FROM energy_observations + WHERE session_dir = ? AND request_id IS NOT NULL + AND task_category != ? + ORDER BY id DESC LIMIT 1 + """, + (report.source, SEED_CATEGORY), + ).fetchone() + if row is None: + row = _most_recent_if_unambiguous(conn) + else: + row = _most_recent_if_unambiguous(conn) finally: conn.close() + if row is AMBIGUOUS: + raise HTTPException( + 409, + "More than one conversation has been routed recently, so this " + "report cannot be attributed with confidence. Pass request_id (the " + "`id` on the completion or any stream chunk) to disambiguate. " + "Refusing rather than guessing: a misattributed outcome penalizes a " + "model for work it never did.", + ) + if row is None: # Deliberately a 404 rather than a silent accept: a client whose # reports go nowhere should find out, not quietly train nothing. raise HTTPException( 404, f"No routed completion found for request_id {report.request_id!r}. " - "Only completions this router dispatched can be reported on.", + "Only completions this router dispatched can be reported on." + if report.request_id + else "No completions have been routed yet, so there is nothing to " + "report on.", ) log_verification( row["model_id"], row["provider"], row["task_category"], - report.request_id, + row["request_id"], kind="client_outcome", verdict="succeeded" if report.ok else "failed", - detail=(report.detail or "")[:300], + detail=" | ".join(x for x in (report.detail, report.source) if x)[:300], ) return { "recorded": True, + "request_id": row["request_id"], "model_id": row["model_id"], "task_category": row["task_category"], "verdict": "succeeded" if report.ok else "failed", @@ -946,6 +1034,73 @@ def route_endpoint(req: TaskRequest): # --- OpenAI-compatible surface ------------------------------------------- +# Absolute paths that look like a project root, for pulling a working +# directory out of an agent's system prompt. +CWD_RE = re.compile(r"(/(?:home|Users|tmp|opt|srv|var)/[\w.@-]+(?:/[\w.@-]+)*)") + + +def session_fingerprint(messages: list[dict]) -> Optional[str]: + """A stable id for the conversation this request belongs to. + + Hashes the opening message, which an agent client holds constant for the + life of a session (it is the system prompt) and which differs between + sessions. That lets the router notice when two clients are talking to it + at once without either of them saying so. + """ + for m in messages: + content = m.get("content") + if isinstance(content, list): + content = " ".join( + p.get("text", "") for p in content if isinstance(p, dict) + ) + if isinstance(content, str) and content.strip(): + return hashlib.sha256(content[:4000].encode()).hexdigest()[:16] + return None + + +def session_directory(messages: list[dict]) -> Optional[str]: + """The working directory this conversation is about, if it reveals one. + + Scans the WHOLE conversation, not just the system prompt. opencode does not + state its project root up front — verified by capturing a real request — + but a coding agent names files inside its project constantly, in tool + calls and results. The directory containing the most of those paths is the + working directory. + + This is what makes outcome attribution exact under concurrent sessions: a + report carries its own directory, and matching it beats guessing which + conversation was most recent. + """ + counts: dict[str, int] = {} + for m in messages[-40:]: + content = m.get("content") + if isinstance(content, list): + content = " ".join( + p.get("text", "") for p in content if isinstance(p, dict) + ) + if not isinstance(content, str): + continue + for path in CWD_RE.findall(content[:4000]): + parts = path.split("/") + # Drop a trailing filename: a path ending in something with an + # extension names a file, and its directory is what a client + # reports as its working directory. + if "." in parts[-1]: + parts = parts[:-1] + # Count every ancestor, so the shared project root accumulates the + # most hits even when each file sits in a different subdirectory. + for depth in range(3, len(parts) + 1): + counts["/".join(parts[:depth])] = counts.get("/".join(parts[:depth]), 0) + 1 + if not counts: + return None + # Deepest directory that still explains most of the paths seen. Ties go to + # the longer path so sibling projects under one parent stay distinct. + best = max(counts.values()) + threshold = max(2, best * 0.6) + winners = [p for p, n in counts.items() if n >= threshold] + return max(winners, key=len) if winners else None + + def estimate_prompt_tokens(messages: list[dict]) -> int: """Floor estimate of a conversation's token count, from its characters. @@ -1098,6 +1253,9 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): # A truncated answer under a cap the CLIENT chose is not the model failing. client_capped = body.get("max_tokens") is not None + # Who is talking to us, so concurrent clients stay distinguishable. + session_key = session_fingerprint(messages) + session_dir = session_directory(messages) upstream_body = {**body, "model": target} streaming = bool(body.get("stream")) if streaming: @@ -1146,8 +1304,10 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): request_id = payload.get("id") log_observation( current_model, provider, category, request_id, - usage.get("prompt_tokens"), completion_tokens, - extract_telemetry(payload), + session_key, session_dir, + prompt_tokens=usage.get("prompt_tokens"), + completion_tokens=completion_tokens, + telemetry=extract_telemetry(payload), ) choice = (payload.get("choices") or [{}])[0] @@ -1251,13 +1411,11 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): upstream.close() # Logged even on a client disconnect — the energy was spent. log_observation( - target, - provider, - category, - stream_request_id, - usage.get("prompt_tokens"), - usage.get("completion_tokens"), - extract_telemetry(collected), + target, provider, category, stream_request_id, + session_key, session_dir, + prompt_tokens=usage.get("prompt_tokens"), + completion_tokens=usage.get("completion_tokens"), + telemetry=extract_telemetry(collected), ) streamed = "".join(content_parts) result = verify_response(streamed, finish_reason) @@ -1320,9 +1478,9 @@ def dispatch_endpoint(req: TaskRequest): selected.provider, decision.classification.task_category, payload.get("id"), - prompt_tokens, - completion_tokens, - telemetry, + prompt_tokens=prompt_tokens, + completion_tokens=completion_tokens, + telemetry=telemetry, ) choices = payload.get("choices") or [{}] diff --git a/schema.sql b/schema.sql index 4343dc6..375d8f9 100644 --- a/schema.sql +++ b/schema.sql @@ -82,6 +82,16 @@ CREATE TABLE IF NOT EXISTS energy_observations ( -- the response body and in every stream chunk, so it is the join key that -- lets a client report back later whether the answer actually worked. request_id TEXT, + -- Fingerprint of the conversation this completion belongs to, derived from + -- its opening message. Stable across a session's turns and distinct + -- between sessions, so the router can tell whether two clients are active + -- WITHOUT the clients cooperating. Used to refuse ambiguous outcome + -- attribution rather than guess. + session_key TEXT, + -- Working directory, when the conversation reveals one. Agent clients + -- usually put the cwd in their system prompt, which makes an outcome + -- report from that directory attributable even under concurrency. + session_dir TEXT, task_category TEXT, prompt_tokens INTEGER, completion_tokens INTEGER, diff --git a/tests/test_session_identity.py b/tests/test_session_identity.py new file mode 100644 index 0000000..25678e0 --- /dev/null +++ b/tests/test_session_identity.py @@ -0,0 +1,99 @@ +"""Tests for session identity — telling concurrent clients apart. + +The router has to attribute an outcome report to the right conversation +without the client cooperating. Getting this wrong is worse than getting no +report at all: a misattributed test failure penalizes a model for work it +never did, and this project has already recorded false failures twice from +harness bugs that took measurement to catch. + +So the rule under test is: identify exactly when possible, refuse when not. +""" + +from dispatcher import session_directory, session_fingerprint + + +SYS_A = "You are opencode. Project root: /home/alee/Sources/6krrt" +SYS_B = "You are opencode. Project root: /tmp/otherproj" + + +def _conv(system, *turns): + return [{"role": "system", "content": system}] + [ + {"role": "user", "content": t} for t in turns + ] + + +# --- fingerprints --------------------------------------------------------- + +def test_two_sessions_are_distinguishable(): + assert session_fingerprint(_conv(SYS_A)) != session_fingerprint(_conv(SYS_B)) + + +def test_a_fingerprint_is_stable_as_a_session_grows(): + # It must survive the conversation accumulating turns, or every message + # would look like a new session + short = session_fingerprint(_conv(SYS_A, "first")) + long = session_fingerprint(_conv(SYS_A, "first", "second", "third")) + assert short == long + + +def test_an_empty_conversation_has_no_fingerprint(): + assert session_fingerprint([]) is None + assert session_fingerprint([{"role": "user", "content": " "}]) is None + + +def test_multimodal_content_still_fingerprints(): + msgs = [{"role": "system", "content": [{"type": "text", "text": SYS_A}]}] + assert session_fingerprint(msgs) is not None + + +# --- directory extraction ------------------------------------------------- + +def test_a_single_mention_is_not_enough_to_claim_a_directory(): + # One stray path proves nothing. Requiring corroboration keeps an + # unrelated path in a chat message from hijacking attribution. + assert session_directory(_conv("see /home/alee/Sources/6krrt once")) is None + + +def test_no_directory_when_the_prompt_names_none(): + assert session_directory(_conv("You are a helpful assistant.")) is None + + +def test_only_the_opening_messages_are_scanned(): + # A path mentioned deep in a long conversation is not the project root + msgs = _conv("You are an assistant.", "x", "y") + [ + {"role": "user", "content": "check /tmp/some/other/path"} + ] + assert session_directory(msgs) is None + + +# --- directory from the whole conversation -------------------------------- + +def test_directory_comes_from_the_files_the_agent_touches(): + # opencode does not state its project root up front — verified by + # capturing a real request — but a coding agent names files inside its + # project constantly. + conv = [ + {"role": "assistant", "content": "reading /tmp/proj/inventory.py"}, + {"role": "user", "content": "ran /tmp/proj/test_inventory.py"}, + {"role": "assistant", "content": "editing /tmp/proj/inventory.py"}, + ] + assert session_directory(conv) == "/tmp/proj" + + +def test_a_filename_is_never_mistaken_for_a_directory(): + conv = [{"role": "assistant", "content": "/tmp/proj/only.py " * 3}] + assert session_directory(conv) == "/tmp/proj" + + +def test_the_shared_root_wins_over_any_one_subdirectory(): + conv = [{"role": "assistant", "content": + "/home/alee/Sources/6krrt/routing.py " + "/home/alee/Sources/6krrt/tests/test_routing.py " + "/home/alee/Sources/6krrt/scoring.py"}] + assert session_directory(conv) == "/home/alee/Sources/6krrt" + + +def test_two_projects_stay_distinct(): + a = [{"role": "assistant", "content": "/tmp/projA/x.py /tmp/projA/y.py /tmp/projA/z.py"}] + b = [{"role": "assistant", "content": "/tmp/projB/x.py /tmp/projB/y.py /tmp/projB/z.py"}] + assert session_directory(a) != session_directory(b) -- 2.49.1 From f0d9bfb81c85fa58b1079c026cac99a3b8f9090c Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Fri, 21 Aug 2026 19:02:07 -0400 Subject: [PATCH 19/20] fix: an agent turn that calls a tool is not a failed answer Both verification paths had mirror halves of the same blind spot, and real traffic is what found it. On the first genuine agent session through this router -- 63 completions that shipped a working feature with 349 passing tests, clean mypy and clean ruff -- the structural checker recorded "malformed: empty response" 29 times and the local LLM checker called "cuts off mid-sentence" on 8 of the 9 answers it graded. Both were describing the same thing from opposite sides: a turn that ends by calling a tool. Its text content is empty, or a half-sentence before the call, and both are correct behaviour rather than a defect. verify_response and worth_local_check now take has_tool_calls, supplied on the non-streaming path from message.tool_calls and accumulated on the streaming path from delta.tool_calls. In verify_response the check outranks even finish_reason == 'length', because stopping mid-sentence at a call boundary is a call boundary, not a budget overrun. worth_local_check declines outright, which also stops paying ~6s of local inference to mis-grade a tool call. Had feedback.py run against those rows it would have applied ~12 false failures to the two models that had just done the work. That is the FOURTH harness bug in this project that would have scored the rig rather than the model, and the first one caught by real traffic instead of a synthetic test. The pre-fix rows are kept with model_attributable = 0 so the record survives without steering routing. client_capped was over-applied in the same area. It marked EVERY verdict non-attributable whenever the client set max_tokens, and opencode always sets it, so genuine failures were invisible to feedback for the entire main workflow. A client's token cap explains a truncated verdict and nothing else; a model emitting unparseable code owes nothing to the client's budget. It is now scoped to exactly that verdict. Also recorded: outcome attribution resolves the session directory by path histogram, and on this session that picked .venv/.../site-packages/c2pa 24 times over ~/Sources/fieldwitness 22, because reading a dependency's source outweighed editing the project. It degrades safely -- 31 reports accepted, 3 refused as ambiguous rather than misattributed -- but the heuristic needs to weight writes over reads. Tests 243 -> 249. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ --- CLAUDE.md | 45 ++++++++++++++++++++++++++++++++++---- dispatcher.py | 35 ++++++++++++++++++++++++----- tests/test_verification.py | 41 ++++++++++++++++++++++++++++++++++ verification.py | 41 +++++++++++++++++++++++++++++----- 4 files changed, 146 insertions(+), 16 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 74ab3ff..e2e3ccc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -376,6 +376,37 @@ parse it. The local LLM check runs only on answers above its response, because it costs ~15% of a median 193-token answer and only pays above ~600 tokens. +### An agent turn is not a prose answer + +Both checkers had mirror halves of the same blind spot, and real traffic is +what found it. On the first genuine agent session (63 completions, a shipped +feature, 349 passing tests, clean mypy and ruff): + +| checker | called it | actually | +|---|---|---| +| structural | `malformed: empty response` x29 | turns that ended in a tool call | +| local LLM | `cuts off mid-sentence` x8 of 9 | same turns, judged from the other side | + +A turn that calls a tool has empty or half-finished text **by design**. Both +paths now take `has_tool_calls` and return `unverifiable`; in `verify_response` +that check outranks even `finish_reason == 'length'`, because stopping +mid-sentence at a call boundary is not a budget overrun. `worth_local_check` +declines outright, which also stops paying ~6s of local inference to +mis-grade a tool call. + +Had `feedback.py` run before this, it would have applied ~12 false failures to +the models that had just shipped the feature. That is the **fourth** harness +bug in this project that would have scored the rig rather than the model, and +the first caught by real traffic instead of a synthetic test. The pre-fix rows +are kept but set `model_attributable = 0`, so the record survives without +steering routing. + +**`client_capped` was also over-applied.** It marked *every* verdict +non-attributable whenever the client set `max_tokens` — and opencode always +does — so real failures were invisible to feedback. A client's token cap +explains a `truncated` verdict and nothing else; it is now scoped to exactly +that. + `feedback.py` folds observed failures into `proficiency`, so routing learns from your traffic rather than only the 23-task benchmark. Only **failures** are folded in: a structural 'ok' means the code parsed, not that it was @@ -448,10 +479,16 @@ that request was two orders of magnitude low. `POST /outcome` is the answer for streamed traffic: it arrives afterwards, so it works identically either way. -4. **Nothing calls `/outcome` yet.** The endpoint exists and round-trips, but - a client has to be taught to use it. For opencode that means reporting - after it runs the tests it already runs — which is where the ground truth - is. +4. **Session-directory attribution picks the wrong directory.** The opencode + plugin now genuinely reports — 31 accepted, 3 refused as ambiguous, 22 + `succeeded` / 4 `failed` from real test runs, all folded in. But the + "most frequent path" heuristic resolved the session to + `.venv/lib/python3.14/site-packages/c2pa` 24 times versus + `~/Sources/fieldwitness` 22, because reading a dependency's source + outweighed editing the project. It degrades safely — an ambiguous session + is refused rather than misattributed, which is why the 3 × 409 — but the + heuristic needs to weight *writes* over reads, or anchor on the client's + cwd instead of a path histogram. ## Known open questions diff --git a/dispatcher.py b/dispatcher.py index 1715885..9fc0468 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -1312,12 +1312,21 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): choice = (payload.get("choices") or [{}])[0] content = (choice.get("message") or {}).get("content") or "" - result = verify_response(content, choice.get("finish_reason")) + has_tools = bool((choice.get("message") or {}).get("tool_calls")) + result = verify_response( + content, choice.get("finish_reason"), has_tool_calls=has_tools + ) log_verification( current_model, provider, category, request_id, kind="structural", verdict=result.verdict, detail=result.detail, completion_tokens=completion_tokens, - model_attributable=not client_capped, + # A client's token cap explains a TRUNCATED answer, nothing + # else. Applied to every verdict it silently excluded all + # traffic from clients that always set max_tokens — opencode + # does — which made real failures invisible to feedback. + model_attributable=not ( + client_capped and result.verdict == "truncated" + ), ) if not result.failed or attempts_used >= budget: @@ -1345,7 +1354,8 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): attempts_used += 1 if cfg.verification.local_llm_enabled and worth_local_check( - result.verdict, completion_tokens, cfg.verification.min_completion_tokens + result.verdict, completion_tokens, + cfg.verification.min_completion_tokens, has_tools, ): background.add_task( run_local_verification, current_model, provider, category, @@ -1377,6 +1387,7 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): content_parts: list[str] = [] finish_reason: Optional[str] = None stream_request_id: Optional[str] = None + stream_tool_calls = False upstream = requests.post( url, headers=headers, json=upstream_body, stream=True, timeout=600 ) @@ -1399,6 +1410,8 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): if chunk.get("usage"): usage = chunk["usage"] for ch in chunk.get("choices") or []: + if (ch.get("delta") or {}).get("tool_calls"): + stream_tool_calls = True piece = (ch.get("delta") or {}).get("content") if piece: content_parts.append(piece) @@ -1418,17 +1431,27 @@ def chat_completions(body: dict[str, Any], background: BackgroundTasks): telemetry=extract_telemetry(collected), ) streamed = "".join(content_parts) - result = verify_response(streamed, finish_reason) + result = verify_response( + streamed, finish_reason, has_tool_calls=stream_tool_calls + ) ct = usage.get("completion_tokens") log_verification( target, provider, category, stream_request_id, kind="structural", verdict=result.verdict, detail=result.detail, - completion_tokens=ct, model_attributable=not client_capped, + completion_tokens=ct, + # A client's token cap explains a TRUNCATED answer, nothing + # else. Applied to every verdict it silently excluded all + # traffic from clients that always set max_tokens — opencode + # does — which made real failures invisible to feedback. + model_attributable=not ( + client_capped and result.verdict == "truncated" + ), ) # Inline rather than a background task: the generator has already # finished streaming, so the client is not waiting on this. if cfg.verification.local_llm_enabled and worth_local_check( - result.verdict, ct, cfg.verification.min_completion_tokens + result.verdict, ct, cfg.verification.min_completion_tokens, + stream_tool_calls, ): run_local_verification( target, provider, category, diff --git a/tests/test_verification.py b/tests/test_verification.py index d4b63c9..73ec994 100644 --- a/tests/test_verification.py +++ b/tests/test_verification.py @@ -256,3 +256,44 @@ def test_empty_response_is_decided_in_code_not_by_a_model(): def test_whitespace_only_beats_the_no_blocks_path(): # It must not fall through to 'unverifiable' just for lacking code fences assert verify_response("\n\n").verdict != "unverifiable" + + +# --- agent turns are not prose answers ------------------------------------ + +def test_a_tool_call_turn_is_not_a_failure(): + # Real measurement: 29 of 63 completions in one agent session were + # recorded as "malformed: empty response" purely for making tool calls. + # Folding those in would have penalized the models that shipped a working + # feature with 349 passing tests. + v = verify_response("", has_tool_calls=True) + assert v.verdict == "unverifiable" + assert not v.failed + + +def test_a_half_sentence_before_a_tool_call_is_not_a_failure(): + v = verify_response("Let me check the config", has_tool_calls=True) + assert not v.failed + + +def test_tool_call_outranks_even_truncation(): + # An agent turn ending in a tool call routinely reports finish_reason + # length; that is the call boundary, not a cut-off answer + assert verify_response("x", "length", has_tool_calls=True).verdict == "unverifiable" + + +def test_a_genuinely_empty_answer_is_still_a_failure(): + # The fix must not blind the checker to real emptiness + assert verify_response("", has_tool_calls=False).verdict == "malformed" + + +def test_broken_code_is_still_caught_when_a_tool_was_not_called(): + assert verify_response("```python\ndef f(\n```").verdict == "malformed" + + +def test_the_local_checker_skips_tool_turns_too(): + # Mirror-image blind spot: "cuts off mid-sentence" is exactly what a turn + # looks like when it ends by calling a tool. Observed producing 8 false + # failures in 9 checks on real agent traffic. + from verification import worth_local_check + assert worth_local_check("unverifiable", 5000, 600, has_tool_calls=True) is False + assert worth_local_check("unverifiable", 5000, 600, has_tool_calls=False) is True diff --git a/verification.py b/verification.py index 4bf4dad..c0723c7 100644 --- a/verification.py +++ b/verification.py @@ -155,20 +155,40 @@ def check_block(block: Block) -> Check: def verify_response( - text: str, finish_reason: Optional[str] = None + text: str, + finish_reason: Optional[str] = None, + has_tool_calls: bool = False, ) -> Verification: """Structurally verify a completion. - ``finish_reason == 'length'`` is decisive on its own: the model ran out of - budget mid-answer, so whatever came back is a fragment even if it happens - to parse. That is checked first because a truncated response that parses - is the most dangerous case — it looks fine. + ``has_tool_calls`` short-circuits everything below it: an agent turn that + calls a tool is not a prose answer and cannot be judged as one. It outranks + even truncation, because a turn routinely stops mid-sentence at the call + boundary and that is correct behaviour, not a budget overrun. + + ``finish_reason == 'length'`` is otherwise decisive on its own: the model + ran out of budget mid-answer, so whatever came back is a fragment even if + it happens to parse. That is checked before the content rules because a + truncated response that parses is the most dangerous case — it looks fine. A response with nothing checkable returns ``unverifiable``, which is NOT a failure. Most prose answers land here, and treating "we could not check this" as "this is wrong" would penalize models for the checker's limits — the same mistake as scoring a judge malfunction against a model. """ + # A turn that calls a tool is not a prose answer and must not be judged as + # one. Its text is routinely empty, or a half-sentence before the call, and + # both are correct behaviour. + # + # This was not theoretical. On a real agent session, 29 of 63 completions + # were recorded as "malformed: empty response" purely for making tool + # calls, and folding those in would have penalized the models that shipped + # a working feature with 349 passing tests, clean mypy and clean ruff. + if has_tool_calls: + return Verification( + "unverifiable", detail="tool-call turn; no prose answer to check" + ) + if finish_reason == "length": return Verification("truncated", detail="finish_reason=length") @@ -297,7 +317,10 @@ def interpret_local_verdict(parsed: Optional[dict]) -> Optional[Check]: def worth_local_check( - verdict: Verdict, completion_tokens: Optional[int], min_tokens: int + verdict: Verdict, + completion_tokens: Optional[int], + min_tokens: int, + has_tool_calls: bool = False, ) -> bool: """Whether a local LLM check earns its cost on this response. @@ -313,4 +336,10 @@ def worth_local_check( """ if verdict != "unverifiable": return False + if has_tool_calls: + # Same reason the structural check declines: an agent turn ending in a + # tool call reads as "cuts off mid-sentence" to a checker expecting a + # finished answer. Observed producing 8 false failures in 9 checks on + # real agent traffic. + return False return bool(completion_tokens and completion_tokens >= min_tokens) -- 2.49.1 From 69ac8f7745aa99f20d2771bd436f71e90584de9f Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Fri, 21 Aug 2026 19:03:17 -0400 Subject: [PATCH 20/20] fix: price is a market signal, not a capability measurement deepseek-v4-flash was losing every routing decision to glm-5.2-fast on real agent traffic, and it turned out to be excluded twice over by the same substitution made in two different places. It has a 1M advertised window, scores 1.00 on all three coding categories, and lists at $0.14/$0.28 per 1M against glm's $1.45/$4.50. The cost axis was measuring the wrong workload. `cost` came from the median billed USD over seed_energy.py's reference sweep, which sends a 400-token prompt with a 400-token completion. Real traffic through this router is a 150,000-token prompt with a ~400-token completion and 84% cache hits, and the ranking does not survive the change of shape: reference sweep (400/400) glm-5.2-fast 3.2x cheaper realistic (70k prompt) deepseek-v4-flash 5.0x cheaper Measured live, three samples each, not inferred. The attribution ratio is what moves: glm sits at 0.006 on a toy prompt and 0.50 on a 70k one, because it batches beautifully at small sizes and badly at real ones, while deepseek barely shifts (0.21 -> 0.25). A fixed-shape benchmark cannot rank models for a workload of another shape, and no amount of re-sweeping fixes that -- it measures one wrong thing more precisely. routing.estimated_cost now prices each request from catalog token prices scaled to that request's actual shape, via objective.assumed_cache_rate (0.84, measured from real traffic) and objective.assumed_completion_tokens. List price is not what gets billed -- NeuralWatt charges per kWh -- but billing is capped at 3x list, so it tracks the real ordering and bounds it, and on the one case checked live it agrees with the measurement in direction and magnitude (7.8x predicted vs 5.0x measured). It is also free, needs no sweep, and refreshes whenever the poller runs. A row the catalog has no price for keeps whatever measured cost it arrived with; a missing list price is not free. Three signals said deepseek -- catalog token price, NeuralWatt's own published per-request energy, and a live 70k measurement. Only the 400-token benchmark disagreed, and it was the one being scored on. Tiering made the identical mistake independently. Tier is a capability FLOOR: routing.py drops any row with tier < required_tier. Resolving tier on completion price alone put deepseek in tier 1 for no reason but being cheap, which excluded it OUTRIGHT from every tier-2 request -- so the cost fix alone would have changed nothing. tiering.resolve_tier now gates tier 1 on tier1_context_max (512000) as well as cost: tier 1 means small AND cheap, not merely cheap. The gate reads the ADVERTISED context_window, whose catalog values are the clean market classes (131056 / 199984 / 262128 / 1048560), rather than effective_context_window, which varies within a class. 512000 sits in the empty band between the 256K and 1M classes with a 2x margin either side, so it is not fitted to any one model. It only ever demotes -- a huge window never promotes an expensive model into tier 1 -- and a missing window does not block tier 1, since absent evidence should not decide anything. Distribution 4/6/9 -> 1/9/9; only the three deepseek rows moved. No model_tiers override was added, deliberately: the point is that the heuristic now gets this right, and pinning it in config would mask whether it does. deepseek now wins coding_general at every context size (16.5x cheaper than kimi-k3 at 200k) and is still correctly absent from tool_use_agentic, where its measured 0.33 drops it out of the quality band. That is the eval data earning it the slot rather than a thumb on the scale. Verified live against the running service. Tests 249 -> 256. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ --- CLAUDE.md | 85 +++++++++++++++++++++++++++++++++++++------ config.py | 10 +++++ config.yaml | 30 +++++++++++++++ dispatcher.py | 4 ++ routing.py | 58 +++++++++++++++++++++++++++-- tests/test_tiering.py | 72 +++++++++++++++++++++++++++++++++++- tier.py | 6 ++- tiering.py | 52 ++++++++++++++++++++++---- 8 files changed, 292 insertions(+), 25 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e2e3ccc..97303cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,7 +28,7 @@ can be added later without a migration. classification - FastAPI for the dispatcher service -## Billing is per-kWh, not per-token — RESOLVED, scoring now uses measured cost +## Billing is per-kWh, not per-token — and neither is what scoring uses **Measured against the live API, 2026-08-11.** Neuralwatt bills a flat **$8.00 per kWh** and the catalog's `input_per_million` / @@ -80,11 +80,46 @@ model and only the 6th cleanest; `kimi-k3-flex` draws 3.7x *less* energy than `kimi-k2.7-code` while emitting 3.6x *more* carbon. Weighting them separately is load-bearing, and `tests/test_routing.py` pins it. -**Resolved.** `cost` and `eco` come from the *median* measured USD and gCO2eq -over the reference sweep, so no billing formula lives in the code at all — -the rule above is documentation for why list price isn't used, not logic. -`flex_cost_multiplier` is gone: a flex row's measured cost already is its -flex cost. +**Superseded — cost no longer comes from the sweep at all.** `cost` was the +*median* measured USD over the reference sweep. That was measured to be WRONG +for real traffic, because the reference workload is the wrong shape. + +The sweep sends a 400-token prompt with a 400-token completion. Real agent +traffic is a 150,000-token prompt with a ~400-token completion and 84% cache +hits. The attribution ratio moves with prompt size, so the ranking inverts: + +| workload | winner | +|---|---| +| reference sweep (400/400) | `glm-5.2-fast`, 3.2x cheaper | +| realistic (70k prompt, short answer) | **`deepseek-v4-flash`, 5.0x cheaper** | + +Same two models, opposite answer. `glm-5.2-fast` sits at attribution 0.006 on +a toy prompt and 0.50 on a 70k one — it batches beautifully on small prompts +and badly on real ones. `deepseek-v4-flash` barely moves (0.21 -> 0.25). + +So `routing.estimated_cost` prices each request from **catalog token prices, +scaled to that request's actual shape** (prompt size, assumed completion +length, `assumed_cache_rate`). List price is not what gets billed, but billing +is capped at 3x list, so it tracks the real ordering and bounds it — and on +the one case that was checked live it agrees with the measurement in direction +and magnitude (7.8x predicted vs 5.0x measured). It is also free, needs no +sweep, and refreshes whenever the poller runs. + +Three signals said `deepseek-v4-flash` — catalog token price (7.8x cheaper), +NeuralWatt's own published per-request energy (~10x lower), and a live 70k +measurement (5.0x cheaper). Only the 400-token benchmark disagreed. Trust the +workload you actually run. + +`eco` still comes from the sweep's median gCO2eq, and is still not an +objective. `flex_cost_multiplier` is gone: a flex row's measured cost already +is its flex cost. + +**Open, and worth knowing:** NeuralWatt's model cards publish *gross* energy +(~1.99e-04 kWh for deepseek, ~1.91e-03 for GLM), while the billed figure is +gross x attribution. GLM burns roughly 7x more actual electricity per request +and charges ~5x less, because far more tenants share its GPUs. Anything built +on `eco` inherits that inversion — the attributed carbon figure answers "what +is my share", not "what was burned". ## Energy attribution: signal that looks like noise @@ -217,7 +252,32 @@ falling back to `capabilities.reasoning`), **not** `supports_reasoning`. param" — it is true for 17 of 19 rows, and tiering on it put 17 models in tier 3 and left tier 1 empty. A `-fast` row does not inherit its sibling's tier 3. Cost is checked before the reasoning rule so $0.28/1M models can -reach tier 1. Current distribution: **4 / 6 / 9**. +reach tier 1. + +**Cheapness is not a capability ceiling** (`tier1_context_max`, default +512000). Tier is a *floor* — `routing.py` drops any row with +`tier < required_tier` — so tier 1 means "simple work only", not "cheap". +Deciding that on price alone put `deepseek-v4-flash` in tier 1 for no reason +but its $0.28/1M completion price, which excluded it **outright** from every +tier-2 request. It has a 1M advertised window and scores 1.00 on all three +coding categories. That was the same substitution the cost axis already had +to unlearn: price is a market signal, not a capability measurement. + +Tier 1 now requires the model to be small **and** cheap. The gate reads the +**advertised** `context_window`, whose catalog values are the clean market +classes — 131056 / 199984 / 262128 / 1048560 — rather than +`effective_context_window`, which varies within a class. 512000 sits in the +empty band between the 256K and 1M classes with a 2x margin either side, so it +is not fitted to any one model. The gate only ever demotes; a huge window +never promotes an expensive model into tier 1, and a *missing* window does not +block it, since absent evidence should not decide anything. + +Distribution moved **4 / 6 / 9 -> 1 / 9 / 9** — only the three deepseek rows +changed. With cost priced per-request, `deepseek-v4-flash` now wins +`coding_general` at every context size (16.5x cheaper than `kimi-k3` at 200k) +and is still correctly absent from `tool_use_agentic`, where its measured 0.33 +drops it out of the quality band. That is the eval data earning it the slot +rather than a thumb on the scale — no `model_tiers` override was needed. ## Proficiency: category now changes routing @@ -466,11 +526,12 @@ that request was two orders of magnitude low. relies entirely on self-eval accumulating. `python leaderboard.py --check` lists what is missing. -2. **Sampling depth for three models.** 7 samples/model gives split-half - agreement within 1.4x for 10 of 13, but `kimi-k2.7-code-fast` (29x), - `kimi-k3` (14x) and `glm-5.2-flex` (2.2x) are still unsettled. Re-run - `seed_energy.py --models ... --samples 21` for those before trusting where - they land. +2. **Sampling depth for three models — now `eco`-only.** 7 samples/model gives + split-half agreement within 1.4x for 10 of 13, but `kimi-k2.7-code-fast` + (29x), `kimi-k3` (14x) and `glm-5.2-flex` (2.2x) are still unsettled. This + no longer touches cost, which is priced per-request from the catalog, so it + only affects `eco` — which is not an objective. Low priority unless eco + comes back. 3. **Retry does not reach streaming.** The iteration budget (`iteration.py`) retries after a failed check, but only on the non-streaming path — once diff --git a/config.py b/config.py index 090f018..fd7cd90 100644 --- a/config.py +++ b/config.py @@ -25,6 +25,8 @@ class Objective(BaseModel): """ quality_tolerance: float = 0.10 + assumed_cache_rate: float = 0.84 + assumed_completion_tokens: int = 500 max_energy_per_request: Optional[float] = None plan_kwh_per_period: Optional[float] = None @@ -77,6 +79,7 @@ class ProficiencyConfig(BaseModel): class TieringConfig(BaseModel): cheap_completion_max: float + tier1_context_max: float = float("inf") model_tiers: dict[str, int] @field_validator("cheap_completion_max") @@ -86,6 +89,13 @@ class TieringConfig(BaseModel): raise ValueError("tiering.cheap_completion_max must be > 0") return v + @field_validator("tier1_context_max") + @classmethod + def context_max_must_be_positive(cls, v: float) -> float: + if v <= 0: + raise ValueError("tiering.tier1_context_max must be > 0") + return v + @field_validator("model_tiers") @classmethod def overrides_in_range(cls, v: dict[str, int]) -> dict[str, int]: diff --git a/config.yaml b/config.yaml index 4d8ec11..9d0146d 100644 --- a/config.yaml +++ b/config.yaml @@ -20,6 +20,23 @@ objective: # Narrow it as samples accumulate. quality_tolerance: 0.10 + # Cost is priced per-request from catalog prices, NOT from a benchmark + # sweep. A fixed 400-token reference task ranked glm-5.2-fast 3.2x cheaper + # than deepseek-v4-flash; on a realistic 70k-token prompt deepseek is 5.0x + # cheaper. Attribution inverts with prompt size, so a fixed-shape benchmark + # cannot rank models for a workload of another shape. Catalog prices scaled + # to the actual request agree with the live measurement, cost nothing, and + # need no sweep. + + # Share of prompt tokens served from the provider's prefix cache. Measured + # from real traffic: 1.9M of 2.2M prompt tokens, 84%. Agent clients resend + # the whole conversation each turn, so most of it is a cache hit. + assumed_cache_rate: 0.84 + + # Completion length assumed when pricing a request. Real sessions here median + # around 200-400 completion tokens against enormous prompts. + assumed_completion_tokens: 500 + # Per-request ceiling on measured ENERGY, in kWh. null disables it. # # Denominated in kWh rather than dollars because the plan is a subscription @@ -55,6 +72,19 @@ tiering: # model_tiers overrides the heuristic per model_id and applies to ALL # providers (limitation vs a (model_id, provider) key). cheap_completion_max: 1.00 + + # Advertised context_window at or above which a model is NOT eligible for + # tier 1, whatever it costs. Tier is a capability FLOOR (routing drops any + # row with tier < required_tier), so tier 1 means "simple work only" — and + # deciding that on price alone excluded deepseek-v4-flash from every tier-2 + # request purely for being $0.28/1M, despite a 1M window and 1.00 on all + # three coding categories. + # + # 512000 sits in the empty band between the catalog's 256K class (262128) + # and its 1M class (1048560) — a 2x margin either side, so it is not fitted + # to any one model. gemma-4-31b (256K) stays tier 1; the 1M rows do not. + tier1_context_max: 512000 + model_tiers: {} proficiency: diff --git a/dispatcher.py b/dispatcher.py index 9fc0468..5a85152 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -487,6 +487,10 @@ def route(req: TaskRequest) -> RouteResponse: eligible, quality_tolerance=cfg.objective.quality_tolerance, max_energy_per_request=cfg.objective.max_energy_per_request, + # Priced for the request in hand, not for a benchmark. + prompt_tokens=classification.required_context_tokens, + completion_tokens=cfg.objective.assumed_completion_tokens, + cache_rate=cfg.objective.assumed_cache_rate, ) return RouteResponse( diff --git a/routing.py b/routing.py index 97f0733..10b8396 100644 --- a/routing.py +++ b/routing.py @@ -97,6 +97,45 @@ def select_candidates( ] +def estimated_cost( + row: dict, + prompt_tokens: int, + completion_tokens: int, + cache_rate: float, +) -> float | None: + """What this request should cost on this model, from catalog prices. + + Replaced a benchmark. The old cost signal came from a fixed 400-token + reference sweep, and it was measured to be WRONG for real traffic: on a + 400-token prompt glm-5.2-fast looked 3.2x cheaper than deepseek-v4-flash, + but on a realistic 70k-token prompt deepseek is 5.0x cheaper. Attribution + inverts with prompt size — glm batches beautifully on toy prompts and badly + on real ones — so a fixed-shape benchmark cannot rank models for a workload + of a different shape. + + Catalog prices, scaled to THIS request's shape, get the same answer as the + live measurement (7.8x vs 5.0x, same direction). They are also free, need + no sweep, and update whenever the poller runs. + + They are not what gets billed — NeuralWatt charges per kWh — but billing is + capped at a multiple of token price, so this tracks the real ordering and + bounds it. Cheap and directionally right beats precise about the wrong + workload. + """ + prompt_price = row.get("cost_per_1m_prompt") + completion_price = row.get("cost_per_1m_completion") + if prompt_price is None or completion_price is None: + return None + # Agent traffic resends the conversation every turn, so most prompt tokens + # hit the provider's prefix cache and are billed at the cached rate. + cached_price = row.get("cost_per_1m_prompt_cached") + if cached_price is None: + cached_price = prompt_price + fresh = prompt_tokens * (1.0 - cache_rate) * prompt_price + cached = prompt_tokens * cache_rate * cached_price + return (fresh + cached + completion_tokens * completion_price) / 1_000_000 + + def within_budget(row: dict, max_energy_kwh: float | None) -> bool: """Whether a candidate's measured energy is inside the per-request ceiling. @@ -120,6 +159,9 @@ def rank_candidates( *, quality_tolerance: float = 0.1, max_energy_per_request: float | None = None, + prompt_tokens: int = 0, + completion_tokens: int = 500, + cache_rate: float = 0.84, ) -> list[dict]: """Order candidates: best quality first, cheapest among equals. @@ -154,15 +196,23 @@ def rank_candidates( """ affordable = [r for r in rows if within_budget(r, max_energy_per_request)] + # Priced for THIS request rather than for a benchmark, which is the whole + # point: the ordering depends on the workload's shape. A row the catalog + # has no price for keeps whatever measured cost it arrived with rather + # than losing the field — a missing list price is not free. + estimates = [] + for r in affordable: + est = estimated_cost(r, prompt_tokens, completion_tokens, cache_rate) + estimates.append(r.get("cost") if est is None else est) # cost_score is retained purely so callers can still see the spread; it # does not enter the ordering. - cost_scores = cost_score([r.get("cost") for r in affordable]) + cost_scores = cost_score(estimates) ranked = [] - for row, c_s in zip(affordable, cost_scores): + for row, c_s, est in zip(affordable, cost_scores, estimates): p_s = proficiency_score(row.get("proficiency")) - ranked.append({**row, "cost_score": c_s, "proficiency_score": p_s, - "composite": p_s}) + ranked.append({**row, "cost": est, "cost_score": c_s, + "proficiency_score": p_s, "composite": p_s}) if not ranked: return [] diff --git a/tests/test_tiering.py b/tests/test_tiering.py index 1ce9a26..9dee9b8 100644 --- a/tests/test_tiering.py +++ b/tests/test_tiering.py @@ -21,8 +21,13 @@ def _row( cost_per_1m_completion: float | None = 1.0, pricing_tbd: bool = False, supports_reasoning: bool = False, + context_window: int | None = None, ) -> dict: - """Build a minimal models-table row dict (ModelRow-shaped).""" + """Build a minimal models-table row dict (ModelRow-shaped). + + ``context_window`` defaults to None — absent, not small — so the tier-1 + context gate is inert unless a test opts in. + """ return { "model_id": model_id, "reasoning_default_enabled": reasoning_default_enabled, @@ -30,6 +35,7 @@ def _row( "cost_per_1m_completion": cost_per_1m_completion, "pricing_tbd": pricing_tbd, "supports_reasoning": supports_reasoning, + "context_window": context_window, } @@ -182,3 +188,67 @@ def test_cost_exactly_at_threshold_tiers_to_2(): tier = resolve_tier(row, cheap_completion_max=1.00, override_map={}) # Then: tier 1 requires strictly-below, so it is tier 2 assert tier == 2 + + +# --- cheapness is not a capability ceiling -------------------------------- + +def test_a_cheap_model_with_a_huge_window_is_not_tier_1(): + # deepseek-v4-flash: $0.28/1M completion, 1M advertised window, 1.00 on all + # three coding categories. Tiering on price alone capped it at tier 1, + # which — because tier is a FLOOR — excluded it outright from every tier-2 + # request. Being inexpensive is not evidence of being incapable. + row = _row( + model_id="deepseek-v4-flash", + reasoning_default_enabled=False, + cost_per_1m_completion=0.28, + context_window=1_048_560, + ) + assert resolve_tier(row, 1.00, {}, 512_000) == 2 + + +def test_a_cheap_model_with_a_small_window_stays_tier_1(): + # gemma-4-31b: same rule, opposite side. 256K is genuinely a small model. + row = _row( + model_id="gemma-4-31b", + reasoning_default_enabled=False, + cost_per_1m_completion=0.42, + context_window=262_128, + ) + assert resolve_tier(row, 1.00, {}, 512_000) == 1 + + +def test_the_context_gate_is_inclusive_at_the_threshold(): + row = _row(reasoning_default_enabled=False, cost_per_1m_completion=0.1, + context_window=512_000) + assert resolve_tier(row, 1.00, {}, 512_000) == 2 + row["context_window"] = 511_999 + assert resolve_tier(row, 1.00, {}, 512_000) == 1 + + +def test_a_missing_window_does_not_block_tier_1(): + # Absent capability evidence must not promote a model — a row with no + # advertised window falls through to the cost rule as before. + row = _row(reasoning_default_enabled=False, cost_per_1m_completion=0.1, + context_window=None) + assert resolve_tier(row, 1.00, {}, 512_000) == 1 + + +def test_the_gate_defaults_to_off_for_callers_that_do_not_pass_it(): + # Omitting the argument keeps the old cost-only behaviour rather than + # silently re-tiering a caller that has not opted in. + row = _row(reasoning_default_enabled=False, cost_per_1m_completion=0.28, + context_window=1_048_560) + assert resolve_tier(row, 1.00, {}) == 1 + + +def test_a_huge_window_does_not_rescue_an_expensive_model_into_tier_1(): + # The gate only ever demotes; it is not a second route into tier 1. + row = _row(reasoning_default_enabled=False, cost_per_1m_completion=15.0, + context_window=131_056) + assert resolve_tier(row, 1.00, {}, 512_000) == 2 + + +def test_override_still_beats_the_context_gate(): + row = _row(model_id="pinned", reasoning_default_enabled=False, + cost_per_1m_completion=0.28, context_window=1_048_560) + assert resolve_tier(row, 1.00, {"pinned": 1}, 512_000) == 1 diff --git a/tier.py b/tier.py index 8da1ed6..0f8876b 100644 --- a/tier.py +++ b/tier.py @@ -41,7 +41,8 @@ def apply_tiering(conn: sqlite3.Connection, config: RouterConfig) -> None: """ rows = conn.execute( "SELECT model_id, provider, supports_reasoning, reasoning_default_enabled, " - "reasoning_mode, cost_per_1m_completion, pricing_tbd FROM models" + "reasoning_mode, cost_per_1m_completion, pricing_tbd, context_window " + "FROM models" ).fetchall() if not any(row[3] for row in rows): @@ -55,6 +56,7 @@ def apply_tiering(conn: sqlite3.Connection, config: RouterConfig) -> None: reasoning_mode, cost, pricing_tbd, + context_window, ) in rows: tier = resolve_tier( { @@ -64,9 +66,11 @@ def apply_tiering(conn: sqlite3.Connection, config: RouterConfig) -> None: "reasoning_mode": reasoning_mode, "cost_per_1m_completion": cost, "pricing_tbd": bool(pricing_tbd), + "context_window": context_window, }, config.tiering.cheap_completion_max, config.tiering.model_tiers, + config.tiering.tier1_context_max, ) conn.execute( "UPDATE models SET tier = ? WHERE model_id = ? AND provider = ?", diff --git a/tiering.py b/tiering.py index fbb9aaa..6d708a6 100644 --- a/tiering.py +++ b/tiering.py @@ -19,25 +19,52 @@ tier 1 empty. The discriminating signals are ``reasoning_default_enabled`` ``reasoning_mode`` (whether this is a ``-fast`` row with thinking disabled or capped). +Tier is a capability FLOOR: ``routing.py`` excludes any row whose +``tier < required_tier``, so tier 1 does not mean "cheap", it means "only +suitable for simple work". That distinction is what rule 3 below turns on. + Resolution precedence (EXACT order): 1. If ``override_map`` has this model's ``model_id`` -> that value wins. 2. Reasoning is "effectively on" when ``reasoning_default_enabled`` is True AND ``reasoning_mode`` is not ``'reduced'``. A ``-fast`` row is the same weights served without chain-of-thought, so it does not earn tier 3 on its sibling's behalf. - 3. Reasoning effectively OFF and ``cost_per_1m_completion`` is not None - and not ``pricing_tbd`` and < ``cheap_completion_max`` -> 1. + 3. Reasoning effectively OFF, ``cost_per_1m_completion`` is not None and + not ``pricing_tbd`` and < ``cheap_completion_max``, AND + ``context_window`` is below ``tier1_context_max`` -> 1. 4. Reasoning effectively ON -> 3. - 5. Otherwise (missing/NULL cost, ``pricing_tbd``, or cost >= threshold) - -> 2. + 5. Otherwise (missing/NULL cost, ``pricing_tbd``, cost >= threshold, or a + large context window) -> 2. Rules 3 and 4 are mutually exclusive, so their relative order is immaterial; cost is stated first because the previous version's reasoning-before-cost ordering is what pushed $0.28/1M models into tier 3. +Why context window gates tier 1: price is a market signal, not a capability +measurement, and this project already learned that once in the cost domain — +list price ranks these models backwards. Tiering on price alone made the same +substitution and it capped ``deepseek-v4-flash`` at tier 1 on nothing but its +$0.28/1M completion price, which excluded it OUTRIGHT from every tier-2 +request. It has a 1M advertised window and scores 1.00 on all three coding +categories in the eval set; "simple tasks only" is not a defensible reading of +that row. A model that can hold a million tokens of context is not a small +model however little it charges, so tier 1 now requires the model to be small +AND cheap rather than merely cheap. + +``context_window`` (advertised) is used rather than +``effective_context_window`` because the advertised figure is the clean market +class — the catalog's values are 131056, 199984, 262128 and 1048560, i.e. +128K/200K/256K/1M — while the effective figures are provider-derived and vary +within a class (a 256K row reports 180212 or 192500 depending on family). +``tier1_context_max`` therefore sits in the wide empty band between the 256K +and 1M classes rather than being fitted to any one model. + NULL-cost semantics: tier 1 REQUIRES a non-NULL completion cost strictly -below the threshold. A strong-cheap non-reasoning model being mis-tiered to -1 is ACCEPTED — the override map is the escape hatch. +below the threshold. A missing/None ``context_window`` does NOT block tier 1 — +absent capability evidence should not promote a model. + +The override map remains the escape hatch for a row the heuristic still gets +wrong. """ from __future__ import annotations @@ -60,15 +87,21 @@ def resolve_tier( model_row: dict, cheap_completion_max: float, override_map: dict[str, int], + tier1_context_max: float = float("inf"), ) -> int: """Resolve a model row to a tier in {1, 2, 3}. ``model_row`` is a row-like dict with at least these keys (from the ``models`` table / ``ModelRow``): ``model_id`` (str), ``reasoning_default_enabled`` (bool), ``reasoning_mode`` (str), - ``cost_per_1m_completion`` (float or None), ``pricing_tbd`` (bool). + ``cost_per_1m_completion`` (float or None), ``pricing_tbd`` (bool) and + ``context_window`` (int or None, the ADVERTISED window). ``supports_reasoning`` is consulted only as a fallback. + ``tier1_context_max`` defaults to infinity so an omitted argument keeps + the pre-existing cost-only behaviour rather than silently re-tiering a + caller that has not opted in. + ``override_map`` is keyed by ``model_id`` only and therefore applies to ALL providers serving that model (documented limitation vs the (model_id, provider) primary key). @@ -80,10 +113,15 @@ def resolve_tier( if not reasoning_effectively_on(model_row): cost = model_row["cost_per_1m_completion"] + # A missing window is not evidence of a small model, so it does not + # block tier 1 — only a window we can see and that is large does. + window = model_row.get("context_window") + large_context = window is not None and window >= tier1_context_max if ( not model_row["pricing_tbd"] and cost is not None and cost < cheap_completion_max + and not large_context ): return 1 return 2 -- 2.49.1