neuralwatt-router-service #1
Merged
alee
merged 20 commits from 2026-08-21 23:10:28 +00:00
neuralwatt-router-service into main
20 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
adlee-was-taken
|
69ac8f7745 |
fix: price is a market signal, not a capability measurement
deepseek-v4-flash was losing every routing decision to glm-5.2-fast on real
agent traffic, and it turned out to be excluded twice over by the same
substitution made in two different places. It has a 1M advertised window,
scores 1.00 on all three coding categories, and lists at $0.14/$0.28 per 1M
against glm's $1.45/$4.50.
The cost axis was measuring the wrong workload. `cost` came from the median
billed USD over seed_energy.py's reference sweep, which sends a 400-token
prompt with a 400-token completion. Real traffic through this router is a
150,000-token prompt with a ~400-token completion and 84% cache hits, and the
ranking does not survive the change of shape:
reference sweep (400/400) glm-5.2-fast 3.2x cheaper
realistic (70k prompt) deepseek-v4-flash 5.0x cheaper
Measured live, three samples each, not inferred. The attribution ratio is what
moves: glm sits at 0.006 on a toy prompt and 0.50 on a 70k one, because it
batches beautifully at small sizes and badly at real ones, while deepseek
barely shifts (0.21 -> 0.25). A fixed-shape benchmark cannot rank models for a
workload of another shape, and no amount of re-sweeping fixes that -- it
measures one wrong thing more precisely.
routing.estimated_cost now prices each request from catalog token prices
scaled to that request's actual shape, via objective.assumed_cache_rate (0.84,
measured from real traffic) and objective.assumed_completion_tokens. List
price is not what gets billed -- NeuralWatt charges per kWh -- but billing is
capped at 3x list, so it tracks the real ordering and bounds it, and on the
one case checked live it agrees with the measurement in direction and
magnitude (7.8x predicted vs 5.0x measured). It is also free, needs no sweep,
and refreshes whenever the poller runs. A row the catalog has no price for
keeps whatever measured cost it arrived with; a missing list price is not
free.
Three signals said deepseek -- catalog token price, NeuralWatt's own published
per-request energy, and a live 70k measurement. Only the 400-token benchmark
disagreed, and it was the one being scored on.
Tiering made the identical mistake independently. Tier is a capability FLOOR:
routing.py drops any row with tier < required_tier. Resolving tier on
completion price alone put deepseek in tier 1 for no reason but being cheap,
which excluded it OUTRIGHT from every tier-2 request -- so the cost fix alone
would have changed nothing. tiering.resolve_tier now gates tier 1 on
tier1_context_max (512000) as well as cost: tier 1 means small AND cheap, not
merely cheap.
The gate reads the ADVERTISED context_window, whose catalog values are the
clean market classes (131056 / 199984 / 262128 / 1048560), rather than
effective_context_window, which varies within a class. 512000 sits in the
empty band between the 256K and 1M classes with a 2x margin either side, so it
is not fitted to any one model. It only ever demotes -- a huge window never
promotes an expensive model into tier 1 -- and a missing window does not block
tier 1, since absent evidence should not decide anything.
Distribution 4/6/9 -> 1/9/9; only the three deepseek rows moved. No
model_tiers override was added, deliberately: the point is that the heuristic
now gets this right, and pinning it in config would mask whether it does.
deepseek now wins coding_general at every context size (16.5x cheaper than
kimi-k3 at 200k) and is still correctly absent from tool_use_agentic, where
its measured 0.33 drops it out of the quality band. That is the eval data
earning it the slot rather than a thumb on the scale. Verified live against
the running service.
Tests 249 -> 256.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ
|
||
|
adlee-was-taken
|
f0d9bfb81c |
fix: an agent turn that calls a tool is not a failed answer
Both verification paths had mirror halves of the same blind spot, and real traffic is what found it. On the first genuine agent session through this router -- 63 completions that shipped a working feature with 349 passing tests, clean mypy and clean ruff -- the structural checker recorded "malformed: empty response" 29 times and the local LLM checker called "cuts off mid-sentence" on 8 of the 9 answers it graded. Both were describing the same thing from opposite sides: a turn that ends by calling a tool. Its text content is empty, or a half-sentence before the call, and both are correct behaviour rather than a defect. verify_response and worth_local_check now take has_tool_calls, supplied on the non-streaming path from message.tool_calls and accumulated on the streaming path from delta.tool_calls. In verify_response the check outranks even finish_reason == 'length', because stopping mid-sentence at a call boundary is a call boundary, not a budget overrun. worth_local_check declines outright, which also stops paying ~6s of local inference to mis-grade a tool call. Had feedback.py run against those rows it would have applied ~12 false failures to the two models that had just done the work. That is the FOURTH harness bug in this project that would have scored the rig rather than the model, and the first one caught by real traffic instead of a synthetic test. The pre-fix rows are kept with model_attributable = 0 so the record survives without steering routing. client_capped was over-applied in the same area. It marked EVERY verdict non-attributable whenever the client set max_tokens, and opencode always sets it, so genuine failures were invisible to feedback for the entire main workflow. A client's token cap explains a truncated verdict and nothing else; a model emitting unparseable code owes nothing to the client's budget. It is now scoped to exactly that verdict. Also recorded: outcome attribution resolves the session directory by path histogram, and on this session that picked .venv/.../site-packages/c2pa 24 times over ~/Sources/fieldwitness 22, because reading a dependency's source outweighed editing the project. It degrades safely -- 31 reports accepted, 3 refused as ambiguous rather than misattributed -- but the heuristic needs to weight writes over reads. Tests 243 -> 249. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ |
||
|
adlee-was-taken
|
6e729ad670 |
feat: parallel-safe outcome attribution, and an opencode plugin to feed it
Closes the ground-truth loop. deploy/opencode-plugin/router-outcome.js hooks tool.execute.after, watches for test and build commands, and reports pass/fail to /outcome. opencode already runs your tests; this is what makes the result reach routing. Command detection is deliberately narrow -- pytest, npm test, cargo, go, ruff, mypy, tsc and friends. A failing `ls` says nothing about model quality, and a false signal is worse than none because it trains the router on noise. Verdict comes from exit status plus text signatures for tools that exit 0 while reporting failures, with "0 failed" and "no errors" guarded against. The router being unreachable never breaks a session. Attribution is the hard part, and two assumptions failed under test. The first fingerprint design keyed on the system prompt. One real opencode run produced TWO distinct keys, because it runs several agents with different prompts -- so that fingerprint identifies AGENTS, not sessions, and would have refused every single run forever. A permanent false positive dressed as safety. The second assumption was that opencode states its project root up front. Capturing a real request showed it does not. Directory is now derived from the file paths an agent touches across the whole conversation, counting every ancestor so the shared project root wins over any one subdirectory, and stripping trailing filenames so a file is never mistaken for a directory. A single mention is not enough; corroboration is required. When a report cannot be matched by directory and more than one conversation was active in the window, /outcome answers 409 and records nothing. Refusing beats guessing: a misattributed failure penalizes a model for work it never did, and this project has already recorded false failures twice from harness bugs that took measurement to catch. The window is 120 seconds, not 30 minutes. At 30 it swept in traffic from earlier in the same work session and refused a legitimate report -- observed directly, not theorised. Tests 232 -> 243. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
e28443cb36 |
feat: POST /outcome — the only ground truth the router can get
Everything else this router records is a proxy. Structural checks know whether code parses. The local checker guesses whether prose looks right. Neither knows whether the answer did the job. The client does: it ran the tests, or used the answer, or watched it fail. Clients report against the provider's completion id, which they already receive in the response body and on every stream chunk. energy_observations and verifications now store that id so a report has something to join on. Two properties make this the highest-value signal available. It is the only quality signal that survives streaming. A retry cannot reach a streamed response -- the bytes are already gone -- but a report arrives afterwards and works identically either way. Every agent client streams, so without this the main workflow had verification and feedback but no route from outcome back into routing. And its successes count. feedback.py folds client outcomes in BOTH directions, unlike checks where only failures do. That asymmetry is deliberate: a parser reporting 'ok' means the code parsed, which is weak evidence that would inflate every score toward the ceiling, while a client reporting 'succeeded' means the work worked. An unknown request_id returns 404 rather than being quietly accepted. A client whose reports go nowhere should find out rather than train nothing. Verified end to end on both paths, including a streamed completion reported as failed after the fact. Tests 227 -> 232. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
e1fd65f3e4 |
feat: tier is an iteration budget, spent on evidence rather than on a hunch
A tier used to mean only a capability floor. It now also buys corrective attempts after a verification failure: tier 1 gets one shot, tier 2 one retry, tier 3 two. Interactive requests are capped below their tier regardless, because every retry doubles time-to-answer and in interactive use latency IS a quality loss. escalation.preemptive_on_low_confidence now defaults to FALSE. Bumping the tier because the classifier was unsure of its own call pays frontier prices before anything has gone wrong. Spending after a check has actually failed is better on both mandates: the cheap attempt usually succeeds and costs nothing extra, and when it fails there is evidence rather than a hunch. Retries are matched to the failure, because the causes differ. Truncation raises the token budget on the same model -- a different one would run out too. Malformed output escalates to the next-ranked candidate, since more tokens will not make unparseable output parse. 'ok' and 'unverifiable' buy nothing; retrying unverifiable would burn quota across the majority of prose traffic for no signal at all. Testing it live exposed that the truncation branch was UNREACHABLE as first written. Through /v1 the token cap is either the client's, which is not ours to override, or absent -- and when absent the model hit its own ceiling, so doubling changes nothing. It now escalates to a candidate with a larger output ceiling, which is the actionable move, and declines when no such candidate exists rather than wasting an attempt against a fixed kWh quota. Known limit, recorded in CLAUDE.md: retry does not reach the streaming path. Once bytes have gone to the client there is nothing to take back, and buffering to allow correction would cost streaming itself. opencode streams, so the main workflow gets verification and feedback but not correction. Tests 211 -> 227. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
42b71b9bbc |
feat: quality is the objective, energy is the constraint, eco is out
Replaces the three-way weighted blend with a rule that states the dual mandate honestly: maximize quality, subject to a per-request energy ceiling, tie-broken by cost. Eco is dropped from the objective (still logged per request). It was 20% of every decision optimizing a goal that is handled outside this router. Cost stops being a weight. Under the blend, 60% of each decision adjudicated fractions of a cent -- all real traffic to date totals $0.07 -- and min-max normalization made "expensive" relative to whoever else happened to be a candidate, so a model could lose for being 2x a very cheap one when both round to nothing. Measurement retired the blend outright: turning the cost weight from 0.4 to ZERO changed the winner in only 2 of 6 categories, so it was never steering on quality. The ceiling is denominated in kWh, not dollars, because the plan is a subscription with a 6.25 kWh quota. Dollars accrue and can be reconciled later; a quota is a wall you hit mid-task. /health now reports burn against the allowance and warns past 80%, with the caveat that it counts only what the router saw. quality_tolerance (0.10) is measurement noise rather than preference: proficiency rests on 2-3 samples per category, so smaller gaps are sampling variation and paying more for them buys noise. Narrow it as samples accumulate. Also corrects a figure the docs asserted as a constant. Grid intensity is not 37 gCO2/kWh: observations show FI at 49 and 50 at different times, FR at 54, US-MIDA-PJM at 442, and the provider's own 24h blended figure was 145.4. Tests 207 -> 211. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
cdb186348a |
docs: correct a stale limitation and document the single-model routing outcome
Two gaps in the README, found by checking it against the running system. The "no escalation feedback" limitation undersold what shipped. Flagging a bad response DOES exist now -- verification.py detects failures and feedback.py folds them into proficiency automatically. What is missing is only the action: nothing retries or escalates. That is deliberate, since whether auto-escalation pays should come from the observed failure rate now being collected. More usefully, the README said nothing about the fact that all 27 routing decisions currently return qwen3.6-35b. A reader would reasonably expect traffic to spread across models and go hunting for a misconfiguration. It is Pareto-dominant on this catalog: cheapest AND cleanest, within 0.15 of the best proficiency. On summarization it costs 7x less and emits 65x less carbon than kimi-k3, which beats it 1.00 to 0.85 -- no defensible weighting picks kimi-k3 there. Also records what is easy to miss: proficiency still reorders the runners-up by category, and the field widens once the leader stops being eligible -- past ~94,196 tokens qwen3.6-35b is filtered out and glm-5.2-fast takes over, past ~790K nothing qualifies and the request 422s. On a large codebase the model will change mid-session, and that is the context filter working. Every figure verified against the live database before writing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
dac40c8fb2 |
docs: refresh module list and test counts (207 across 11 files)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
6c6a8a0edc |
feat: fold observed verification failures back into proficiency
The eval harness measures models on a fixed 23-task benchmark. feedback.py measures them on real traffic, which is more predictive of routing quality and accumulates for free as you work. Proven end to end: a truncated response dropped qwen3.6-35b's coding_general from 1.00 to 0.75 while leaving coding_refactor untouched. Only FAILURES are folded in, deliberately. A structural 'ok' means the code parsed, not that it was correct -- a model emitting syntactically valid nonsense would score 1.0. Recording passes would flood self_eval_score with 1.0 samples and wash out the benchmark's discrimination; the coding categories already sit at 1.00 for every model and this would spread that flatness everywhere. A model that never fails keeps its benchmark score untouched; one that does is penalized in proportion to how often. Testing it exposed a flaw in the design. The failure I forced was caused by MY OWN max_tokens=40, not by the model -- and without a fix, any agent setting a tight cap would systematically drag down whatever model it routed to. Now a truncation under a client-supplied max_tokens is recorded (the response really was unusable) but marked model_attributable=0 and excluded from feedback. The score that penalty wrongly cost has been restored. applied_at makes application idempotent, so re-running cannot punish a model repeatedly for the same bad response. Tests 193 -> 207. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
cf9f5c6be9 |
feat: local LLM verification for responses nothing structural can check
Runs on the local model AFTER the response has gone back to the client, so its ~6s never lands on anyone's latency. It exists to learn which models fail on real work, not to gate answers. Gated on answer size, which is economics rather than taste: a local check costs ~15% of a median 193-token answer, so it would only pay if such answers failed more than ~15% of the time. At 1,500 completion tokens the break-even failure rate drops to ~1.9%. Below the threshold a check costs more than the risk it removes, so small answers are left alone. Three defects were found and fixed by running it, each of which would have recorded false failures against models: 1. Head-truncating the answer for the checker made a complete 6,637-character response end mid-sentence, and the checker duly reported "cut off mid-thought". The harness manufactured the defect it then detected. Now the MIDDLE is elided and the cut is labelled, so the ending being judged is the real ending. 2. The checker then flagged the elision marker itself as a defect, so the system prompt now explains that the marker is the harness, not the model. 3. With thinking enabled the local model never emitted a verdict at all: 2,048 tokens produced 7,880 characters of reasoning and EMPTY content, and raising the cap to 8,192 simply bought more reasoning. Verification does not need chain-of-thought. Switched to Ollama's native endpoint, the only one that exposes `think`, and with think=False it answers in 25 tokens. An empty response is now settled by an if-statement rather than a model: asked about one, the local checker returned ok=true with the reason "Answer is too short" -- self-contradictory. Never ask a model what code can decide. After those fixes the checker returned the same verdict on 3/3 repeats of four labelled cases (complete, truncated, refusal, wrong-question). A malfunctioning checker still records NO sample rather than a failure. Tests 179 -> 193. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
c360bde17f |
feat: verify every response structurally, without executing it
Free, exact checks on what a model just returned. Extracts fenced blocks and validates them by PARSING: ast.parse for Python, json.loads, yaml.safe_load. Plus finish_reason='length' and unterminated fences, which catch the failure mode that has bitten this project repeatedly -- a truncated answer looks complete to the client, which then acts on a fragment. The router deliberately does NOT execute model output, unlike eval_proficiency.py. There the prompts are ones this project authored, so what comes back is bounded. Here the code is whatever the user asked for and could do anything, so running it as a side effect of routing would be indefensible. Two tests assert non-execution, including one that would delete a file. 'unverifiable' is recorded and is NOT a failure. Most prose lands there, and counting "we could not check this" as "this was wrong" would penalize models for the checker's limits -- the same mistake as scoring a judge malfunction against a model, which this project made once already. Streaming accumulates deltas so a streamed answer gets the same check as a buffered one; otherwise streaming, which is how every agent client talks to the router, would be the unverified path. Non-streaming reports the verdict in an X-Router-Verification header rather than the body, so the response stays a valid OpenAI object and a client that ignores headers is unaffected. Why this is worth running unconditionally: measured on real traffic, a wasted cloud completion costs about 52 local checks at 1,500 tokens and 139 at 4,000. A check that costs nothing at all clears that bar trivially. Tests 159 -> 179. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
bf16327b22 |
docs: add README, and tighten the context estimate that real traffic exceeded
README.md (from an opencode session routed through the router itself) documents what is actually built, module by module, with the pure/IO split and the four eval scoring kinds. It is accurate: every module it names exists, all four "pure" modules import no I/O, and it correctly files RAG context assembly under limitations rather than features. CLAUDE.md now points at it for "what is built" and at the design doc for "what is planned", and no longer claims 74 tests when there are 156. Also fixes a guardrail that real use walked straight through. CHARS_PER_TOKEN was 4, which holds for prose but not for agent traffic. A 106,158-token opencode prompt was routed to qwen3.6-35b, whose effective window is 94,196 -- overshooting the context hard filter by ~12k tokens. True density was at most 3.55 chars/token; code, JSON and tool schemas pack far denser than prose. It survived only on the 0.75 safety factor's slack against the advertised 131,056 limit. Set to 3, deliberately conservative. The asymmetry is the point: underestimating silently admits a model that cannot hold the prompt, while overestimating merely picks a roomier one. Tests cover the exact failing case. Worth recording separately: the opencode session's chat summary claimed a profiler.py module, 13 modules and 5,920 words. There is no profiler.py, there are 12 modules, and the file is 2,614 words. The artifact was accurate; the model's self-report about the artifact was not. Tests 156 -> 159. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
82d06680f3 |
docs: record the routing survey with all three axes live
With cost, eco and proficiency all populated, 27 routing decisions produce ONE model. That is not a regression: qwen3.6-35b is Pareto-dominant in the routable set -- cheapest and cleanest -- while scoring within 0.15 of the best proficiency. On summarization it costs 7x less and emits 65x less carbon than kimi-k3, which beats it 1.00 to 0.85 on quality; no defensible weighting prefers kimi-k3 there. The earlier 7-model spread was measured while energy_observations was empty, so cost and eco were scoring the neutral 0.5 and proficiency decided alone. Recording both numbers, since the difference between them is the whole argument for measuring rather than assuming. Proficiency remains live below the top slot: runner-up ordering reorders by category, with kimi-k3-fast climbing to 3rd on docs_writing and summarization (proficiency 1.00) and leaving the top four elsewhere. The axis works; it cannot overturn a leader that wins on two axes simultaneously. The seed timer's first run has landed, so cost/eco medians now span two time points rather than one moment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
2a6b48449c |
fix: restore energy data, surface empty scoring axes, and sample across time
Asking whether the service was running current code turned up three problems it was not. 1. energy_observations was EMPTY. Recreating the DB for the base_model_id schema change dropped every reference-workload row, so cost_score and eco_score had silently fallen back to the neutral 0.5. The 7-distinct-model routing result reported earlier was therefore proficiency ALONE, not proficiency plus measured cost/eco. Re-swept. 2. The units would not have survived a reboot. Lingering was never enabled, so the user manager stops at logout. Enabled, and moved into the documented install steps rather than a footnote. 3. glm-5.2-flex had no proficiency at all, from a real bug. Propagation copied from the row whose id equals base_model_id -- glm-5.2 -- which is canary and never evaluated. Matching was also on family alone, so a '-fast' row could inherit its reasoning-ENABLED sibling's scores. Inheritance now requires matching (base_model_id, reasoning_mode, context_variant), and a flex row with no evaluated equivalent is measured directly instead of scoring blank. /health now reports scoring coverage and warns when an axis has no data. An empty axis is not an error -- every candidate takes 0.5 and routing still works -- which is exactly what makes it dangerous: a 0.4 weight can contribute nothing while health says "ok". It has now happened twice, so it is surfaced rather than inferred. Adds llm-router-seed.timer. Attribution reproduces within about 30 minutes (0.3-1.1x on a spot-check) but not across hours: between two sweeps deepseek-v4-flash moved ~50x and qwen3.6-35b ~7x the other way, enough to invert their cost ranking. More samples inside one sweep measures one moment more precisely; coverage across time is what actually helps. load_candidates already medians over all seed_reference rows, so a periodic small sweep turns that into a median-across-time for free. Until several sweeps accumulate, the cost and eco ordering is provisional -- a single sweep's ranking is one sample of a moving quantity. Tests 153 -> 156. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
ef06e7d0d1 |
feat: populate proficiency, so task_category finally changes routing
proficiency_score is the only category-dependent term in the composite, so with the table empty the classifier's category output was computed, paid for at ~10s a request, and then discarded. Across 27 decisions (9 categories x 3 tiers) routing produced 2 distinct models under list-price scoring and 3 under measured cost/eco. It now produces 7, with four different models winning tier 1 depending on category. Adds: - proficiency.py / proficiency_store.py -- pure blending plus the single write path, so blended_score and source cannot drift from their inputs. Scores accumulate into a running mean rather than replacing, so re-running the harness tightens estimates instead of discarding history. - leaderboards.yaml / leaderboard.py -- curated per-family priors and their importer, for cold start: a newly listed NeuralWatt family has no self-eval history and would otherwise be indistinguishable from a model measured and found average. Ships EMPTY on purpose; inventing benchmark numbers would put fabricated data into routing, the same failure as the provider's static_fallback carbon constant this project already excludes. `leaderboard.py --check` names every family missing a prior. - evals/tasks.yaml / eval_proficiency.py -- 23 tasks over all 9 categories, scored objectively wherever the category admits it: code executed against checks, exact answers compared, tool calls inspected structurally. Only the four prose categories use a judge, and a judge never grades its own family. - base_model_id on models, so -flex rows inherit their family's scores rather than being re-measured: same weights, different queue. The blending rule needed a fallback the design doc did not specify. Read literally, a model with no leaderboard prior and 9 real samples scores nothing. Self-eval now carries it, labelled self_eval_thin so thin evidence stays distinguishable from evidence that cleared the threshold. Findings: coding does NOT discriminate this catalog -- all 13 rows score 1.00 on all three coding categories even after the tasks were hardened with touching intervals, present-but-falsy defaults, late-binding closures and a binary search that infinite-loops. What discriminates is tool use, arithmetic traps and prose. deepseek-v4-flash scores 1.00 on coding but 0.33 on tool_use_agentic: given a prompt containing both times it needed, it calls two tools instead of subtracting. The router now avoids it there while still choosing it for coding. Three harness defects were found and fixed along the way, each of which scored the rig rather than the model: a token budget shared between a reasoning trace and the answer (empty completions scored 0.00), a single leading space making valid code an IndentationError, and judge malfunctions recorded as model failures. tests/test_task_set.py now validates every task against a reference solution so a broken check cannot masquerade as difficulty -- it caught one on its first run. Tests 134 -> 153. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
3d264c8a8d |
fix: make the classifier path survive a slow or unhappy local model
Three defects, all found by pointing opencode at the running service rather than by testing the classifier in isolation. 1. max_retries=0 on the classifier client. The OpenAI SDK retries twice by default, so classifier.timeout_seconds was silently a 3x wall-clock bound: a request hung past 250s against a 120s setting and logged nothing at all. 2. Cap the classifier's generation at 1024 tokens. qwen3.5 is a reasoning model and will otherwise emit an unbounded chain of thought. That does not fail in isolation, it cascades -- Ollama keeps generating after the client gives up AND serializes per model, so one runaway request queues every later request behind it. Observed as alternating 120s timeouts and 14s successes on an idle GPU with the model resident. 256 was tried first and was too tight: the thinking trace consumed the whole budget and the model was truncated before emitting any JSON, which surfaced as "Classifier returned non-JSON output: ''". Prompt-dependent, so /route passed six consecutive calls while /v1/chat/completions failed. 3. Classifier failure degrades instead of aborting. A timeout, transport error, or unparseable response now yields a configured fallback tier and category flagged source='fallback', rather than 502/503. The caller is a coding agent that would rather have a mid-tier answer than an error. Escalation skips fallbacks deliberately: bumping a low-confidence fallback would send every request to the frontier tier precisely when the local model is unavailable, which is the expensive failure mode. Effect: eight varied prompts through /v1/chat/completions all return 200 (4-25s, the spread being how much the model chooses to think), where before the same path cascaded into timeouts. opencode round-trips cleanly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
7675c61e2d |
docs: mark the GLM grid question answered
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
b3127d0eb0 |
fix: score on attributed cost and carbon, and drop fabricated grid data
Two corrections, both surfaced by a routing decision that looked wrong to a human before it looked wrong in the data. 1. Carbon reported as carbon_source='static_fallback' is excluded from eco scoring. NeuralWatt substitutes a constant (475.0 gCO2/kWh, roughly a global average) when it cannot resolve live grid data, while still echoing the original grid_id -- so glm-5.2-fast appeared to sit on a grid 13.6x dirtier than its neighbours on the strength of a placeholder. Affected models now carry no eco figure and score the neutral 0.5, which is honest; ranking them against an invented number was not. 2. Cost and eco use the median rather than the mean. Distributions are right-skewed: deepseek-v4-flash sampled 7.7e-07 .. 1.0e-04, one spike 33x the median, enough to rank it at roughly twice gemma-4-31b's cost when by median it is a fraction. Also records avg_power_watts, duration_seconds and attribution_ratio, which made a third question answerable. Billed energy decomposes exactly as power x duration x attribution_ratio, and that ratio looks like noise up close -- eight rapid identical calls spanned 20x, correlating +0.997 with it. Scoring on the pre-attribution product was implemented and then reverted: the median ratio spans 750x BETWEEN models against ~1.8x within one, and the values are quantized (0.001, 0.25, 0.5, 0.75). That is serving concurrency, a stable per-model property and real money -- deepseek-v4-flash bills ~1000x under its share of pool gross. Normalizing it away would discard a 750x signal to suppress a 1.8x one. gross_energy_kwh survives as a diagnostic. Split-half validation on a 7-sample sweep: 10 of 13 models agree within 1.4x. kimi-k2.7-code-fast (29x), kimi-k3 (14x) and glm-5.2-flex (2.2x) do not, and are flagged as needing more samples. Effect: tier 1 now routes to deepseek-v4-flash, which is both the cheapest (9.0e-06, 5x under the next) and the cleanest (4.7e-05, 5x under). Category still cannot influence a decision -- proficiency remains the only category-dependent term and is still empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
0fce98d416 |
feat: score cost and eco on measured billing and carbon, not list price
Adds seed_energy.py, which runs a fixed reference task N times per routable model and writes energy_observations rows tagged 'seed_reference'. Scoring now reads mean measured USD billed and mean measured gCO2eq from those rows instead of the catalog's list price. List price was not merely imprecise, it was inverted: deepseek-v4-flash lists 33% cheaper than gemma-4-31b ($0.28 vs $0.42 per 1M) but costs 95% more to run ($9.80e-05 vs $5.02e-05) and emits 284% more carbon. Billing is min($8.00/kWh x energy, 3 x list token price), validated against all 65 samples; the ceiling bound twice, both deepseek energy spikes, matching to the cent. That rule stays in the docs as rationale — no billing formula is in the code, since measured cost already accounts for it, including the flex tier. flex_cost_multiplier is therefore removed rather than left as a guess. Cost and eco deliberately stay separate axes. Collapsing them looked right — cost = 8 x energy, so they seemed to be one signal — but carbon is energy times the serving region's grid intensity, and the catalog spans 37 gCO2/kWh (FI) to 505 (US-MIDA-PJM). They rank models differently: glm-5.2-fast is 2nd cheapest and 6th cleanest; kimi-k3-flex draws 3.7x less energy than kimi-k2.7-code while emitting 3.6x more carbon. A test pins it. Only reference-workload observations steer routing. Organic traffic stays logged for accounting but is excluded, because per-request energy varies more with request shape than with the model — 19x across organic traffic on a single model, purely from differing prompt and completion sizes. cost_score and eco_score were byte-identical implementations; both now alias one normalize_inverted. Effect: 27 routing decisions (9 categories x 3 tiers) went from 2 distinct models to 3, and tier 1 flipped from deepseek-v4-flash to gemma-4-31b. Category still cannot influence a decision — proficiency is the only category-dependent term and remains empty, which is now the single highest-value gap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |
||
|
adlee-was-taken
|
c3484f05e5 |
feat: retarget router to NeuralWatt-only, add serving-class routing and an OpenAI-compatible dispatcher
Drops OpenRouter entirely. The provider column and (model_id, provider) key stay so a second provider needs no migration. Verified against the live API — the poller's field mappings were previously unconfirmed guesses and turned out correct. Routing correctness: - Tier on metadata.reasoning.default_enabled, not capabilities.reasoning. The latter only means "the endpoint accepts a reasoning param" and is true for 17 of 19 rows, which put 17 models in tier 3 and left tier 1 empty. Cost is now checked before the reasoning rule so $0.28/1M models can reach tier 1. Distribution goes from 2/17 to 4/6/9. - Capture serving class. NeuralWatt ships ~6 base models as 19 rows whose id suffixes are three orthogonal dimensions (hence glm-5.2-short-fast-flex): -flex is discounted async held during peak, -fast is reasoning disabled or capped, -short is a 200K pool. They carry identical catalog pricing, so without these columns all 7 GLM rows tie exactly and an interactive request could land on a preemptible row. Latency tolerance is a hard filter, not a weight. Suffixes match whole segments so deepseek-v4-flash is not read as a -fast row. - Exclude access-gated models. 6 of 19 rows are grant-gated or canary, marked only in prose, and would 403 at dispatch. - Log the provider's real billed cost and carbon rather than a tokens x list-price estimate, and score eco on carbon per design doc §4. New dispatcher.py exposes /health, /route (dry run, no spend), /dispatch, plus an OpenAI-compatible /v1/models and /v1/chat/completions so any normal client can use it. Streaming is proxied chunk by chunk; NeuralWatt emits energy and cost as SSE comment lines, which clients ignore and the router reads on the way past — otherwise streamed calls would log no energy at all. Classifier now gets the allowed category list injected from config (it was returning invented labels that join against nothing) and runs at temperature 0, because the same prompt was classifying tier 2 then tier 1 and routing to different models. Ships systemd user units. The poller timer is load-bearing, not housekeeping: stale_after_days is 3 with exclude_stale true, so an unpolled catalog eventually marks every row stale and the router returns no candidates at all. Documents the finding that most affects this project: NeuralWatt bills a flat $8.00/kWh, not per token. List price ranks models backwards — on the same prompt kimi-k2.7-code-fast ($4/1M) cost 10x more than kimi-k3-fast ($15/1M). scoring.cost_score still reads list price; re-basing it is the open call. Tests 28 -> 74. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa |