neuralwatt-router-service #1

alee · 2026-08-21T23:10:15Z

alee commented

2026-08-21 23:10:15 +00:00

Getting her somewhat updated.

alee added 20 commits 2026-08-21 23:10:16 +00:00

feat: retarget router to NeuralWatt-only, add serving-class routing and an OpenAI-compatible dispatcher c3484f05e5

Drops OpenRouter entirely. The provider column and (model_id, provider) key
stay so a second provider needs no migration.

Verified against the live API — the poller's field mappings were previously
unconfirmed guesses and turned out correct.

Routing correctness:

- Tier on metadata.reasoning.default_enabled, not capabilities.reasoning.
  The latter only means "the endpoint accepts a reasoning param" and is true
  for 17 of 19 rows, which put 17 models in tier 3 and left tier 1 empty.
  Cost is now checked before the reasoning rule so $0.28/1M models can reach
  tier 1. Distribution goes from 2/17 to 4/6/9.

- Capture serving class. NeuralWatt ships ~6 base models as 19 rows whose id
  suffixes are three orthogonal dimensions (hence glm-5.2-short-fast-flex):
  -flex is discounted async held during peak, -fast is reasoning disabled or
  capped, -short is a 200K pool. They carry identical catalog pricing, so
  without these columns all 7 GLM rows tie exactly and an interactive request
  could land on a preemptible row. Latency tolerance is a hard filter, not a
  weight. Suffixes match whole segments so deepseek-v4-flash is not read as
  a -fast row.

- Exclude access-gated models. 6 of 19 rows are grant-gated or canary, marked
  only in prose, and would 403 at dispatch.

- Log the provider's real billed cost and carbon rather than a
  tokens x list-price estimate, and score eco on carbon per design doc §4.

New dispatcher.py exposes /health, /route (dry run, no spend), /dispatch,
plus an OpenAI-compatible /v1/models and /v1/chat/completions so any normal
client can use it. Streaming is proxied chunk by chunk; NeuralWatt emits
energy and cost as SSE comment lines, which clients ignore and the router
reads on the way past — otherwise streamed calls would log no energy at all.

Classifier now gets the allowed category list injected from config (it was
returning invented labels that join against nothing) and runs at
temperature 0, because the same prompt was classifying tier 2 then tier 1 and
routing to different models.

Ships systemd user units. The poller timer is load-bearing, not
housekeeping: stale_after_days is 3 with exclude_stale true, so an unpolled
catalog eventually marks every row stale and the router returns no candidates
at all.

Documents the finding that most affects this project: NeuralWatt bills a flat
$8.00/kWh, not per token. List price ranks models backwards — on the same
prompt kimi-k2.7-code-fast ($4/1M) cost 10x more than kimi-k3-fast ($15/1M).
scoring.cost_score still reads list price; re-basing it is the open call.

Tests 28 -> 74.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

feat: score cost and eco on measured billing and carbon, not list price 0fce98d416

Adds seed_energy.py, which runs a fixed reference task N times per routable
model and writes energy_observations rows tagged 'seed_reference'. Scoring
now reads mean measured USD billed and mean measured gCO2eq from those rows
instead of the catalog's list price.

List price was not merely imprecise, it was inverted: deepseek-v4-flash
lists 33% cheaper than gemma-4-31b ($0.28 vs $0.42 per 1M) but costs 95%
more to run ($9.80e-05 vs $5.02e-05) and emits 284% more carbon. Billing is
min($8.00/kWh x energy, 3 x list token price), validated against all 65
samples; the ceiling bound twice, both deepseek energy spikes, matching to
the cent. That rule stays in the docs as rationale — no billing formula is
in the code, since measured cost already accounts for it, including the flex
tier. flex_cost_multiplier is therefore removed rather than left as a guess.

Cost and eco deliberately stay separate axes. Collapsing them looked
right — cost = 8 x energy, so they seemed to be one signal — but carbon is
energy times the serving region's grid intensity, and the catalog spans
37 gCO2/kWh (FI) to 505 (US-MIDA-PJM). They rank models differently:
glm-5.2-fast is 2nd cheapest and 6th cleanest; kimi-k3-flex draws 3.7x less
energy than kimi-k2.7-code while emitting 3.6x more carbon. A test pins it.

Only reference-workload observations steer routing. Organic traffic stays
logged for accounting but is excluded, because per-request energy varies
more with request shape than with the model — 19x across organic traffic on
a single model, purely from differing prompt and completion sizes.

cost_score and eco_score were byte-identical implementations; both now alias
one normalize_inverted.

Effect: 27 routing decisions (9 categories x 3 tiers) went from 2 distinct
models to 3, and tier 1 flipped from deepseek-v4-flash to gemma-4-31b.
Category still cannot influence a decision — proficiency is the only
category-dependent term and remains empty, which is now the single
highest-value gap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

fix: score on attributed cost and carbon, and drop fabricated grid data b3127d0eb0

Two corrections, both surfaced by a routing decision that looked wrong to a
human before it looked wrong in the data.

1. Carbon reported as carbon_source='static_fallback' is excluded from eco
   scoring. NeuralWatt substitutes a constant (475.0 gCO2/kWh, roughly a
   global average) when it cannot resolve live grid data, while still
   echoing the original grid_id -- so glm-5.2-fast appeared to sit on a grid
   13.6x dirtier than its neighbours on the strength of a placeholder.
   Affected models now carry no eco figure and score the neutral 0.5, which
   is honest; ranking them against an invented number was not.

2. Cost and eco use the median rather than the mean. Distributions are
   right-skewed: deepseek-v4-flash sampled 7.7e-07 .. 1.0e-04, one spike 33x
   the median, enough to rank it at roughly twice gemma-4-31b's cost when by
   median it is a fraction.

Also records avg_power_watts, duration_seconds and attribution_ratio, which
made a third question answerable. Billed energy decomposes exactly as
power x duration x attribution_ratio, and that ratio looks like noise up
close -- eight rapid identical calls spanned 20x, correlating +0.997 with it.
Scoring on the pre-attribution product was implemented and then reverted: the
median ratio spans 750x BETWEEN models against ~1.8x within one, and the
values are quantized (0.001, 0.25, 0.5, 0.75). That is serving concurrency, a
stable per-model property and real money -- deepseek-v4-flash bills ~1000x
under its share of pool gross. Normalizing it away would discard a 750x
signal to suppress a 1.8x one. gross_energy_kwh survives as a diagnostic.

Split-half validation on a 7-sample sweep: 10 of 13 models agree within 1.4x.
kimi-k2.7-code-fast (29x), kimi-k3 (14x) and glm-5.2-flex (2.2x) do not, and
are flagged as needing more samples.

Effect: tier 1 now routes to deepseek-v4-flash, which is both the cheapest
(9.0e-06, 5x under the next) and the cleanest (4.7e-05, 5x under). Category
still cannot influence a decision -- proficiency remains the only
category-dependent term and is still empty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

docs: mark the GLM grid question answered 7675c61e2d

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

fix: make the classifier path survive a slow or unhappy local model 3d264c8a8d

Three defects, all found by pointing opencode at the running service rather
than by testing the classifier in isolation.

1. max_retries=0 on the classifier client. The OpenAI SDK retries twice by
   default, so classifier.timeout_seconds was silently a 3x wall-clock bound:
   a request hung past 250s against a 120s setting and logged nothing at all.

2. Cap the classifier's generation at 1024 tokens. qwen3.5 is a reasoning
   model and will otherwise emit an unbounded chain of thought. That does not
   fail in isolation, it cascades -- Ollama keeps generating after the client
   gives up AND serializes per model, so one runaway request queues every
   later request behind it. Observed as alternating 120s timeouts and 14s
   successes on an idle GPU with the model resident.

   256 was tried first and was too tight: the thinking trace consumed the
   whole budget and the model was truncated before emitting any JSON, which
   surfaced as "Classifier returned non-JSON output: ''". Prompt-dependent,
   so /route passed six consecutive calls while /v1/chat/completions failed.

3. Classifier failure degrades instead of aborting. A timeout, transport
   error, or unparseable response now yields a configured fallback tier and
   category flagged source='fallback', rather than 502/503. The caller is a
   coding agent that would rather have a mid-tier answer than an error.
   Escalation skips fallbacks deliberately: bumping a low-confidence fallback
   would send every request to the frontier tier precisely when the local
   model is unavailable, which is the expensive failure mode.

Effect: eight varied prompts through /v1/chat/completions all return 200
(4-25s, the spread being how much the model chooses to think), where before
the same path cascaded into timeouts. opencode round-trips cleanly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

feat: populate proficiency, so task_category finally changes routing ef06e7d0d1

proficiency_score is the only category-dependent term in the composite, so
with the table empty the classifier's category output was computed, paid for
at ~10s a request, and then discarded. Across 27 decisions (9 categories x 3
tiers) routing produced 2 distinct models under list-price scoring and 3
under measured cost/eco. It now produces 7, with four different models
winning tier 1 depending on category.

Adds:
- proficiency.py / proficiency_store.py -- pure blending plus the single
  write path, so blended_score and source cannot drift from their inputs.
  Scores accumulate into a running mean rather than replacing, so re-running
  the harness tightens estimates instead of discarding history.
- leaderboards.yaml / leaderboard.py -- curated per-family priors and their
  importer, for cold start: a newly listed NeuralWatt family has no self-eval
  history and would otherwise be indistinguishable from a model measured and
  found average. Ships EMPTY on purpose; inventing benchmark numbers would put
  fabricated data into routing, the same failure as the provider's
  static_fallback carbon constant this project already excludes.
  `leaderboard.py --check` names every family missing a prior.
- evals/tasks.yaml / eval_proficiency.py -- 23 tasks over all 9 categories,
  scored objectively wherever the category admits it: code executed against
  checks, exact answers compared, tool calls inspected structurally. Only the
  four prose categories use a judge, and a judge never grades its own family.
- base_model_id on models, so -flex rows inherit their family's scores rather
  than being re-measured: same weights, different queue.

The blending rule needed a fallback the design doc did not specify. Read
literally, a model with no leaderboard prior and 9 real samples scores
nothing. Self-eval now carries it, labelled self_eval_thin so thin evidence
stays distinguishable from evidence that cleared the threshold.

Findings: coding does NOT discriminate this catalog -- all 13 rows score 1.00
on all three coding categories even after the tasks were hardened with
touching intervals, present-but-falsy defaults, late-binding closures and a
binary search that infinite-loops. What discriminates is tool use, arithmetic
traps and prose. deepseek-v4-flash scores 1.00 on coding but 0.33 on
tool_use_agentic: given a prompt containing both times it needed, it calls two
tools instead of subtracting. The router now avoids it there while still
choosing it for coding.

Three harness defects were found and fixed along the way, each of which
scored the rig rather than the model: a token budget shared between a
reasoning trace and the answer (empty completions scored 0.00), a single
leading space making valid code an IndentationError, and judge malfunctions
recorded as model failures. tests/test_task_set.py now validates every task
against a reference solution so a broken check cannot masquerade as
difficulty -- it caught one on its first run.

Tests 134 -> 153.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

fix: restore energy data, surface empty scoring axes, and sample across time 2a6b48449c

Asking whether the service was running current code turned up three problems
it was not.

1. energy_observations was EMPTY. Recreating the DB for the base_model_id
   schema change dropped every reference-workload row, so cost_score and
   eco_score had silently fallen back to the neutral 0.5. The 7-distinct-model
   routing result reported earlier was therefore proficiency ALONE, not
   proficiency plus measured cost/eco. Re-swept.

2. The units would not have survived a reboot. Lingering was never enabled,
   so the user manager stops at logout. Enabled, and moved into the documented
   install steps rather than a footnote.

3. glm-5.2-flex had no proficiency at all, from a real bug. Propagation copied
   from the row whose id equals base_model_id -- glm-5.2 -- which is canary and
   never evaluated. Matching was also on family alone, so a '-fast' row could
   inherit its reasoning-ENABLED sibling's scores. Inheritance now requires
   matching (base_model_id, reasoning_mode, context_variant), and a flex row
   with no evaluated equivalent is measured directly instead of scoring blank.

/health now reports scoring coverage and warns when an axis has no data. An
empty axis is not an error -- every candidate takes 0.5 and routing still
works -- which is exactly what makes it dangerous: a 0.4 weight can contribute
nothing while health says "ok". It has now happened twice, so it is surfaced
rather than inferred.

Adds llm-router-seed.timer. Attribution reproduces within about 30 minutes
(0.3-1.1x on a spot-check) but not across hours: between two sweeps
deepseek-v4-flash moved ~50x and qwen3.6-35b ~7x the other way, enough to
invert their cost ranking. More samples inside one sweep measures one moment
more precisely; coverage across time is what actually helps. load_candidates
already medians over all seed_reference rows, so a periodic small sweep turns
that into a median-across-time for free.

Until several sweeps accumulate, the cost and eco ordering is provisional --
a single sweep's ranking is one sample of a moving quantity.

Tests 153 -> 156.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

docs: record the routing survey with all three axes live 82d06680f3

With cost, eco and proficiency all populated, 27 routing decisions produce
ONE model. That is not a regression: qwen3.6-35b is Pareto-dominant in the
routable set -- cheapest and cleanest -- while scoring within 0.15 of the
best proficiency. On summarization it costs 7x less and emits 65x less
carbon than kimi-k3, which beats it 1.00 to 0.85 on quality; no defensible
weighting prefers kimi-k3 there.

The earlier 7-model spread was measured while energy_observations was empty,
so cost and eco were scoring the neutral 0.5 and proficiency decided alone.
Recording both numbers, since the difference between them is the whole
argument for measuring rather than assuming.

Proficiency remains live below the top slot: runner-up ordering reorders by
category, with kimi-k3-fast climbing to 3rd on docs_writing and summarization
(proficiency 1.00) and leaving the top four elsewhere. The axis works; it
cannot overturn a leader that wins on two axes simultaneously.

The seed timer's first run has landed, so cost/eco medians now span two time
points rather than one moment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

docs: add README, and tighten the context estimate that real traffic exceeded bf16327b22

README.md (from an opencode session routed through the router itself)
documents what is actually built, module by module, with the pure/IO split
and the four eval scoring kinds. It is accurate: every module it names
exists, all four "pure" modules import no I/O, and it correctly files RAG
context assembly under limitations rather than features. CLAUDE.md now points
at it for "what is built" and at the design doc for "what is planned", and no
longer claims 74 tests when there are 156.

Also fixes a guardrail that real use walked straight through.
CHARS_PER_TOKEN was 4, which holds for prose but not for agent traffic. A
106,158-token opencode prompt was routed to qwen3.6-35b, whose effective
window is 94,196 -- overshooting the context hard filter by ~12k tokens.
True density was at most 3.55 chars/token; code, JSON and tool schemas pack
far denser than prose. It survived only on the 0.75 safety factor's slack
against the advertised 131,056 limit.

Set to 3, deliberately conservative. The asymmetry is the point:
underestimating silently admits a model that cannot hold the prompt, while
overestimating merely picks a roomier one. Tests cover the exact failing
case.

Worth recording separately: the opencode session's chat summary claimed a
profiler.py module, 13 modules and 5,920 words. There is no profiler.py, there
are 12 modules, and the file is 2,614 words. The artifact was accurate; the
model's self-report about the artifact was not.

Tests 156 -> 159.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

feat: verify every response structurally, without executing it c360bde17f

Free, exact checks on what a model just returned. Extracts fenced blocks and
validates them by PARSING: ast.parse for Python, json.loads, yaml.safe_load.
Plus finish_reason='length' and unterminated fences, which catch the failure
mode that has bitten this project repeatedly -- a truncated answer looks
complete to the client, which then acts on a fragment.

The router deliberately does NOT execute model output, unlike
eval_proficiency.py. There the prompts are ones this project authored, so what
comes back is bounded. Here the code is whatever the user asked for and could
do anything, so running it as a side effect of routing would be indefensible.
Two tests assert non-execution, including one that would delete a file.

'unverifiable' is recorded and is NOT a failure. Most prose lands there, and
counting "we could not check this" as "this was wrong" would penalize models
for the checker's limits -- the same mistake as scoring a judge malfunction
against a model, which this project made once already.

Streaming accumulates deltas so a streamed answer gets the same check as a
buffered one; otherwise streaming, which is how every agent client talks to
the router, would be the unverified path. Non-streaming reports the verdict
in an X-Router-Verification header rather than the body, so the response
stays a valid OpenAI object and a client that ignores headers is unaffected.

Why this is worth running unconditionally: measured on real traffic, a wasted
cloud completion costs about 52 local checks at 1,500 tokens and 139 at 4,000.
A check that costs nothing at all clears that bar trivially.

Tests 159 -> 179.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

feat: local LLM verification for responses nothing structural can check cf9f5c6be9

Runs on the local model AFTER the response has gone back to the client, so
its ~6s never lands on anyone's latency. It exists to learn which models fail
on real work, not to gate answers.

Gated on answer size, which is economics rather than taste: a local check
costs ~15% of a median 193-token answer, so it would only pay if such answers
failed more than ~15% of the time. At 1,500 completion tokens the break-even
failure rate drops to ~1.9%. Below the threshold a check costs more than the
risk it removes, so small answers are left alone.

Three defects were found and fixed by running it, each of which would have
recorded false failures against models:

1. Head-truncating the answer for the checker made a complete 6,637-character
   response end mid-sentence, and the checker duly reported "cut off
   mid-thought". The harness manufactured the defect it then detected. Now the
   MIDDLE is elided and the cut is labelled, so the ending being judged is the
   real ending.

2. The checker then flagged the elision marker itself as a defect, so the
   system prompt now explains that the marker is the harness, not the model.

3. With thinking enabled the local model never emitted a verdict at all: 2,048
   tokens produced 7,880 characters of reasoning and EMPTY content, and raising
   the cap to 8,192 simply bought more reasoning. Verification does not need
   chain-of-thought. Switched to Ollama's native endpoint, the only one that
   exposes `think`, and with think=False it answers in 25 tokens.

An empty response is now settled by an if-statement rather than a model: asked
about one, the local checker returned ok=true with the reason "Answer is too
short" -- self-contradictory. Never ask a model what code can decide.

After those fixes the checker returned the same verdict on 3/3 repeats of four
labelled cases (complete, truncated, refusal, wrong-question). A malfunctioning
checker still records NO sample rather than a failure.

Tests 179 -> 193.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

feat: fold observed verification failures back into proficiency 6c6a8a0edc

The eval harness measures models on a fixed 23-task benchmark. feedback.py
measures them on real traffic, which is more predictive of routing quality and
accumulates for free as you work. Proven end to end: a truncated response
dropped qwen3.6-35b's coding_general from 1.00 to 0.75 while leaving
coding_refactor untouched.

Only FAILURES are folded in, deliberately. A structural 'ok' means the code
parsed, not that it was correct -- a model emitting syntactically valid
nonsense would score 1.0. Recording passes would flood self_eval_score with
1.0 samples and wash out the benchmark's discrimination; the coding categories
already sit at 1.00 for every model and this would spread that flatness
everywhere. A model that never fails keeps its benchmark score untouched; one
that does is penalized in proportion to how often.

Testing it exposed a flaw in the design. The failure I forced was caused by
MY OWN max_tokens=40, not by the model -- and without a fix, any agent setting
a tight cap would systematically drag down whatever model it routed to. Now a
truncation under a client-supplied max_tokens is recorded (the response really
was unusable) but marked model_attributable=0 and excluded from feedback. The
score that penalty wrongly cost has been restored.

applied_at makes application idempotent, so re-running cannot punish a model
repeatedly for the same bad response.

Tests 193 -> 207.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

docs: refresh module list and test counts (207 across 11 files) dac40c8fb2

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

docs: correct a stale limitation and document the single-model routing outcome cdb186348a

Two gaps in the README, found by checking it against the running system.

The "no escalation feedback" limitation undersold what shipped. Flagging a bad
response DOES exist now -- verification.py detects failures and feedback.py
folds them into proficiency automatically. What is missing is only the action:
nothing retries or escalates. That is deliberate, since whether auto-escalation
pays should come from the observed failure rate now being collected.

More usefully, the README said nothing about the fact that all 27 routing
decisions currently return qwen3.6-35b. A reader would reasonably expect
traffic to spread across models and go hunting for a misconfiguration. It is
Pareto-dominant on this catalog: cheapest AND cleanest, within 0.15 of the
best proficiency. On summarization it costs 7x less and emits 65x less carbon
than kimi-k3, which beats it 1.00 to 0.85 -- no defensible weighting picks
kimi-k3 there.

Also records what is easy to miss: proficiency still reorders the runners-up
by category, and the field widens once the leader stops being eligible --
past ~94,196 tokens qwen3.6-35b is filtered out and glm-5.2-fast takes over,
past ~790K nothing qualifies and the request 422s. On a large codebase the
model will change mid-session, and that is the context filter working.

Every figure verified against the live database before writing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

feat: quality is the objective, energy is the constraint, eco is out 42b71b9bbc

Replaces the three-way weighted blend with a rule that states the dual
mandate honestly: maximize quality, subject to a per-request energy ceiling,
tie-broken by cost.

Eco is dropped from the objective (still logged per request). It was 20% of
every decision optimizing a goal that is handled outside this router.

Cost stops being a weight. Under the blend, 60% of each decision adjudicated
fractions of a cent -- all real traffic to date totals $0.07 -- and min-max
normalization made "expensive" relative to whoever else happened to be a
candidate, so a model could lose for being 2x a very cheap one when both round
to nothing. Measurement retired the blend outright: turning the cost weight
from 0.4 to ZERO changed the winner in only 2 of 6 categories, so it was never
steering on quality.

The ceiling is denominated in kWh, not dollars, because the plan is a
subscription with a 6.25 kWh quota. Dollars accrue and can be reconciled
later; a quota is a wall you hit mid-task. /health now reports burn against
the allowance and warns past 80%, with the caveat that it counts only what the
router saw.

quality_tolerance (0.10) is measurement noise rather than preference:
proficiency rests on 2-3 samples per category, so smaller gaps are sampling
variation and paying more for them buys noise. Narrow it as samples
accumulate.

Also corrects a figure the docs asserted as a constant. Grid intensity is not
37 gCO2/kWh: observations show FI at 49 and 50 at different times, FR at 54,
US-MIDA-PJM at 442, and the provider's own 24h blended figure was 145.4.

Tests 207 -> 211.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

feat: tier is an iteration budget, spent on evidence rather than on a hunch e1fd65f3e4

A tier used to mean only a capability floor. It now also buys corrective
attempts after a verification failure: tier 1 gets one shot, tier 2 one retry,
tier 3 two. Interactive requests are capped below their tier regardless,
because every retry doubles time-to-answer and in interactive use latency IS a
quality loss.

escalation.preemptive_on_low_confidence now defaults to FALSE. Bumping the
tier because the classifier was unsure of its own call pays frontier prices
before anything has gone wrong. Spending after a check has actually failed is
better on both mandates: the cheap attempt usually succeeds and costs nothing
extra, and when it fails there is evidence rather than a hunch.

Retries are matched to the failure, because the causes differ. Truncation
raises the token budget on the same model -- a different one would run out
too. Malformed output escalates to the next-ranked candidate, since more
tokens will not make unparseable output parse. 'ok' and 'unverifiable' buy
nothing; retrying unverifiable would burn quota across the majority of prose
traffic for no signal at all.

Testing it live exposed that the truncation branch was UNREACHABLE as first
written. Through /v1 the token cap is either the client's, which is not ours
to override, or absent -- and when absent the model hit its own ceiling, so
doubling changes nothing. It now escalates to a candidate with a larger output
ceiling, which is the actionable move, and declines when no such candidate
exists rather than wasting an attempt against a fixed kWh quota.

Known limit, recorded in CLAUDE.md: retry does not reach the streaming path.
Once bytes have gone to the client there is nothing to take back, and
buffering to allow correction would cost streaming itself. opencode streams,
so the main workflow gets verification and feedback but not correction.

Tests 211 -> 227.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

feat: POST /outcome — the only ground truth the router can get e28443cb36

Everything else this router records is a proxy. Structural checks know whether
code parses. The local checker guesses whether prose looks right. Neither knows
whether the answer did the job. The client does: it ran the tests, or used the
answer, or watched it fail.

Clients report against the provider's completion id, which they already
receive in the response body and on every stream chunk. energy_observations
and verifications now store that id so a report has something to join on.

Two properties make this the highest-value signal available.

It is the only quality signal that survives streaming. A retry cannot reach a
streamed response -- the bytes are already gone -- but a report arrives
afterwards and works identically either way. Every agent client streams, so
without this the main workflow had verification and feedback but no route from
outcome back into routing.

And its successes count. feedback.py folds client outcomes in BOTH directions,
unlike checks where only failures do. That asymmetry is deliberate: a parser
reporting 'ok' means the code parsed, which is weak evidence that would
inflate every score toward the ceiling, while a client reporting 'succeeded'
means the work worked.

An unknown request_id returns 404 rather than being quietly accepted. A client
whose reports go nowhere should find out rather than train nothing.

Verified end to end on both paths, including a streamed completion reported as
failed after the fact.

Tests 227 -> 232.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

feat: parallel-safe outcome attribution, and an opencode plugin to feed it 6e729ad670

Closes the ground-truth loop. deploy/opencode-plugin/router-outcome.js hooks
tool.execute.after, watches for test and build commands, and reports pass/fail
to /outcome. opencode already runs your tests; this is what makes the result
reach routing.

Command detection is deliberately narrow -- pytest, npm test, cargo, go, ruff,
mypy, tsc and friends. A failing `ls` says nothing about model quality, and a
false signal is worse than none because it trains the router on noise. Verdict
comes from exit status plus text signatures for tools that exit 0 while
reporting failures, with "0 failed" and "no errors" guarded against. The router
being unreachable never breaks a session.

Attribution is the hard part, and two assumptions failed under test.

The first fingerprint design keyed on the system prompt. One real opencode run
produced TWO distinct keys, because it runs several agents with different
prompts -- so that fingerprint identifies AGENTS, not sessions, and would have
refused every single run forever. A permanent false positive dressed as
safety.

The second assumption was that opencode states its project root up front.
Capturing a real request showed it does not. Directory is now derived from the
file paths an agent touches across the whole conversation, counting every
ancestor so the shared project root wins over any one subdirectory, and
stripping trailing filenames so a file is never mistaken for a directory. A
single mention is not enough; corroboration is required.

When a report cannot be matched by directory and more than one conversation
was active in the window, /outcome answers 409 and records nothing. Refusing
beats guessing: a misattributed failure penalizes a model for work it never
did, and this project has already recorded false failures twice from harness
bugs that took measurement to catch.

The window is 120 seconds, not 30 minutes. At 30 it swept in traffic from
earlier in the same work session and refused a legitimate report -- observed
directly, not theorised.

Tests 232 -> 243.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa

fix: an agent turn that calls a tool is not a failed answer f0d9bfb81c

Both verification paths had mirror halves of the same blind spot, and real
traffic is what found it. On the first genuine agent session through this
router -- 63 completions that shipped a working feature with 349 passing
tests, clean mypy and clean ruff -- the structural checker recorded
"malformed: empty response" 29 times and the local LLM checker called
"cuts off mid-sentence" on 8 of the 9 answers it graded.

Both were describing the same thing from opposite sides: a turn that ends by
calling a tool. Its text content is empty, or a half-sentence before the call,
and both are correct behaviour rather than a defect.

verify_response and worth_local_check now take has_tool_calls, supplied on the
non-streaming path from message.tool_calls and accumulated on the streaming
path from delta.tool_calls. In verify_response the check outranks even
finish_reason == 'length', because stopping mid-sentence at a call boundary is
a call boundary, not a budget overrun. worth_local_check declines outright,
which also stops paying ~6s of local inference to mis-grade a tool call.

Had feedback.py run against those rows it would have applied ~12 false
failures to the two models that had just done the work. That is the FOURTH
harness bug in this project that would have scored the rig rather than the
model, and the first one caught by real traffic instead of a synthetic test.
The pre-fix rows are kept with model_attributable = 0 so the record survives
without steering routing.

client_capped was over-applied in the same area. It marked EVERY verdict
non-attributable whenever the client set max_tokens, and opencode always sets
it, so genuine failures were invisible to feedback for the entire main
workflow. A client's token cap explains a truncated verdict and nothing else;
a model emitting unparseable code owes nothing to the client's budget. It is
now scoped to exactly that verdict.

Also recorded: outcome attribution resolves the session directory by path
histogram, and on this session that picked .venv/.../site-packages/c2pa 24
times over ~/Sources/fieldwitness 22, because reading a dependency's source
outweighed editing the project. It degrades safely -- 31 reports accepted, 3
refused as ambiguous rather than misattributed -- but the heuristic needs to
weight writes over reads.

Tests 243 -> 249.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ

fix: price is a market signal, not a capability measurement 69ac8f7745

deepseek-v4-flash was losing every routing decision to glm-5.2-fast on real
agent traffic, and it turned out to be excluded twice over by the same
substitution made in two different places. It has a 1M advertised window,
scores 1.00 on all three coding categories, and lists at $0.14/$0.28 per 1M
against glm's $1.45/$4.50.

The cost axis was measuring the wrong workload. `cost` came from the median
billed USD over seed_energy.py's reference sweep, which sends a 400-token
prompt with a 400-token completion. Real traffic through this router is a
150,000-token prompt with a ~400-token completion and 84% cache hits, and the
ranking does not survive the change of shape:

    reference sweep (400/400)      glm-5.2-fast      3.2x cheaper
    realistic (70k prompt)         deepseek-v4-flash 5.0x cheaper

Measured live, three samples each, not inferred. The attribution ratio is what
moves: glm sits at 0.006 on a toy prompt and 0.50 on a 70k one, because it
batches beautifully at small sizes and badly at real ones, while deepseek
barely shifts (0.21 -> 0.25). A fixed-shape benchmark cannot rank models for a
workload of another shape, and no amount of re-sweeping fixes that -- it
measures one wrong thing more precisely.

routing.estimated_cost now prices each request from catalog token prices
scaled to that request's actual shape, via objective.assumed_cache_rate (0.84,
measured from real traffic) and objective.assumed_completion_tokens. List
price is not what gets billed -- NeuralWatt charges per kWh -- but billing is
capped at 3x list, so it tracks the real ordering and bounds it, and on the
one case checked live it agrees with the measurement in direction and
magnitude (7.8x predicted vs 5.0x measured). It is also free, needs no sweep,
and refreshes whenever the poller runs. A row the catalog has no price for
keeps whatever measured cost it arrived with; a missing list price is not
free.

Three signals said deepseek -- catalog token price, NeuralWatt's own published
per-request energy, and a live 70k measurement. Only the 400-token benchmark
disagreed, and it was the one being scored on.

Tiering made the identical mistake independently. Tier is a capability FLOOR:
routing.py drops any row with tier < required_tier. Resolving tier on
completion price alone put deepseek in tier 1 for no reason but being cheap,
which excluded it OUTRIGHT from every tier-2 request -- so the cost fix alone
would have changed nothing. tiering.resolve_tier now gates tier 1 on
tier1_context_max (512000) as well as cost: tier 1 means small AND cheap, not
merely cheap.

The gate reads the ADVERTISED context_window, whose catalog values are the
clean market classes (131056 / 199984 / 262128 / 1048560), rather than
effective_context_window, which varies within a class. 512000 sits in the
empty band between the 256K and 1M classes with a 2x margin either side, so it
is not fitted to any one model. It only ever demotes -- a huge window never
promotes an expensive model into tier 1 -- and a missing window does not block
tier 1, since absent evidence should not decide anything.

Distribution 4/6/9 -> 1/9/9; only the three deepseek rows moved. No
model_tiers override was added, deliberately: the point is that the heuristic
now gets this right, and pinning it in config would mask whether it does.

deepseek now wins coding_general at every context size (16.5x cheaper than
kimi-k3 at 200k) and is still correctly absent from tool_use_agentic, where
its measured 0.33 drops it out of the quality band. That is the eval data
earning it the slot rather than a thumb on the scale. Verified live against
the running service.

Tests 249 -> 256.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ

alee merged commit 82d6db5746 into main

2026-08-21 23:10:28 +00:00

alee referenced this issue from a commit

2026-08-21 23:10:29 +00:00

Merge pull request 'neuralwatt-router-service' (#1) from neuralwatt-router-service into main

alee referenced this issue from a commit

2026-08-23 17:03:46 +00:00

fix: resolve remaining capability-gate review findings

alee referenced this pull request

2026-08-23 17:03:55 +00:00

fix: resolve remaining capability-gate review findings #3

Sign in to join this conversation.