6krrt/plans/local-dispatch-inert-and-test-coupling.md

Local dispatch is inert, and the tests are coupled to deployment config

Status: done -- local dispatch dormant under the default profile, by design

Status: FINAL — decision-complete except for one question only the user can answer (§1). Findings are from the first live execution of expand-local-llm-usage todo 13 on 2026-09-02/03, against a Quadro RTX 6000 at a measured tariff of $0.159/kWh.

Three of the defects this run found are already fixed and committed in 35a5939. What remains is one design question and two test-hygiene items.

1. The blocker: local dispatch can never be selected

The feature works end to end — catalog row, hard filter, metering, measured pricing — and then never fires, because it cannot win the ranking:

model	file_summarization	8k+400 request
deepseek-v4-flash (cloud)	0.95	$0.001232
qwen2.5-coder-router:14b (local)	0.767	$0.000135

The gap is 0.183. objective.quality_tolerance is 0.1. Ranking is quality-first with cost as a tiebreak only inside that band, so the local model's 9.1x cost advantage is never consulted. Verified live:

POST /route {"task_category":"file_summarization","task_tier":1}
  -> selected: deepseek-v4-flash  prof=0.95

The hard filter is fine — coding_general correctly excludes the local row. The problem is purely that quality-first ranking makes the cost axis unreachable for any model more than quality_tolerance below the best.

This is not a bug. It is the documented routing philosophy ("quality is the objective and cost is the tiebreak") doing exactly what it says. The plan built the mechanism without checking that a local model could clear the bar it would be judged against.

The question for the user, and it is genuinely theirs: what is local dispatch for? Three coherent answers, with different implementations:

• (a) It is a cost/privacy lever the user opts into. Then quality-first is the wrong rule for these categories and it needs an explicit override — e.g. a per-category prefer_local or a cost-ceiling rule that admits the cheapest candidate above a quality FLOOR rather than the best candidate within a tolerance. Do NOT implement by widening quality_tolerance globally: 0.183 would change routing in every other category too, and the tolerance exists to describe measurement noise (2-3 samples), not preference.
• (b) It should fire only when it is genuinely competitive. Then nothing to build; the feature is correct and dormant until a local model scores within 0.1 of the cloud leader. Say so in the docs so the next reader does not re-diagnose it, and drop the eligible categories to avoid implying otherwise.
• (c) It is for when the cloud is unavailable (out of credits, offline). Then it is a fallback, not a routing candidate, and belongs on the failover path next to circuit_breaker, not in the ranking.

DECIDED 2026-09-03 by the user: (b) AND (c). They compose cleanly — (b) governs the normal path, (c) the degraded one — so implement both:

• (b): change nothing in ranking. Local stays an ordinary candidate that wins only if it scores within quality_tolerance of the cloud leader. Today it does not, so it is dormant under the default profile — by design. Say so in docs/ (and in the local_dispatch_models config comment) with the measured numbers, so the next reader does not spend an evening re-diagnosing a working system. Do NOT widen quality_tolerance, and do NOT strip eligible_categories — they are what make (c) safe.

  Word this as "dormant under the default profile", not "dormant". The project is moving toward named profiles (2026-09-03): default stays the feature-rich automatic router, alongside user-named model sets — "locality", "onlycheaps", "bigboybritches". Option (a) above is therefore NOT rejected, only scoped: a per-category local preference is the wrong thing to impose globally and the right thing to put inside a locality profile. A doc that says "local never routes" becomes false the day profiles land, which is exactly the staleness this section exists to prevent.

  Architectural note for whoever builds profiles: auto:batch is already one. It is a named variant of auto that widens the candidate set to admit -flex rows, so auto:locality extends a documented pattern rather than inventing a mechanism. A profile is a named HARD FILTER over candidates — the stage routing.py already owns, pure and injected — not a new ranking path. eligible_categories is the same idea from the model's side. The open design question is whether a profile is only a candidate set or also carries objective overrides (quality_tolerance, max_energy_per_request); a local-ONLY set works fine as a pure filter, but a MIXED set that prefers local needs the override, or it reproduces the inertness diagnosed here.

• (c): add local fallback to the failover path, so an unavailable cloud degrades to a local answer instead of an error.

(c) design notes — read before implementing

Credit exhaustion is ACCOUNT-level, and that breaks the existing failover. circuit_breaker records a failure only on 5xx and fails over to the next-ranked cloud candidate. An out-of-credits response is a 4xx (402/403/429): it will not trip the breaker, it is not transient, and every cloud row shares one account — so trying the next cloud candidate is guaranteed to fail identically. Walking the whole candidate list before giving up wastes the user's latency to reach a foregone conclusion.

So the fallback trigger is NOT "5xx after retries". It is closer to: a non-retryable, account-level upstream refusal, or no cloud candidate left.

The exact signature is UNKNOWN and must not be invented. The credits ran out on 2026-09-02/03 and the router journal recorded no upstream failure at all — no non-200 /v1/chat/completions, no traceback. We therefore do not know whether NeuralWatt returns 402, 403, or 429, nor its body. First task: add logging that captures the full status line and body of any non-2xx upstream response, so the next occurrence is diagnosable. Until that signature is observed, implement the trigger conservatively (see below) rather than hard-coding a guessed status code.

Conservative trigger that does not need the signature: fall back to local when the dispatch loop has exhausted every cloud candidate, OR when the upstream returns any 4xx that is not attributable to the request itself (i.e. not 400/404/422, which mean the request was malformed and would fail locally too).

Respect eligible_categories even in fallback. It is tempting to serve any category when the cloud is down — "something beats nothing". It does not, for code: this model scored 0/4 at detecting bugs and FABRICATED on 3 of 6 summaries at 4B, and even at 14b it is a summarization-grade model. A confidently wrong code answer is worse than an honest 502. Serve the eligible categories; fail loudly for the rest.

Follow the local_vision precedent, which is the same shape. _run_local_vision / _local_vision_response already implement "cloud cannot serve this, answer locally, return a normal OpenAI-shaped response (including a stream-wrapped variant)". Reuse that structure rather than inventing a second one; CLAUDE.md documents its budget guards and its deliberate choice to fail through to the ordinary error when the local call fails, both of which apply here unchanged.

Mark the decision row. A fallback answer must be distinguishable in route_decisions from a chosen one, or the proficiency loop will read degraded-mode traffic as evidence that local won on merit. Reuse or extend the existing kind values (local_vision is the precedent) rather than silently recording it as chat.

Note the measured asymmetry, which bears on (a): local is 26x cheaper on prompt tokens ($0.0054 vs $0.14 per 1M) but only 1.2x cheaper on completions ($0.2286 vs $0.28). Since prompts dominate real traffic, local wins most on exactly the large-context summarization it is assigned — the cost case is strongest precisely where it is currently blocked.

2. Seven tests fail, and none of them is a product defect

Reproduce: pytest -n auto -q with the current config/config.yaml. With local_energy.enabled: false the count drops to 2, which isolates the two causes cleanly.

2a. Five tests assume local_energy is disabled (test-design coupling)

tests/test_metrics_endpoint.py:27 does CFG = load_config(str(ROOT / "config" / "config.yaml")) — it loads the REAL deployment config. The tests then assert:

assert data["local_energy"] is None

which only holds while the feature is off. Enabling a documented, supported, default-off feature therefore turns the suite red without any code changing.

The suite's correctness must not depend on deployment config. This is the mirror of the rule CLAUDE.md already states in the other direction ("never point config/config.yaml at test fixtures"). Fix by monkeypatching cfg.local_energy.enabled to a known value in the affected tests, or by asserting against the loaded config rather than a hard-coded None. Do NOT fix it by turning the feature back off.

Affected (7 total across both causes; confirm the exact list by running): tests/test_metrics_endpoint.py, tests/test_admin_snapshot.py.

2b. Two tests hardcode the old dispatch model name

tests/test_local_dispatch.py::test_the_shipped_config_has_one_dispatch_model asserts nemotron-mini-router:4b. The shipped model is now qwen2.5-coder-router:14b (see §3). Update the expected value.

Consider whether that test should assert a specific model id at all, or only the shape (exactly one entry, with eligible_categories set). Pinning the id means every deliberate model change is a test failure — which is arguably the point, but it should be a decision rather than an accident.

3. config/config.yaml is deliberately uncommitted

The working tree holds three changes, and they must NOT be committed as one:

change	commit?
local_energy.tariff_usd_per_kwh: 0.159	NO — user's electricity bill
local_energy.enabled: true	NO — user's deployment choice
dispatch/classifier/verifier model -> qwen2.5-coder-router:14b	YES
tiering.model_tiers pin for the new model	YES

The model selection is a project decision backed by measurement and belongs in the repo. The tariff is personal data; expand-local-llm-usage todo 13 says so explicitly. Split the file when committing.

4. Why the model changed from the plan's choice

The plan specified nemotron-mini:4b. Measurement disqualified it:

• diff_checking: answered "no" to 8/8 tasks, and to 3/3 blatant probes including return a + b -> return a - b. A detector that never fires.
• file_summarization: 0.25, and it FABRICATED on 3 of 6 — "falsely claims new exception", "wrongly assumes FK enforcement".

Candidates benched on file_summarization (n=6, judge-scored):

model	score
deepseek-v4-flash (cloud reference)	0.95
qwen2.5-coder-router:14b	0.767
qwen3-router:8b	0.75 (n=4; 2 truncated on reasoning trace)
qwen2.5-coder-router:7b	0.667
nemotron-mini-router:4b	0.25

14b vs 7b is 0.6 of a task on n=6 — inside noise. The 14b was chosen because it avoided the FABRICATION the 7b made on the sqlite-FK task, not for the 0.10.

It also replaced mistral-nemo-router:12b as classifier and verifier, on a 14-case bench replicating CLAUDE.md's original methodology (cold load excluded — the first run did not exclude it and was misleading):

	14b	mistral-nemo:12b
category correct	11/14	10/14
hard failures	0/14	1/14
latency mean	1.07s	1.25s
latency MAX	1.14s	6.10s

Accuracy is a tie (one case). The tail is not: mistral-nemo still shows the documented runaway mode — 6.1s spent to return no parseable JSON, degrading that request to source: "fallback" (tier 2, general_chat).

diff_checking cannot rank models and should not be used to. It has 8 tasks over only 4 scenarios, so one task moves a score 0.125 and the entire observed spread (0.50-0.75) is two tasks. At temperature: 0 re-running is byte-identical, so more samples means more TASKS, not more runs. Either expand it to ~12 scenarios or stop treating its numbers as a ranking.

Non-goals

• Do not widen objective.quality_tolerance to make §1 go away.
• Do not disable local_energy to make §2a go away.
• Do not commit the tariff.
• Do not re-run the seed step expecting different numbers; 35a5939 fixed it and the fit was clean (r^2 = 0.9985).

Success criteria

• §1 answered by the user and implemented, or explicitly deferred with the reason recorded in docs/ so it is not re-diagnosed.
• pytest -n auto -q green with local_energy.enabled: true AND with it false.
• A test proves the suite is insensitive to local_energy.enabled.
• config.yaml's model change committed; tariff and enabled still local.
• docs/ states that local dispatch is quality-gated and, if §1(b) is chosen, that it is dormant by design.