6krrt/plans/local-decision-classifier-results.md

Gate G — local_decision measurement results

Status: done -- gate G measured; VRAM passes with the 14b at 16k and local vision off (section 9); accuracy and latency pass

Date: 2026-09-28 Plan: plans/local-decision-classifier.md Hardware: Quadro RTX 6000, 24 GB (24576 MiB total) Models: qwen3.5:4b (classifier), qwen2.5-coder-router:14b (router, resident at 32768 ctx = 17.76 GB) Ollama: http://localhost:11434

Headline

Criterion	Measured	Threshold	Verdict
Eval-set accuracy (evals/tasks.yaml)	97.8% (tuned)	≥ 85%	PASS
Held-out accuracy (evals/heldout.yaml)	90.0% (tuned)	≥ 90%	PASS (at threshold)
p95 latency (14b resident)	502 ms	≤ 800 ms	PASS
14b router resident after 100 calls	EVICTED	must stay resident	FAIL

Gate verdict: STOP. The 14b router model is evicted whenever qwen3.5:4b is loaded — on this 24 GB card there is no num_ctx at which the two coexist. Per the plan, "STOP AND REPORT after G if ... the 14b router model was evicted. Do NOT start todos 4-17 in that case." Todos 4-17 are NOT started.

---

1. Accuracy per set

Measured with the eval harness (--backend decision --decision-base-url http://localhost:11434), majority-vote overall accuracy over the clean/short/long noise variants of each task.

Baseline (original _DECISION_DESCRIPTIONS)

Set	clean	short	long	overall
evals/tasks.yaml (46)	0.739	0.783	0.761	0.761
evals/heldout.yaml (30)	0.800	0.933	0.900	0.967

Baseline confusion (eval-set, dominant cells): coding_general → debugging (14) and → reasoning_math (4); file_summarization → debugging (9). The debugging description was attracting both code-tracing and code-summary tasks.

After description tuning

_DECISION_DESCRIPTIONS changes: widen coding_general to include "tracing what existing code returns" and "implementing a feature or algorithm"; narrow debugging to "fixing a bug"; narrow reasoning_math to "pure math or logic word problem (no code)"; soften file_summarization to "summarizing or explaining".

Set	clean	short	long	overall
evals/tasks.yaml (46)	0.978	0.978	0.978	0.978
evals/heldout.yaml (30)	0.800	0.900	0.833	0.900

Tuning lifted the eval-set from 76.1% → 97.8% (the confusion cells are gone: eval-set coding_general 27/27, file_summarization 18/18, debugging 18/18). It cost 6.7 points on held-out (96.7% → 90.0%), landing exactly on the 90% threshold. Held-out residual misses: h2 coding_general→file_summarization, h30 general_chat→file_summarization, h9 debugging→general_chat — all low-confidence (0.33-0.49) edge cases the confidence cascade would absorb. Further tuning to chase these risks overfitting evals/tasks.yaml (the exact failure the plan warns about), so tuning stops here.

2. Calibration (tuned)

Mean confidence when correct vs incorrect (per-noise-level rows).

Set	correct mean	incorrect mean
evals/tasks.yaml	0.798	0.350
evals/heldout.yaml	0.802	0.413

Calibrated in the right direction on both sets: the model is roughly twice as confident when right as when wrong. (Baseline was 0.833/0.438 eval-set, 0.815/0.300 heldout — the tuning moved both closer together but kept correct ≫ incorrect.)

3. Latency (p50 / p95)

Per-call latency of a single classify_choice over the eval-set, qwen3.5:4b, num_ctx 8192, with the 14b router resident (the production steady state).

single category call: n=46 p50=363ms p95=502ms p99=598ms mean=379ms

p95 502 ms ≤ 800 ms — PASS.

4. Open question (a) — noise isolation

Compare classify_choice on raw wrapped (noisy) text vs text passed through local_encoder._isolate_task_text first.

Set	no-isolation (short+long)	isolated (short+long)	clean baseline
evals/tasks.yaml	0.772	0.739	0.739
evals/heldout.yaml	0.917	0.800	0.800

Isolation HURTS on both sets (−3.3 pt eval, −11.7 pt heldout). For local_decision the fenced-code/tool content _isolate_task_text strips is itself the signal (code-tracing, file-summary tasks). Do NOT run _isolate_task_text before classify_choice; send raw text. This overrides the prototype's earlier "noise cost about 2 points" note — for this backend isolation is strictly worse.

5. Open question (b) — tier call strategy

Measure separate-request latency vs a joint prompt reading two token positions.

strategy	p50	p95	reliability
category call (10 options)	361 ms	595 ms	—
tier call (3 options, separate)	326 ms	553 ms	—
separate, sequential total	689 ms	1154 ms	100%
joint prompt, two token positions	434 ms	635 ms	5/30 = 17%

The joint two-position read is faster (p50 434 vs 689 ms) but unreliable: only 5/30 (17%) prompts produced valid letter mass (>0.5) at BOTH token positions — the model does not reliably answer two questions with two letters in one generation.

Recommendation: two separate calls fired in parallel (category + tier). Parallel wall-clock = max(category, tier) ≈ p95 ~595 ms, fully reliable, and matches the plan's component-4 design ("fired in parallel ... so it adds no wall-clock time"). The joint cleverness is rejected on reliability, not speed.

6. Open question (c) — num_ctx trade-off

100-call run at each num_ctx, /api/ps before/after, qwen3.5:4b footprint and 14b residency.

num_ctx	p50	p95	qwen3.5:4b VRAM	14b router resident?
4096	361 ms	588 ms	6.15 GB	NO
8192	364 ms	594 ms	6.32 GB	NO
16384	364 ms	591 ms	6.65 GB	NO

Latency is essentially flat across num_ctx (the classification prompts are short), and VRAM grows 6.15 → 6.65 GB. If coexistence were possible, num_ctx 4096 would be the best trade-off. It is not possible: see below.

7. VRAM / router-model eviction (HARD CONSTRAINT — FAILS)

/api/ps before and after a 100-call run at num_ctx 8192:

BEFORE: {'qwen2.5-coder-router:14b': 17.76}
AFTER : {'qwen3.5:4b': 6.32}     # 14b router GONE
14b router resident after 100 calls: False

Definitive coexistence check — 14b loaded at its production context (32768 = 17.76 GB), then a single qwen3.5:4b call at each num_ctx:

after qwen3.5@4096 : {'qwen3.5:4b': 6.15}                coexist? False
after qwen3.5@8192 : {'qwen3.5:4b': 6.32}                coexist? False
after qwen3.5@16384: {'qwen3.5:4b': 6.65}                coexist? False

Why: the RTX 6000 has 24 GB. The 14b router occupies 17.76 GB (Q4_K_M @ 32768 ctx) plus ~1.2 GB of other GPU processes → ~18.4 GB used, ~5.5 GB free. qwen3.5:4b needs ≥ 6.15 GB at any num_ctx, so loading it always exceeds free VRAM and Ollama evicts the 14b router. 17.76 + 1.2 + 6.15 = 25.1 GB > 24 GB — it cannot physically fit at num_ctx 4096, let alone 8192.

(The earlier plan note "the card sat at 23.4/24 GB with both resident at num_ctx 8192" was not reproducible — the 14b router here is 17.76 GB at its production 32768 context, and the two do not coexist.)

8. Gate summary and next action

• Eval-set 97.8% ≥ 85% — PASS
• Held-out 90.0% ≥ 90% — PASS
• p95 502 ms ≤ 800 ms — PASS
• 14b router resident after 100 calls — FAIL (evicted)

The local_decision approach fails gate G on VRAM and must NOT proceed to todos 4-17. The accuracy and latency bars are met and the description tuning is complete, but the classifier cannot run on this 24 GB card without evicting the 14b router model that local dispatch depends on. Options to revisit before re-running gate G (out of scope here): a smaller classifier model that coexists in the remaining ~5.5 GB (e.g. a ~3-4 GB 2-3B model at low num_ctx), or a model that is small enough that 14b + classifier + other procs fit in 24 GB.

9. Re-measure with the 14b at 16k (reviewer, 2026-09-28, owner option A)

The owner chose option A: drop the router 14b to num_ctx 16384 and re-measure. Local vision fallback (qwen3-vl-router:4b, 5.6 GB) is being turned off: it has never fired (0 route decisions have ever selected a qwen3-vl model).

Measured with a separate test tag (FROM qwen2.5-coder:14b, PARAMETER num_ctx 16384), so the production tag was untouched. The tag was deleted afterwards.

state	/api/ps size_vram	nvidia-smi used
14b @ 16k alone	12.26 GB	14.2 GB
+ qwen3.5:4b @ 4096	12.26 + 5.73 GB	19.7 GB
+ qwen3.5:4b @ 8192	12.26 + 5.89 GB	19.9 GB (about 4 GB spare)

At 16k the two coexist, with about 4 GB of headroom. The vision model (5.6 GB) would not also fit, which is why it is being turned off.

Gate G's "evicted at every num_ctx" is not stable. During the latency run, production traffic reloaded its own 14b at 32k, and it stayed resident alongside the 4b: 16.54 + 5.89 GB, 22.9 of 24 GB used, about 1 GB spare. Ollama's eviction follows its own memory estimate, which varies with load order and other GPU processes (gate G saw the same 14b as 17.76 GB). At 32k the pair fits only by about 1 GB and can flip back to eviction; 16k is the robust setting.

Latency, 100 calls of classify_choice (num_ctx 8192) with both models resident: p50 321 ms, p95 474 ms, max 2545 ms (the first cold call). The 14b stayed resident throughout. p95 474 ms, PASS.

Revised gate verdict: PASS on every criterion provided the 14b router tag runs at num_ctx 16384 and local vision is off. Both are production changes and the owner's call; todos 4-17 may proceed once they are made.