6krrt/design/local-llm-model-router.md

Local LLM Model Router

Status: design draft Owner: Aaron Lee Hardware: RTX 6000 24GB (local classifier/context-assembly), routes to Neuralwatt Cloud

Revised 2026-08-11 after first live run. Neuralwatt is now the only provider (OpenRouter dropped). Two findings from real API traffic change this document materially: billing is per-kWh, not per-token (§4.1), and one base model appears as several catalog rows that differ only by serving class (§3.4). Sections below are annotated where the original design assumed otherwise.

1. Goal

Use a local model on the RTX 6000 to analyze incoming coding/documentation tasks — pulling in relevant architecture docs and code context — and route each task to the cheapest/best-fit open-weight model across multiple providers, rather than defaulting every task to a single expensive cloud model. Router overhead runs on local hardware (electricity, not API spend), so the marginal cost of a routing decision is near zero.

2. Architecture

                    ┌─────────────────────┐
  incoming task ───▶│  Context Assembler   │  (RAG-lite: pulls relevant docs/code)
                    └──────────┬───────────┘
                               ▼
                    ┌─────────────────────┐
                    │  Local Classifier    │  (RTX 6000, e.g. Qwen2.5-32B)
                    │  - task_tier         │
                    │  - required_context  │
                    │  - confidence        │
                    └──────────┬───────────┘
                               ▼
                    ┌─────────────────────┐
                    │  Decision Table       │◀── scheduled pollers (pricing/benchmarks)
                    │  (SQLite, weighted)   │
                    └──────────┬───────────┘
                               ▼
                    ┌─────────────────────┐
                    │  Dispatcher           │──▶ Neuralwatt Cloud
                    │  (OpenAI-compatible)  │──▶ (future providers)
                    └──────────┬───────────┘
                               ▼
                    ┌─────────────────────┐
                    │  Verify / Escalate    │  (local model spot-checks output)
                    └─────────────────────┘

Fits the same pattern as vigilar: scheduled pollers publish updates, a service holds current state, consumers read from it rather than hitting live APIs per-request.

3. Decision table schema

Two tables: a models table (one row per model+provider) and a proficiency table (one row per model+category), joined at query time.

3.1 models

column	type	notes
model_id	text	canonical model name
provider	text	neuralwatt today; column kept for future providers
cost_per_1m_prompt	real	listed token price
cost_per_1m_completion	real	listed token price
cost_per_1m_prompt_cached	real	e.g. Neuralwatt bills cached prefix at 25%
energy_mwh_per_req	real	Neuralwatt-only; null for token-only providers
carbon_g_co2eq_per_req	real	derived from energy × grid intensity, where available
context_window	int	max tokens, provider-advertised
effective_context_window	int	advertised minus a safety margin (see §3.2)
tier	int	1 (cheap/simple) – 3 (frontier reasoning)
availability	text	active, deprecated, stale
last_updated	timestamp

3.2 Context window criteria

Advertised context length isn't the number to route on directly — quality tends to degrade before the hard limit, and you need headroom for the response itself. Rules of thumb to encode:

• effective_context_window = advertised context × 0.75, minus a reserved output budget (e.g. 4K tokens reserved for completion on coding tasks). Adjust the 0.75 factor per model once you have real data — some models hold up much better near their limit than others, and that's worth capturing per-model rather than as one global constant.
• Classifier estimates required_context for the task (assembled docs/code + prompt overhead), not a vague size bucket — an actual token estimate, so it can be compared directly against effective_context_window.
• Hard filter, not a weighted factor: any model where effective_context_window < required_context is excluded from candidates entirely before scoring runs — a cheap model that can't physically hold the context isn't a cost tradeoff, it's disqualified.
• Context size tiers (for quick filtering / dashboarding, not routing itself):
  • small: ≤ 8K
  • medium: 8K–32K
  • large: 32K–128K
  • xlarge: 128K+
• Overflow fallback: if required context exceeds every available model's effective window, the router should hand back to the context assembler to trim/summarize rather than silently picking the largest model and truncating mid-task — truncation failures are worse than a slower round-trip.

3.3 proficiency (benchmark scores by category)

One row per (model, category):

column	type	notes
model_id	text
category	text	see category list below
score	real	0–1, normalized
source	text	leaderboard, self_eval, blended
sample_size	int	for self-run evals, so thin data can be flagged/discounted
last_updated	timestamp

Starting category set (extend as needed — these map to the task types you're actually routing):

category	example task	typical source
coding_general	boilerplate, small functions	Aider polyglot, self-eval
coding_refactor	multi-file refactors, architecture-aware edits	self-eval (most predictive here)
debugging	root-cause / fix-from-error-trace	self-eval
docs_writing	docstrings, README, comments	self-eval, cheap to score
summarization	log/alert summarization (vigilar-style)	self-eval
translation	natural-language translation	leaderboard
reasoning_math	multi-step logic, math-adjacent	LiveBench, leaderboard
tool_use_agentic	function calling, multi-step agent tasks	leaderboard + self-eval
general_chat	catch-all / low-stakes	leaderboard

Blending rule: score = 0.3 × leaderboard_score + 0.7 × self_eval_score once sample_size for self-eval crosses a minimum threshold (e.g. 10 runs); before that, fall back to leaderboard_score alone so a thin/noisy self-eval doesn't dominate early. Store both components, not just the blend, so you can re-weight later without re-running evals.

Category selection at routing time: the local classifier tags each incoming task with a primary category (and optionally a secondary one for hybrid tasks, e.g. "refactor + docs"). proficiency_score used in the composite scoring formula (§4) is the score for that specific category — not an average across categories — so a model that's great at docs_writing but mediocre at coding_refactor doesn't get miscredited on refactor tasks.

3.4 Serving class (added after first live run)

Neuralwatt ships ~6 base models as 19 catalog rows. The id suffixes are three orthogonal dimensions — hence ids like glm-5.2-short-fast-flex — and they are not visible anywhere except the id string and the prose description:

suffix	column	meaning
-flex	latency_class	discounted async tier; requests held server-side during peak until a capacity gap opens
-fast	reasoning_mode	thinking disabled or capped to a short budget. Not queue priority
-short	context_variant	200K pool with a bounded reasoning budget

These rows carry identical catalog pricing, so they tie exactly on every scored dimension. Left to the weighted score, a coin flip decides whether an interactive request lands on a preemptible flex row. So latency tolerance is a third hard filter, not a weight: interactive (default) excludes -flex; batch admits it.

Access gating is likewise prose-only — 6 of 19 rows say "Private preview (grant-gated)" or "(Canary)" in free text with no structured field. Those are parsed into access_level and excluded by default, since routing to one earns a 403 at dispatch rather than a bad answer.

4. Weighted scoring

Router computes a composite score per candidate model and picks the max, after three hard filters are applied (not weighted — these disqualify a candidate outright):

1. effective_context_window >= required_context (§3.2)
2. tier >= required_tier (from classifier)
3. serving class compatible with the request's latency tolerance, and access_level actually reachable by this account (§3.4)

score = (w_cost   × cost_score)
      + (w_eco    × eco_score)
      + (w_prof   × proficiency_score[task_category])

• cost_score: normalized inverse cost (cheapest candidate = 1.0, scaled down from there). ⚠️ See §4.1 — the input this currently reads is not the price being billed.
• eco_score: normalized inverse carbon_g_co2eq_per_req, which Neuralwatt reports per-completion; models without a carbon figure get a neutral/default score rather than being penalized, so a model new to the table isn't downweighted just for lacking history.
• proficiency_score: from the benchmark refresh — general leaderboard score blended with your own eval set (see §5), weighted toward the latter over time as it accumulates.

w_cost, w_eco, w_prof live in a small config file (not hardcoded) so the balance can shift per-project or per-mood — e.g. crank w_eco up for background/batch jobs where latency and cost don't matter much, crank w_prof up for anything touching production code.

Hard floor, not just a weight: task tier from the classifier acts as a minimum bar (don't let a great cost/eco score smuggle a task into a model that's actually too weak for it) — the weights decide within the eligible set, not whether a bad-fit model can win purely on price.

4.1 Billing is per-kWh — the cost axis needs rethinking

The line above ("take whichever the account is actually billed under") turned out to be the whole ballgame. Measured across five models on 2026-08-11, cost_usd / energy_kwh came back 8.00 every time: Neuralwatt bills a flat $8.00/kWh and the catalog's input_per_million / output_per_million figures are not what this account is charged.

The ranking consequence is not subtle. On the same prompt:

model	list $/1M out	billed USD	kWh
kimi-k2.7-code-fast	4.00	2.25e-03	2.81e-04
kimi-k3-fast	15.00	2.17e-04	2.71e-05

The model listing at a quarter the price cost 10x more to run. Any cost_score computed from list price ranks these backwards.

This collapses two of the three scoring axes into one. cost_usd = 8.00 x energy_kwh exactly, so w_cost and w_eco are weighting the same underlying quantity — 0.6 of the composite on one signal, with proficiency left holding 0.4. Cost and eco separate only through grid carbon intensity (grid_carbon_intensity_gco2perkwhr, 37.0 on grid FI at time of writing), which is a real distinction only when routing across regions.

Two further wrinkles before this can be fixed:

• Energy is noisy. Identical token counts (18 -> 900) billed anywhere from 1.8e-05 to 8.8e-05 — a 4x spread run to run. A single observation is not a per-model estimate.
• Cold start. energy_observations is empty until real traffic accrues, so energy-based scoring needs a seeding sweep (fixed prompt, N runs per routable model) before it can drive anything.

4.2 Resolved: measured cost, and the two axes stay separate

Both sub-scores now read measured values from a fixed reference workload (seed_energy.py, 13 models x 5 samples, tagged seed_reference): mean USD actually billed for cost, mean gCO2eq for eco. No billing formula lives in the code — the rule in §4.1 is documentation for why list price is not used, not logic.

The tempting simplification — cost = 8 x energy, therefore cost and eco are one axis — is wrong, and the sweep is what caught it. Cost tracks energy, but carbon is energy x the serving region's grid intensity, and the catalog spans 37 gCO2/kWh (FI) to 505 (US-MIDA-PJM). So the axes disagree: glm-5.2-fast is the 2nd cheapest model and the 6th cleanest; kimi-k3-flex draws 3.7x less energy than kimi-k2.7-code while emitting 3.6x more carbon. Collapsing them would have silently picked a side.

Only reference-workload observations feed scoring. Organic traffic stays in the table for cost accounting but is excluded from routing, because per-request energy varies far more with the shape of a request than with the model — across organic traffic on a single model it spanned 19x purely from differing prompt and completion sizes. Averaging that in would rank models by what they happened to be asked.

Effect on the hard-filter-plus-score pipeline: with list-price scoring, 27 routing decisions (9 categories x 3 tiers) produced 2 distinct models. With measured scoring, 3 — and tier 1 flipped from deepseek-v4-flash to gemma-4-31b, correctly. Category still has zero influence, because proficiency remains empty and is the only category-dependent term (§3.3).

5. Freshness / scheduled jobs

Two independent pollers, different cadences:

• Pricing poller (hourly–daily): hits Neuralwatt /v1/models (unauthenticated for public listing), writes cost/context/serving-class/access columns, flags rows unchanged > N days as stale. Note the catalog carries no energy fields at all — energy arrives only per-completion, so the dispatcher populates energy_observations, not this poller.
• Benchmark poller (weekly, or on-demand): blends general leaderboard data (LMSYS Arena, LiveBench, Aider polyglot) with a small self-run eval set built from your actual task patterns (docstring quality, refactor correctness, doc-formatting) — the self-run set is more work but far more predictive of real routing quality than a general leaderboard number.

Alert (MQTT topic, matching vigilar's pattern) on: price change beyond some threshold, model deprecated/removed, or a row going stale — these are the silent-failure modes that break a router without it noticing.

6. Escalation path

More important than the initial routing decision. Needs to be cheap and low-friction:

• Downstream agent/human can flag "wrong tier" → bumps the task one tier and retries
• Local verify step (optional): before accepting cheap-tier output, local model does a fast sanity pass; on failure, auto-escalate rather than surfacing bad output
• Track escalation rate per task-type — a category that escalates often is a signal the classifier's tier mapping needs adjusting, not that escalation is broken

7. Metrics to track (not just savings)

• $ saved vs. always-frontier baseline
• kWh / carbon g CO2eq per task-batch (Neuralwatt gives this natively)
• Escalation rate (overall and per task-type)
• Silent failure rate — cheap-tier output accepted but later found wrong (harder to measure, worth a periodic manual audit sample)

A router that's cheaper but quietly worse isn't a win — savings and error rate need to be reported together.

8. Open questions

• How much of "required_context" should the local classifier assemble itself vs. defer to a separate RAG step? (Leaning: keep these decoupled — context assembly is its own mini-project.)
• Self-run eval set: what's the minimum viable set of representative tasks to start scoring proficiency meaningfully, without it becoming a maintenance burden on its own?
• Whether eco_score should factor in grid carbon intensity at request time (Neuralwatt exposes real-time grid data) vs. a stable per-model average — real-time is more accurate but adds volatility to routing decisions that may not be worth it for marginal gains.

9. Next build steps

1. Poller: Neuralwatt → SQLite decision table done, verified against the live API
2. Minimal FastAPI dispatcher: reads table, applies weighted score, calls provider, returns done — /health, /route (dry run), /dispatch
3. Local classifier prompt/schema done — the allowed category list is injected from config at call time, since the model otherwise invents labels that join against nothing
4. Re-base the cost axis done (§4.2) — cost and eco now read measured billed USD and gCO2eq rather than list price
5. Seed energy_observations with a fixed-prompt sweep done — seed_energy.py
6. Populate proficiency — the last inert axis. 0.4 of the composite weight does nothing, and task_category cannot change a routing decision until it does
7. Wire into one real workflow (coding agent subagent spin-off) end to end before generalizing