6krrt/README.md

Local LLM Model Router

A router that uses a local model (served via Ollama) to classify incoming coding/documentation tasks — category, tier, required context size — and dispatch each task to the cheapest/best-fit open-weight model on Neuralwatt Cloud, weighted by cost, per-category proficiency, and a per-request energy ceiling.

The local model does the classifying, so it wants a GPU, but not necessarily your GPU — classifier.base_url takes any OpenAI-compatible endpoint, so the usual shape is the router on your laptop and Ollama on a workstation across a VPN. See Setup.

Numbers in this README are measurements, not specifications. They come from one deployment against one provider account, and the catalog, prices, grid intensity and pool load all move. They are here because the reasoning behind a design choice is worth more than the choice, and re-running the measurement is how you check whether it still holds for you.

Neuralwatt is the only provider. The provider column and the (model_id, provider) primary key stay so a second provider can be added later without a migration.

At a Glance

Dimension	Detail
Cost model	Per-kWh, not per-token. Flat $8.00/kWh measured across the catalog.
Latency	Classification is the floor: ~5-15 s warm on a local reasoning model, up to ~120 s cold. A cloud classifier measured ~1 s. Verification is async and never blocks the response.
Quality	Quality is the objective; cost is a per-request kWh ceiling plus a tiebreak. Eco is logged but no longer optimized. Proficiency blends external benchmarks with a self-run eval harness.
Verification	Every response structurally checked (free) + local LLM spot-check on long prose answers (~6 s, never blocks). Failures fold back into proficiency via feedback.py.
Fault tolerance	Classifier failure degrades to a mid-tier fallback rather than 502/503. SDK retry count is zero, to prevent timeout_seconds silently becoming a 3× wall-clock bound.
API surface	OpenAI-compatible /v1 endpoints, streaming chunk proxy with SSE telemetry scraping.

Verification Pipeline

Every completion passes through a two-layer check before routing learns from it and without adding to client-facing latency for the async layer.

Structural Verification — verification.py

Free, exact checks on what a model just returned — no code execution. Extracts fenced code blocks and validates them by parsing, not running:

Language	Check	Safeguard
Python (python, py, python3)	ast.parse	Syntax tree, no evaluation
JSON (json, jsonc)	json.loads	Strict parse
YAML (yaml, yml)	yaml.safe_load	No arbitrary objects
Shell (bash, sh, zsh)	bash -n (caller's job)	Parse-only, never runs
Other	unverifiable (not a failure)	Prose answers land here

Plus two universal checks:

• finish_reason == "length" — decisive truncation, even if parsed content looks fine (the most dangerous case: a fragment that parses)
• Unterminated code fences — the response ran out of tokens mid-block

Key design decisions:

• Never executes model output. The eval harness (eval_proficiency.py) does run generated code, but there the prompts are ones this project authored, so what comes back is bounded. Here the code is whatever the user asked for and could do anything.
• unverifiable is not a failure. Most prose answers land there (correctly — there's nothing structural to check), and counting it as wrong would penalize models for the checker's limits.
• Measured on real traffic: a wasted cloud completion costs about 52 local checks at 1,500 tokens and 139 at 4,000. A zero-cost check pays trivially.

Local LLM Verification — verification.py (cont.)

For answers nothing structural can judge (prose, reasoning, refusals), a local model spot-checks the response after it has gone back to the client, so its ~6 s never lands on anyone's latency.

The prompt frames the local model as a judge: "Does this answer actually address the user's request?" and returns {"ok": true|false, "reason": "..."}.

Gating and safeguards:

• Size threshold — only responses ≥ 600 completion tokens trigger a local check (configurable via verification.min_completion_tokens). A check on a 193-token answer costs ~15% of the answer; at 1,500 tokens the break-even failure rate drops to ~1.9%.
• Middle elision — if the answer is longer than the limit (default 4,000), the middle is elided, not the head, so the checker judges the real ending. An elision marker tells the checker not to flag it as a defect.
• Thinking disabled — the local model is a reasoning model and will otherwise emit unbounded chain-of-thought. Verification uses Ollama's native endpoint with think=False, so it answers in ~25 tokens instead of burning through the budget on reasoning.
• Malfunction safety — an empty or unparseable verdict produces no sample rather than a false failure. An empty response is settled by an if-statement, never asked of a model what code can decide.

Feedback Loop — feedback.py

Observation → learning. feedback.py folds verification failures into proficiency so routing improves on your traffic, not just the fixed 23-task benchmark:

python feedback.py --dry-run     # preview what would change
python feedback.py               # apply

Key behaviors:

• Only failures are folded in. A structural 'ok' means the code parsed, not that it was correct — scoring a parse-success as a proficiency 1.0 would flatten every score toward the ceiling. The coding categories already sit at 1.00 for every model; this would spread that flatness everywhere.
• One 0.0 sample per failure, added to the running mean so the penalty scales with failure rate rather than replacing the benchmark score outright.
• Idempotent — applied_at marks consumed rows so re-running cannot penalize a model repeatedly for the same response.
• Not the model's fault — a client that sets max_tokens=40 and gets a truncated answer caused that itself. Those failures are recorded in the verifications table but excluded from proficiency feedback.

Architecture

                    ┌─────────────────────┐
  incoming task ───▶│  Local Classifier    │  Ollama (mistral-nemo:12b)
                    │  - task_category     │  ~4s warm, ~120s cold cap
                    │  - task_tier         │  temperature: 0, max 1024 tokens
                    │  - required_context  │  max_retries: 0 (silent 3× cap guard)
                    │  - confidence        │
                    └──────────┬───────────┘
                               ▼
                    ┌─────────────────────┐
                    │ Classifier Fallback  │  tier 2 / general_chat on failure
                    │ (graceful degrade)   │  — not a 502/503
                    └──────────┬───────────┘
                               ▼
                       ┌─────────────────┐
                       │ Escalation       │  low-confidence tier bump
                       │ (optional)       │  threshold: 0.6 confidence
                       └────────┬─────────┘
                               ▼
                    ┌─────────────────────┐
                    │ Hard Filters         │  context window ≥ required
                    │ (routing.py)         │  tier ≥ required
                    │                      │  freshness (active, not stale)
                    │                      │  access_level allowed
                    │                      │  latency_class compatible
                    └──────────┬───────────┘
                               ▼
                    ┌─────────────────────┐
                    │ Quality-first select │  max proficiency, cheapest
                    │ (routing.py)         │  among equals, under a
                    │                      │  per-request kWh ceiling
                    └──────────┬───────────┘
                               ▼
                     ┌──────────────────────┐
                     │ Dispatcher / Provider │──▶ Neuralwatt Cloud
                     │ (FastAPI API)         │──▶ OpenAI-compatible /v1
                     │                       │    Streaming SSE + energy scrape
                     └───────┬──────────────┘
                             ▼
               ┌─────────────────────────┐
               │  Verification Pipeline   │
               │  Layer 1: Structural     │  ast.parse, json.load, yaml.safe_load
               │  Layer 2: Local LLM      │  async, ~6s, >600 token gate
               └──────┬──────────────────┘
                      │
                      ▼
               ┌─────────────────────┐
               │  feedback.py        │  failures → proficiency updates
               │  (on-demand agent)   │  idempotent, attributed-only
               └─────────────────────┘

Tech Stack

Layer	Technology
Language	Python 3.10+ (IO-bound provider APIs; iteration speed matters more than raw speed)
Framework	FastAPI + uvicorn
Database	SQLite (router.db) — decision table, energy observations, proficiency, verifications
Local Classification	Ollama, OpenAI-compatible — localhost:11434/v1 or an Ollama across your VPN
Local Model	classifier.model — mistral-nemo:12b by default; any Ollama model works
Cloud Provider	Neuralwatt only
Config	config.yaml loaded & validated by Pydantic (config.py)
OpenAI Client	openai==3.0.0 (official SDK)
HTTP	requests for poller, httpx (via openai/uvicorn)
Testing	pytest — 356 tests across 19 files, all offline
Config Files	config.yaml, leaderboards.yaml, evals/tasks.yaml
Deployment	systemd user units (.service + .timer files in deploy/)
Integration	opencode.json in the repo routes through it by default; any OpenAI-compatible client works

Pinned Dependencies (requirements.txt)

pyyaml==6.0.3
pydantic==2.13.4
requests==2.34.2
fastapi==0.141.1
uvicorn[standard]==0.52.1
openai==3.0.0
python-dotenv==1.2.2
pytest==9.1.1
pytest-cov==7.1.0

Pinned intentionally. Recreating the venv with >= constraints silently jumped openai 2.53 → 3.0 and httpx → httpx2 without warning; a service that restarts on boot shouldn't change its dependency tree underneath itself.

Modules

Module	Role	I/O?
dispatcher.py	FastAPI service: routes, calls providers, logs, streams	Yes (DB, network)
poller.py	Fetches Neuralwatt catalog, normalizes, upserts models table	Yes (network, DB)
scoring.py	normalize_inverted + composite_score weighted formula	Pure
routing.py	Hard filters (select_candidates) + ranking (rank_candidates)	Pure
tiering.py	Pure tier resolver: 1=cheap, 2=mid, 3=frontier	Pure
tier.py	DB tiering pass: reads models, resolves, writes tier column	Yes (DB)
proficiency.py	Score blending: leaderboard + self-eval → weighted composite	Pure
proficiency_store.py	DB access for proficiency table; single write path ensuring blended_score never drifts	Yes (DB)
seed_energy.py	Reference workload sweep: fixed prompt × N runs per model	Yes (network, DB)
eval_proficiency.py	Self-eval harness: 4 scoring kinds against N models	Yes (network, DB, subprocess)
verification.py	Two-layer check: structural parse (always) + local LLM spot-check (async, size-gated). Never executes model output	Pure
feedback.py	Folds observed verification failures into proficiency; routing learns from real traffic	Yes (DB)
leaderboard.py	Imports leaderboards.yaml priors into proficiency; --check reports gaps	Yes (DB)
iteration.py	Retry budget per tier, and matching the retry to the failure kind	Pure
config.py	YAML loader + Pydantic validators (blend weights sum to 1, valid tiers, endpoints separately addressable)	Yes (file)
metrics.py	Read-only aggregations for /health and GET /metrics: quota burn, coverage, recent decisions, per-model totals, verdict mix, top proficiency	Yes (DB)
tui.py	Textual terminal dashboard over GET /metrics; foreground tool, not a service	Yes (network)
router_cli.py	One-shot routing probe: POSTs to /route and prints the decision tree	Yes (network)

Decision Table Schema (SQLite)

Three data tables plus one observability table, PRAGMA foreign_keys = ON:

models — one row per served model variant

Column	Type	Notes
model_id	TEXT	Full catalog id (e.g. glm-5.2-short-fast-flex)
provider	TEXT	neuralwatt
base_model_id	TEXT	Model family (e.g. glm-5.2). Proficiency/leaderboard keys here.
display_name	TEXT	Human-readable name
cost_per_1m_prompt	REAL	Listed USD per 1M input tokens
cost_per_1m_completion	REAL	Listed USD per 1M output tokens
cost_per_1m_prompt_cached	REAL	Cached prefix price (null if no cache discount)
context_window	INTEGER	Advertised max tokens
effective_context_window	INTEGER	advertised × safety_factor − reserve
max_output_tokens	INTEGER
tier	INTEGER	1–3, set by tier.py pass
supports_tools	INTEGER	Boolean 0/1
supports_json_mode	INTEGER
supports_vision	INTEGER
supports_reasoning	INTEGER	"API accepts reasoning param" — NOT a quality signal
reasoning_default_enabled	INTEGER	The actual tier-bearing signal
latency_class	TEXT	standard | flex (-flex: discounted async, held during peak)
reasoning_mode	TEXT	default | reduced (-fast: reasoning capped)
context_variant	TEXT	full | short (-short: 200K pool bounded budget)
access_level	TEXT	public | preview | canary
pricing_tbd	INTEGER
deprecated	INTEGER
availability	TEXT	active | deprecated | stale
last_updated	TEXT	ISO8601

Serving class: Neuralwatt ships ~6 base models as 19 catalog rows. The id suffixes are three orthogonal dimensions (glm-5.2-short-fast-flex), parsed by poller.parse_serving_class into columns. Rows carry identical catalog pricing, so routing would pick between them arbitrarily without these — the latency_tolerance hard filter resolves it.

Access gating: 6 of 19 rows are prose-gated ("Private preview (grant-gated)", "(Canary)"). poller.parse_access_level parses them into access_level and routing.allowed_access_levels (default [public]) excludes them, so dispatch won't earn a 403.

proficiency — one row per (model, provider, category)

Column	Type	Notes
model_id	TEXT
provider	TEXT
category	TEXT	See category list below
leaderboard_score	REAL	0–1, from external benchmarks
self_eval_score	REAL	0–1, from self-eval harness
self_eval_samples	INTEGER	Evidence count for blending threshold
blended_score	REAL	Computed: w_lb × lb + w_se × se
source	TEXT	blended | self_eval | self_eval_thin | leaderboard
inherited_from	TEXT	Model this row was copied from, NULL if measured directly
last_updated	TEXT	ISO8601

Category set (9 categories, defined in config.yaml):

Category	Example	Scoring type
coding_general	Merge intervals, parse semver, word wrap	Code (execution)
coding_refactor	Remove repetition, refactor dispatch chain	Code (execution)
debugging	Fix closure leak, fix binary search, fix regex	Code (execution)
reasoning_math	Percent trap, rate trap, counting	Exact match
tool_use_agentic	Right tool / right args / no tool when empty	Structural
docs_writing	Docstring quality, must-mention gotchas	Judge
summarization	Root-cause isolation, buried-lede identification	Judge
translation	Technical register, hedging/informal tone	Judge
general_chat	Simple explanations, measured pushback	Judge

Scoring kinds (4 types, objective wherever the category admits it):

• code — runs model-generated Python in a subprocess, scores fraction of checks passing
• exact — normalizes & compares a single answer
• tool — structural: which tool was called, which args matched
• judge — a strong model scores against a rubric (prose categories only)

Blending: Once self_eval_samples ≥ self_eval_min_samples (default 10): blended = 0.3 × leaderboard + 0.7 × self_eval. Before that, falls back to leaderboard alone. If neither exists, the row scores neutral 0.5.

Source labels distinguish confidence levels: self_eval_thin means real measurement but below the sample threshold — a caller wanting to exclude it can.

Proficiency inheritance: propagate_to_variants copies evaluated scores to equivalent serving variants (same weights, same reasoning setting, same context pool), but never over a row that was measured directly. A -fast row is not equivalent to its -standard sibling.

"Measured directly" is inherited_from IS NULL, not self_eval_samples > 0 — inheritance copies the sample count too, so sample count alone cannot tell an inherited row from a measured one, and using it meant a variant inherited exactly once and then froze forever. ensure_columns() adds the column and backfills provenance on databases that predate it.

energy_observations — per-request telemetry

Column	Type	Notes
id	INTEGER	Autoincrement
model_id / provider	TEXT
task_category	TEXT
prompt_tokens / completion_tokens	INTEGER
energy_kwh	REAL	Attributed billed figure (noisy, 20× within-model)
energy_btu	REAL	kwh × 3412.14, dashboard value
avg_power_watts / duration_seconds	REAL	Pre-attribution product, ~1.8× within-model
attribution_ratio	REAL	Request's share of shared GPU pool (stable quantized: 0.001, 0.25, 0.5, 0.75)
carbon_g_co2eq	REAL	Reported by provider
grid_carbon_intensity	REAL	gCO2/kWh at call time
grid_id	TEXT	e.g. FI
carbon_source	TEXT	static_fallback (constant, excluded from routing) or live measurement
cost_usd	REAL	Billed figure, not tokens × list price
allowance_remaining_usd	REAL
service_tier	TEXT	As billed
observed_at	TEXT	ISO8601

Attribution noise: Billed energy_kwh = avg_power_watts × duration × attribution_ratio. The attribution term looks like noise up close (8 identical calls varied 20×), but ranks 750× between models while within-model spread is 1.8× — it's a stable per-model property reflecting serving concurrency. Routing scores on the attributed figures with a median over all seed_reference rows, so repeated sweeps accumulate into a median-across-time.

verifications — response quality observations

Column	Type	Notes
id	INTEGER	Autoincrement
model_id / provider	TEXT
task_category	TEXT	Optional (prose answers may lack a category)
kind	TEXT	structural | local_llm | client_outcome
verdict	TEXT	ok | truncated | malformed | unverifiable | succeeded | failed
detail	TEXT	Human-readable reason
completion_tokens	INTEGER	Wasted answer cost, for payoff sum
observed_at	TEXT	ISO8601
applied_at	TEXT	Set by feedback.py when folded into proficiency
model_attributable	INTEGER	1 = model's fault; 0 = client caused (e.g. tight cap)

Verifications drive the feedback loop: feedback.py reads unanswered failures, applies a 0.0 sample per failure to proficiency, and marks them applied_at for idempotency. unverifiable is recorded but not treated as a failure — it means the checker had nothing to say, not that the model failed.

route_decisions — routing observability

Column	Type	Notes
id	INTEGER	Autoincrement
observed_at	TEXT	ISO8601, UTC
kind	TEXT	route | dispatch | chat | passthrough | local_vision
task_category	TEXT
task_tier	INTEGER	1–3
required_context_tokens	INTEGER
confidence	REAL	Classifier confidence
classifier_ms	INTEGER	Classification latency
classification_source	TEXT	classifier | override | fallback
latency_tolerance	TEXT	interactive | batch
candidates_considered	INTEGER	How many survived hard filters
selected_model	TEXT	Null when no model was selected
selected_provider	TEXT	neuralwatt or local (vision fallback)
runner_up_models	TEXT	JSON array of up to 3 runner-up candidates
est_cost_usd	REAL	Estimated cost of the selected model
est_proficiency	REAL	Estimated proficiency for the task category
rejected_reason	TEXT	Active filters when nothing was selected
session_key	TEXT	Hashed session fingerprint ONLY
tools	INTEGER	0/1 — request carried a tools array
images	INTEGER	0/1 — request carried image parts
json_mode	INTEGER	0/1 — request required JSON mode
streamed	INTEGER	0/1 — response was streamed

route_decisions stores one row per routing decision so "how is routing performing" is answerable: which model was picked, for what category/tier, how long classification took, and — when nothing was selected — which hard filter shut it out. It is an observability table: nothing in routing reads it.

It stores only a hashed session fingerprint in session_key; session_dir, prompts, and answers are deliberately excluded. A test enforces that the write path does not store prompt or answer text. The write is gated by logging.log_route_decisions and is best-effort: a failed write is logged at warning and swallowed so monitoring cannot slow or fail a request.

Weighted Scoring

Six hard filters are applied before scoring (not weighted — outright disqualification):

1. effective_context_window ≥ required_context_tokens
2. tier ≥ required_tier (from classifier)
3. Serving class compatible with request's latency_tolerance; access_level reachable
4. tool_use_agentic proficiency ≥ routing.min_tool_proficiency, only when the request carries a tools array (ships disabled — null)
5. supports_vision = 1 when the request carries image_url parts; NULL fails closed (an unknown flag means the capability cannot be confirmed)
6. supports_json_mode = 1 when response_format.type is json_object or json_schema; NULL also fails closed

Filters 4–6 are read from the request body rather than inferred. A tools array states whether tool definitions are on the table, image_url parts state whether vision is needed, and response_format states whether JSON mode is needed. Local classifiers identified unambiguous tool-use prompts only 1-2 times in 6, so inferring capability requirements does not work; the request states them exactly and for free.

The tool gate is a quality measurement filter: a model with no measured tool_use_agentic score is admitted, because "unproven" is not "proven bad". Only a measured score below routing.min_tool_proficiency is dropped. The vision and JSON-mode gates are capability flag filters. There every routable catalog row has supports_tools = 1, so a flag gate would be inert; vision and JSON mode are not universal, and an absent or NULL flag means "cannot confirm the capability", so the model is dropped. A wrong guess on a capability flag is a guaranteed provider 400.

The tool filter ships off (null), because agent clients send tools on nearly every request, so enabling it excludes the cheapest model from ordinary agent traffic. Whether that trade is worth it is an empirical question best settled with POST /outcome data rather than a 3-task benchmark score. Set it to 0.5 to turn it on. The vision and JSON-mode gates ship on, because a wrong guess is a guaranteed 400.

Local Vision Fallback

Cloud vision is not universal in the catalog, and the most economical coding rows do not declare supports_vision. Routing an image request through them would earn a provider-side 400, so when routing.require_vision is true only rows with supports_vision = 1 survive the hard filters. If no cloud candidate survives, the router can fall back to a local vision model instead of returning 422.

local_vision: in config.yaml controls this path:

Key	Default	Purpose
enabled	true	Whether the fallback runs at all
base_url	http://localhost:11434/v1	OpenAI-compatible Ollama endpoint
api_key_env	null	Env var holding an API key, if the endpoint needs one
model	qwen3-vl:4b	Vision model on that Ollama
timeout_seconds	60	Request timeout
max_images	4	Refuse requests with more image parts
max_image_bytes	9437184 (9 MiB)	Refuse requests whose image payload exceeds this

The fallback is enabled by default both in config.yaml and in LocalVisionConfig, so omitting the section still turns it on. Disable it explicitly (enabled: false) on a host with no local Ollama or one that has not pulled the vision model.

When enabled and no cloud candidate is selected, _run_local_vision sends the original message list — with image_url parts intact — to the configured Ollama model. The local answer then replaces the cloud completion: _local_vision_response returns a normal OpenAI-shaped response, including a stream-wrapped version when the client asked for stream: true. It does not inject a caption into a cloud call, because the streaming proxy cannot rewrite bytes mid-stream.

Security and budget guards:

• Only inline data: URIs are accepted. A remote http(s) image URL is declined, because pointing a local model at an arbitrary URL would let an unauthenticated caller make the router fetch internal resources (SSRF).
• Image count and total payload size are bounded by max_images and max_image_bytes before the local call is made.
• If the local call fails for any reason, the request falls through to the ordinary 422 No model satisfies the hard filters rather than returning a silent empty response.

Pull the model on whichever Ollama the fallback points at:

ollama pull qwen3-vl:4b

Config is strict (extra="forbid"): a misspelled or misplaced key fails at load instead of being silently ignored.

1. drop candidates whose measured energy exceeds objective.max_energy_per_request
2. rank by proficiency for the task's category
3. treat differences smaller than objective.quality_tolerance as equal
4. among equals, pick the cheapest

Setting	Value	Notes
quality_tolerance	0.10	Measurement noise, not preference: scores rest on 2-3 samples, so smaller gaps are sampling variation
assumed_cache_rate	0.917	Share of prompt tokens served from the provider's prefix cache. Agent clients resend the conversation each turn, so most of it hits. Measured token-weighted over 40.7M tokens; measure your own against the provider's session view
assumed_completion_tokens	500	Completion length assumed when pricing a candidate
max_energy_per_request	null	Per-request kWh ceiling — a wall, not a bill. Null disables it
plan_kwh_per_period	6.25	Set this to your own plan's quota. Reported in /health as burn against the allowance; it does not gate anything

This replaced a weighted blend (cost 0.4 / eco 0.2 / proficiency 0.4). Measurement retired it: turning the cost weight from 0.4 to zero changed the winner in only 2 of 6 categories, so the blend was never steering on quality — while 60% of every decision adjudicated fractions of a cent.

Cost is priced per request, from catalog token prices scaled to the request's shape — prompt size, assumed_completion_tokens, assumed_cache_rate. It is deliberately not a benchmark average.

Neuralwatt bills flat per-kWh rather than per-token, so list price is not what gets charged. Scoring used measured billed cost from a fixed 400-token reference sweep for exactly that reason — and that was wrong for real traffic, because the ranking depends on the workload's shape, not just the model. On a 400-token prompt one model looked 3.2× cheaper than another; on a realistic 70k-token prompt the same pair inverted and the second was 5.0× cheaper. The provider's attribution ratio moves with prompt size, so a fixed-shape benchmark cannot rank models for a workload of a different shape.

List price is still not what is billed, but billing is capped at a multiple of it, so it tracks the real ordering and bounds it — and it is free, needs no sweep, and refreshes whenever the poller runs.

Why cost ≠ eco: Cost tracks energy (kWh), but carbon is energy × grid intensity. Grid intensity spanned ~49 gCO2/kWh (FI) to ~442 (US-MIDA-PJM) when measured — and it moves with time of day. The models disagree: glm-5.2-fast is 2nd cheapest but 6th cleanest; kimi-k3-flex draws 3.7× less energy than kimi-k2.7-code while emitting 3.6× more carbon. Collapsing them picks a side.

What routing actually returns, and why it moves

Sweeping /route across every category and tier is the fastest way to see whether your data is doing anything. On the deployment this was written against, 9 categories × 3 tiers currently yields 5 distinct winners (qwen3.6-35b, gemma-4-31b, deepseek-v4-flash, kimi-k3, kimi-k3-fast).

That number is a diagnostic, not a target, and it is worth knowing what each outcome means:

• One winner everywhere is a legitimate answer, not a misconfiguration. It happened here: with cost, eco and proficiency all populated, one model was Pareto-dominant — cheapest and cleanest in the routable set while scoring within quality_tolerance of the best. No defensible weighting picks anything else. If you see this, check whether the leader really is dominant before reaching for the config.
• Winners that change with context size are the hard filters working. A model is dropped once required_context_tokens exceeds its window, so a long session can change model mid-conversation. Past the largest window, /v1/chat/completions returns 422 naming the constraint rather than silently truncating.
• Winners that change by category mean proficiency is live. That is the only category-dependent term, so until the proficiency table has data, task_category cannot change a decision at all — the classifier computes it and the router pays for it for nothing.

The spread here widened for two reasons worth copying: cost became a per-request estimate rather than a benchmark average, and tier stopped being inferred from price alone. Both had been quietly excluding a cheap large-context model from every request above tier 1.

If you want a different balance, the levers are objective.quality_tolerance (how big a quality gap must be before it outranks cost) and objective.max_energy_per_request (a hard ceiling). There is no weight to tune — quality is the objective and cost is the tiebreak, which replaced an earlier weighted blend.

API Endpoints

The dispatcher binds 127.0.0.1:8080. No auth of its own — loopback is the only thing standing between the open internet and your billing allowance.

Method	Path	Description
GET	/health	Catalog/reachability status, scoring coverage, warnings
GET	/metrics	Aggregated observability JSON: quota burn, coverage, recent decisions, per-model totals, verdict mix, top proficiency; loopback-only, no auth
POST	/route	Classify task, rank candidates, return selected model — no provider call, no cost
POST	/dispatch	Same as /route, plus complete the provider call, stream response, log observation
GET	/v1/models	OpenAI-compatible model list (router virtual models + catalog)
POST	/v1/chat/completions	OpenAI-compatible completions — routes then proxies, streaming supported

/metrics returns a single JSON object with these top-level keys:

• quota — kWh metered in the last 30 days against objective.plan_kwh_per_period
• coverage — routable-model counts with energy/proficiency data plus warnings
• recent_decisions — last 50 rows from route_decisions
• per_model — per-model aggregates over the last 30 days of energy_observations
• verdict_mix — counts by verification verdict over the last 7 days
• top_proficiency — top models by blended_score for coding_general
• generated_at — ISO8601 timestamp

It exposes no conversation text, prompts, or session_dir; it is bound to loopback and unauthenticated exactly like /health.

Input to /route and /dispatch can include task_category, task_tier, and required_context_tokens overrides — these skip the classifier, useful for testing routing without the classifier in the loop.

Two virtual router models:

• auto — normal routing, -flex rows excluded (interactive)
• auto:batch — admits -flex rows (overnight/async work)

Ask for any real model id in /v1/chat/completions and it dispatches directly, still logged — routing is transparent, not opaque.

Streaming (chunk-by-chunk proxy): Tokens render as they arrive. Neuralwatt emits energy and cost as SSE comment lines (: energy {...}) before data: [DONE] — ordinary clients ignore comments, so the stream flows untouched while the router scrapes telemetry on the way past. Without this, streamed calls would log no energy at all.

Verification headers (non-streaming): When streaming is not used, the structural verification verdict surfaces in the X-Router-Verification header so a client can inspect it without parsing the response body. Valid values: ok, truncated, malformed, unverifiable, none.

Capability 422s: When no model survives the hard filters, the 422 names the active constraints. That now includes "vision-capable model" or "json-mode-capable model" when the request carried images or a JSON-mode response_format, alongside the existing context/tier/latency/tool reasons.

Quick Usage

# Pick a model without spending anything
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
  -d '{"task":"Refactor this 800-line Django view into service objects."}'

# With explicit category/tier overrides (skips classifier)
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
  -d '{"task":"Refactor this 800-line Django view","task_category":"coding_refactor","task_tier":2}'

# Actually call the winner and log energy/cost
curl -s -X POST localhost:8080/dispatch -H 'content-type: application/json' \
  -d '{"task":"What is a Python context manager?"}'

# OpenAI-compatible — works with any SDK or agent client
curl -s localhost:8080/v1/models
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'

# Admit flex rows for overnight/async work
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
  -d '{"task":"nightly code review","latency_tolerance":"batch"}'

# Image URL request (routed only to vision-capable catalog rows)
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":[{"type":"text","text":"Describe this"},{"type":"image_url","image_url":{"url":"data:image/gif;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs="}}]}]}'

# JSON-mode request via response_format
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"Return a JSON object with field answer"}],"response_format":{"type":"json_object"}}'

# Monitoring: aggregate router health
curl -s localhost:8080/metrics | python -m json.tool

# Terminal dashboard foreground tool (requires `textual`; runs until you press q)
python tui.py

# One-shot routing probe with no spend
python router_cli.py "Refactor this Django view into service objects"
python router_cli.py "Summarize this diff" --category summarization --tier 2

Monitoring

Three foreground tools read the running router without spending quota:

• GET /metrics — JSON summary of quota, coverage, recent routing decisions, per-model usage, verdict mix, and top proficiency. No auth; loopback only.
• tui.py — Textual terminal dashboard that polls /metrics every few seconds. Run it in a terminal with the service already up. It is a separate entrypoint, not a systemd unit.
• router_cli.py "<task>" — POSTs to /route once and prints the full decision tree, including candidates, selected model, estimated cost, and proficiency. Use --category, --tier, and --context to override the classifier deterministically, or --json for raw output.

textual is pinned in requirements.txt solely for tui.py. It is imported only by that module; the FastAPI service dispatch path never touches it, so the router itself has no UI dependency.

Logging and Traceability

One route line per request says what was decided; one dispatch line says what it cost. Both carry a trace id, and rid/sess are columns in energy_observations, so a journal line pivots to its database row and back.

route id=r9116d9 cat=coding_refactor tier=2 ctx=500 src=classifier tools=0
  latency=interactive cand=8 pick=deepseek-v4-flash est_usd=0.00016296 ms=1868
dispatch id=r9116d9 model=deepseek-v4-flash rid=chatcmpl-... sess=c090d751
  p_tok=59511 c_tok=415 kwh=4.8e-05 usd=0.000783 verdict=ok total_ms=5692

At debug it also says why — the classifier's answer and timing, every candidate that was dropped and by which filter, and the ranking with scores:

classify id=r9116d9 cat=coding_refactor tier=2 confidence=0.95 ms=1863
filter   id=r9116d9 model=gemma-4-31b reason=tier(1<2)
filter   id=r9116d9 model=glm-5.2-short reason=access_level(preview)
filter   id=r9116d9 model=kimi-k3-flex reason=latency_class(flex)
rank     id=r9116d9 pos=0 model=deepseek-v4-flash prof=1 est_usd=0.00016296

No conversation text is logged at any level, prompt or answer — prompts here run 60k–150k tokens and the journal is on disk. A test enforces it.

Set the level in config.yaml (logging.level), or override it without touching a tracked file:

systemctl --user edit llm-router      # Environment="LLM_ROUTER_LOG_LEVEL=debug"
systemctl --user restart llm-router

Reading it back:

journalctl --user -u 'llm-router*' -f            # quote the glob in zsh
journalctl --user -u llm-router -f -o cat        # message only
journalctl --user -u llm-router -p warning       # severity actually filters
journalctl --user -u llm-router --grep 'chatcmpl-abc123'   # one request's trail
journalctl --user -u llm-router --grep ' id=r9116d9'       # every stage of it

-p works because the service emits journald priority prefixes when — and only when — systemd owns its stderr. Run uvicorn in a terminal and the lines come out clean.

Scheduled Jobs (systemd)

Five user units. See deploy/README.md for full install/operate instructions.

Unit	Schedule	What it runs
llm-router.service	Continuous	FastAPI dispatcher
llm-router-poller.timer	2 min after boot, then every 2 h	triggers the poller unit
llm-router-poller.service	oneshot	poller.py → tier.py
llm-router-seed.timer	Every 6 h	triggers the seed unit
llm-router-seed.service	oneshot	small seed_energy.py sweep

The poller timer is load-bearing, not optional. freshness.stale_after_days is 3 with exclude_stale: true — an unpolled catalog marks every row stale in 3 days and the router returns zero candidates for everything.

The seed timer spans time. Energy attribution drifts with pool load across hours (~50× for one model between sweeps), so a single sweep measures one moment. The median has to span time — every 6 h sweep accumulates into a median-across-time automatically.

This now feeds eco only. Cost is priced per request from catalog prices, so routing no longer depends on the sweep at all; disabling this timer costs you carbon figures, not routing quality.

Self-Eval Harness (eval_proficiency.py)

python eval_proficiency.py                      # every routable model × every task
python eval_proficiency.py --models kimi-k3     # subset of models
python eval_proficiency.py --categories coding_general
python eval_proficiency.py --dry-run            # plan only

• Runs every task through the target provider, scores it, writes to proficiency.
• Scores accumulate (running mean), so repeated runs tighten estimates.
• propagate_to_variants copies evaluated scores to equivalent serving variants.
• Safety: code tasks run in a temp directory with a 15s wall-clock timeout — bounded isolation, not a container.
• Judge tasks skip the model being judged (avoids self-scoring bias).

Classifier Reliability Notes

The classifier is the one blocking LLM call on the request path, so it sets the latency floor for every routed request. classifier.base_url / api_key_env / model accept any OpenAI-compatible endpoint — a local Ollama, an Ollama on another machine across a VPN, or a cloud model.

Pick a non-reasoning model. The classifier's entire output is ~45 tokens of JSON, and a reasoning model spends its budget getting there. Replacing qwen3.5 with mistral-nemo:12b on identical prompts, cold load excluded:

	qwen3.5	mistral-nemo:12b
category correct	10/14	9/14
tier correct	1/5 (2 crashed)	3/5
hard failures	4 of 19 calls	0
latency mean / max	6.6s / 15.6s	1.7s / 1.9s

Accuracy is a wash; the failure column and the tail are not. Every failure was an unbounded thinking trace producing no JSON, which degrades to source: "fallback" after ~15s. End-to-end /route went ~10s → ~1.7s.

Several settings keep it from cascading failures:

• max_input_chars: 8000 clamps what the classifier is shown to head + tail. It decides a category and a tier; it does not need the document, and feeding it one is harmful rather than wasteful — on a ~20k-token prompt one model returned empty after 28.7s and the other spent 41.8s echoing the input into its JSON. Clamped, the same prompts classify in ~2.2s.

• max_retries: 0 on the classifier client. The OpenAI SDK retries twice by default, so timeout_seconds silently became a 3× wall-clock bound — a request could hang past 250s on a 120s setting with no log.

• max_output_tokens: 1024 bounds a reasoning model's chain of thought. Without it, a reasoning model emits unbounded traces — Ollama keeps generating after the client gives up AND serializes per model, so one runaway request queues every later request behind it. 256 was tried first and was too tight; the trace would consume the whole budget and the model would be truncated before emitting JSON.

• temperature: 0 prevents non-reproducible classifications. At the default, the same prompt classified tier 2 then tier 1 on consecutive calls, routing to two different models.

• Graceful fallback routes to fallback_tier: 2 / fallback_category: general_chat when the classifier times out, errors, or returns unparseable output. A coding agent would rather have a mid-tier answer than an error. Escalation deliberately skips fallbacks so an unavailable local model doesn't silently promote every request to the frontier tier.

Pointing a Coding Agent at It

The /v1 endpoints are OpenAI-compatible, so any normal client works — opencode, an SDK, plain curl. A repo-local opencode.json is included, so running opencode from a clone of this repo routes through the router by default. For global use, merge its provider.llm-router block into ~/.config/opencode/opencode.json.

Virtual model names:

• auto → router picks, interactive mode (flex rows excluded)
• auto:batch → router picks, admits flex rows for async work

opencode image support: The repo's opencode.json declares "modalities": {"input": ["text", "image"]} for every llm-router model. This is required: opencode strips image parts client-side unless the provider model declares image input. Without it, images never reach the router at all. For global use, make sure the merged ~/.config/opencode/opencode.json entry carries the same modality block.

Testing

python -m pytest        # 356 tests
python -m pytest --cov  # with coverage

No test calls a provider or a local model — the pure modules take rows and config as arguments, so the suite runs offline on a clean checkout.

Test file	What it covers
test_scoring.py	normalize_inverted, composite_score, None-handling
test_routing.py	Hard filters, candidate selection & ranking across all 9 categories × 3 tiers
test_tiering.py	Tier resolver precedence: override → reasoning → cost + context window → mid
test_apply_tiering.py	DB tiering pass, sanity guards
test_poller_parsing.py	Serving class, base model, access level parsing
test_proficiency.py	Blending rule, accumulation
test_load_candidates.py	load_candidates: cost/eco/proficiency join
test_eval_scoring.py	Code, exact, tool, and judge scoring
test_task_set.py	Reference solutions validating every code task's checks & exact answers
test_verification.py	Structural checks on every language, verifier edge cases, local-LLM gating
test_feedback.py	Failure identification, idempotency, attribution filtering
test_iteration.py	Retry budget per tier, and matching the retry to the failure kind
test_session_identity.py	Outcome attribution: session matching, ambiguity refusal
test_config_endpoints.py	Classifier and verifier are separately addressable; guards on the split

Setup

You need: a Neuralwatt API key, Python 3.10+ (suite verified on 3.10 and 3.14), and an Ollama reachable from wherever this runs with a classifier model pulled. Nothing else is assumed about the host — routing itself is SQLite and arithmetic.

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
sqlite3 router.db < schema.sql
cp .env.example .env   # fill in NEURALWATT_API_KEY
python poller.py       # populate the catalog
python tier.py         # resolve tiers
python config.py       # sanity-check config loads
python -m uvicorn dispatcher:app --reload

Then edit config.yaml for your own setup — at minimum:

Key	Why
classifier.model	must match a model ollama list reports
classifier.base_url	where that Ollama actually is
objective.plan_kwh_per_period	your plan's quota; /health reports burn against it
objective.assumed_cache_rate	0.917 was measured from one client's traffic (40.7M tokens). Check yours against the provider's per-session cache-hit figures

python seed_energy.py is optional. It sweeps a fixed reference workload to populate eco, which is logged but is not an objective — routing works without it. It costs real money and quota, so it is not in the path above.

Where Ollama lives

ollama pull mistral-nemo:12b   # or whatever you set as classifier.model

It does not have to be on the machine running the router; the box with the GPU usually isn't the laptop. To use one across a VPN, point both endpoints at it:

classifier:
  base_url: "http://<vpn-ip>:11434/v1"
verification:
  base_url: "http://<vpn-ip>:11434"    # same host, so `model` can stay null

and apply deploy/ollama-over-vpn.conf on the serving host — Ollama binds 127.0.0.1 by default and will otherwise refuse. Bind it to the VPN address rather than 0.0.0.0: Ollama has no authentication, so anything reaching the port can run inference and enumerate your models.

Both endpoints move together because the verifier speaks Ollama's native API and cannot follow the classifier to a cloud provider. Config load refuses the case where they are on different hosts and verification.model is null, because that combination fails silently.

Known Limitations & Open Items

• Leaderboard priors unfilled — leaderboards.yaml ships empty. python leaderboard.py --check names every active family lacking a prior. Must source figures from published benchmarks; inventing plausible numbers is the same failure as the provider's static_fallback carbon constant.
• Three models unsettled — split-half stability at 7 samples: kimi-k2.7-code-fast (29×), kimi-k3 (14×), glm-5.2-flex (2.2×). More samples needed before their positioning is trustworthy.
• Retry does not reach streaming — a failed structural check now buys corrective attempts scaled by tier (iteration.py), but only on the non-streaming path. Once bytes have gone to the client there is nothing to take back, and buffering to allow correction would cost streaming itself. POST /outcome is the answer for streamed traffic: the client reports afterwards, so it works identically either way.
• No auth — the service holds a billable API key with no authentication of its own. It binds loopback for that reason; widening the bind needs an auth layer first. The same applies to an Ollama shared over a VPN, which also has no auth — bind it to the VPN address, never 0.0.0.0.
• Local energy is not accounted for — the router measures what the cloud provider bills and reports, but the electricity its own classifier and verifier burn on your hardware is invisible to it. Since the whole premise is spending cheap local compute to avoid wasting expensive cloud compute, that half of the ledger is currently taken on faith. Closing it means metering local draw and pricing it against a real tariff rather than assuming local is free.
• Context assembly (RAG) is out of scope — the classifier sees the full conversation but does not perform document/code retrieval.