# Local LLM Model Router

A router that uses a local model (served via Ollama) to classify incoming
coding/documentation tasks — category, tier, required context size — and
dispatch each task to the cheapest/best-fit open-weight model on **Neuralwatt
Cloud**, weighted by cost, per-category proficiency, and a per-request energy
ceiling.

The local model does the classifying, so it wants a GPU, but not necessarily
*your* GPU — `classifier.base_url` takes any OpenAI-compatible endpoint, so
the usual shape is the router on your laptop and Ollama on a workstation
across a VPN. See [Setup](#setup).

**Numbers in this README are measurements, not specifications.** They come
from one deployment against one provider account, and the catalog, prices,
grid intensity and pool load all move. They are here because the reasoning
behind a design choice is worth more than the choice, and re-running the
measurement is how you check whether it still holds for you.

Neuralwatt is the only provider. The `provider` column and the
`(model_id, provider)` primary key stay so a second provider can be added
later without a migration.

## At a Glance

| Dimension | Detail |
|---|---|
| **Cost model** | Per-kWh, not per-token. Flat $8.00/kWh measured across the catalog. |
| **Latency** | Classification is the floor: ~5-15 s warm on a local reasoning model, up to ~120 s cold. A cloud classifier measured ~1 s. Verification is async and never blocks the response. |
| **Quality** | Quality is the objective; cost is a per-request kWh ceiling plus a tiebreak. Eco is logged but no longer optimized. Proficiency blends external benchmarks with a self-run eval harness. |
| **Verification** | Every response structurally checked (free) + local LLM spot-check on long prose answers (~6 s, never blocks). Failures fold back into proficiency via `feedback.py`. |
| **Fault tolerance** | Classifier failure degrades to a mid-tier fallback rather than 502/503. SDK retry count is zero, to prevent `timeout_seconds` silently becoming a 3× wall-clock bound. |
| **API surface** | OpenAI-compatible `/v1` endpoints, streaming chunk proxy with SSE telemetry scraping. |

## Verification Pipeline

Every completion passes through a two-layer check **before** routing learns from
it and **without** adding to client-facing latency for the async layer.

### Structural Verification — `verification.py`

Free, exact checks on what a model just returned — no code execution.
Extracts fenced code blocks and validates them by **parsing**, not running:

| Language | Check | Safeguard |
|---|---|---|
| Python (`python`, `py`, `python3`) | `ast.parse` | Syntax tree, no evaluation |
| JSON (`json`, `jsonc`) | `json.loads` | Strict parse |
| YAML (`yaml`, `yml`) | `yaml.safe_load` | No arbitrary objects |
| Shell (`bash`, `sh`, `zsh`) | `bash -n` (caller's job) | Parse-only, never runs |
| Other | `unverifiable` (not a failure) | Prose answers land here |

Plus two universal checks:
- **`finish_reason == "length"`** — decisive truncation, even if parsed content
  looks fine (the most dangerous case: a fragment that parses)
- **Unterminated code fences** — the response ran out of tokens mid-block

**Key design decisions:**
- Never executes model output. The eval harness (`eval_proficiency.py`) does
  run generated code, but there the prompts are ones this project authored,
  so what comes back is bounded. Here the code is whatever the user asked for
  and could do anything.
- `unverifiable` is **not** a failure. Most prose answers land there (correctly
  — there's nothing structural to check), and counting it as wrong would
  penalize models for the checker's limits.
- Measured on real traffic: a wasted cloud completion costs about 52 local
  checks at 1,500 tokens and 139 at 4,000. A zero-cost check pays trivially.

### Local LLM Verification — `verification.py` (cont.)

For answers nothing structural can judge (prose, reasoning, refusals), a local
model spot-checks the response **after** it has gone back to the client, so its
~6 s never lands on anyone's latency.

The prompt frames the local model as a judge: "Does this answer actually
address the user's request?" and returns `{"ok": true|false, "reason": "..."}`.

Gating and safeguards:
- **Size threshold** — only responses ≥ 600 completion tokens trigger a local
  check (configurable via `verification.min_completion_tokens`). A check on
  a 193-token answer costs ~15% of the answer; at 1,500 tokens the break-even
  failure rate drops to ~1.9%.
- **Middle elision** — if the answer is longer than the limit (default 4,000),
  the **middle** is elided, not the head, so the checker judges the real ending.
  An elision marker tells the checker not to flag it as a defect.
- **Thinking disabled** — the local model is a reasoning model and will otherwise
  emit unbounded chain-of-thought. Verification uses Ollama's native endpoint
  with `think=False`, so it answers in ~25 tokens instead of burning through
  the budget on reasoning.
- **Malfunction safety** — an empty or unparseable verdict produces no sample
  rather than a false failure. An empty response is settled by an if-statement,
  never asked of a model what code can decide.

### Feedback Loop — `feedback.py`

Observation → learning. `feedback.py` folds verification failures into
`proficiency` so routing improves on **your traffic**, not just the fixed
23-task benchmark:

```bash
python feedback.py --dry-run     # preview what would change
python feedback.py               # apply
```

Key behaviors:
- **Only failures** are folded in. A structural 'ok' means the code parsed,
  not that it was correct — scoring a parse-success as a proficiency 1.0 would
  flatten every score toward the ceiling. The coding categories already sit
  at 1.00 for every model; this would spread that flatness everywhere.
- **One 0.0 sample per failure**, added to the running mean so the penalty
  scales with failure rate rather than replacing the benchmark score outright.
- **Idempotent** — `applied_at` marks consumed rows so re-running cannot
  penalize a model repeatedly for the same response.
- **Not the model's fault** — a client that sets `max_tokens=40` and gets a
  truncated answer caused that itself. Those failures are recorded in the
  verifications table but excluded from proficiency feedback.

## Architecture

```
                    ┌─────────────────────┐
  incoming task ───▶│  Local Classifier    │  Ollama (mistral-nemo:12b)
                    │  - task_category     │  ~4s warm, ~120s cold cap
                    │  - task_tier         │  temperature: 0, max 1024 tokens
                    │  - required_context  │  max_retries: 0 (silent 3× cap guard)
                    │  - confidence        │
                    └──────────┬───────────┘
                               ▼
                    ┌─────────────────────┐
                    │ Classifier Fallback  │  tier 2 / general_chat on failure
                    │ (graceful degrade)   │  — not a 502/503
                    └──────────┬───────────┘
                               ▼
                       ┌─────────────────┐
                       │ Escalation       │  low-confidence tier bump
                       │ (optional)       │  threshold: 0.6 confidence
                       └────────┬─────────┘
                               ▼
                    ┌─────────────────────┐
                    │ Hard Filters         │  context window ≥ required
                    │ (routing.py)         │  tier ≥ required
                    │                      │  freshness (active, not stale)
                    │                      │  access_level allowed
                    │                      │  latency_class compatible
                    └──────────┬───────────┘
                               ▼
                    ┌─────────────────────┐
                    │ Quality-first select │  max proficiency, cheapest
                    │ (routing.py)         │  among equals, under a
                    │                      │  per-request kWh ceiling
                    └──────────┬───────────┘
                               ▼
                     ┌──────────────────────┐
                     │ Dispatcher / Provider │──▶ Neuralwatt Cloud
                     │ (FastAPI API)         │──▶ OpenAI-compatible /v1
                     │                       │    Streaming SSE + energy scrape
                     └───────┬──────────────┘
                             ▼
               ┌─────────────────────────┐
               │  Verification Pipeline   │
               │  Layer 1: Structural     │  ast.parse, json.load, yaml.safe_load
               │  Layer 2: Local LLM      │  async, ~6s, >600 token gate
               └──────┬──────────────────┘
                      │
                      ▼
               ┌─────────────────────┐
               │  feedback.py        │  failures → proficiency updates
               │  (on-demand agent)   │  idempotent, attributed-only
               └─────────────────────┘
```

## Tech Stack

| Layer | Technology |
|---|---|
| **Language** | Python 3.10+ (IO-bound provider APIs; iteration speed matters more than raw speed) |
| **Framework** | FastAPI + uvicorn |
| **Database** | SQLite (`router.db`) — decision table, energy observations, proficiency, verifications |
| **Local Classification** | Ollama, OpenAI-compatible — `localhost:11434/v1` or an Ollama across your VPN |
| **Local Model** | `classifier.model` — `mistral-nemo:12b` by default; any Ollama model works |
| **Cloud Provider** | Neuralwatt only |
| **Config** | `config.yaml` loaded & validated by Pydantic (`config.py`) |
| **OpenAI Client** | `openai==3.0.0` (official SDK) |
| **HTTP** | `requests` for poller, `httpx` (via openai/uvicorn) |
| **Testing** | `pytest` — 356 tests across 19 files, all offline |
| **Config Files** | `config.yaml`, `leaderboards.yaml`, `evals/tasks.yaml` |
| **Deployment** | systemd user units (`.service` + `.timer` files in `deploy/`) |
| **Integration** | `opencode.json` in the repo routes through it by default; any OpenAI-compatible client works |

## Pinned Dependencies (requirements.txt)

```
pyyaml==6.0.3
pydantic==2.13.4
requests==2.34.2
fastapi==0.141.1
uvicorn[standard]==0.52.1
openai==3.0.0
python-dotenv==1.2.2
pytest==9.1.1
pytest-cov==7.1.0
```

**Pinned intentionally.** Recreating the venv with `>=` constraints silently jumped
`openai 2.53 → 3.0` and `httpx → httpx2` without warning; a service that
restarts on boot shouldn't change its dependency tree underneath itself.

## Modules

| Module | Role | I/O? |
|---|---|---|
| **`dispatcher.py`** | FastAPI service: routes, calls providers, logs, streams | Yes (DB, network) |
| **`poller.py`** | Fetches Neuralwatt catalog, normalizes, upserts `models` table | Yes (network, DB) |
| **`scoring.py`** | `normalize_inverted` + `composite_score` weighted formula | Pure |
| **`routing.py`** | Hard filters (`select_candidates`) + ranking (`rank_candidates`) | Pure |
| **`tiering.py`** | Pure tier resolver: 1=cheap, 2=mid, 3=frontier | Pure |
| **`tier.py`** | DB tiering pass: reads models, resolves, writes `tier` column | Yes (DB) |
| **`proficiency.py`** | Score blending: leaderboard + self-eval → weighted composite | Pure |
| **`proficiency_store.py`** | DB access for `proficiency` table; single write path ensuring `blended_score` never drifts | Yes (DB) |
| **`seed_energy.py`** | Reference workload sweep: fixed prompt × N runs per model | Yes (network, DB) |
| **`eval_proficiency.py`** | Self-eval harness: 4 scoring kinds against N models | Yes (network, DB, subprocess) |
| **`verification.py`** | Two-layer check: structural parse (always) + local LLM spot-check (async, size-gated). Never executes model output | Pure |
| **`feedback.py`** | Folds observed verification failures into `proficiency`; routing learns from real traffic | Yes (DB) |
| **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) |
| **`iteration.py`** | Retry budget per tier, and matching the retry to the failure kind | Pure |
| **`config.py`** | YAML loader + Pydantic validators (blend weights sum to 1, valid tiers, endpoints separately addressable) | Yes (file) |
| **`metrics.py`** | Read-only aggregations for `/health` and `GET /metrics`: quota burn, coverage, recent decisions, per-model totals, verdict mix, top proficiency | Yes (DB) |
| **`tui.py`** | Textual terminal dashboard over `GET /metrics`; foreground tool, not a service | Yes (network) |
| **`router_cli.py`** | One-shot routing probe: POSTs to `/route` and prints the decision tree | Yes (network) |

## Decision Table Schema (SQLite)

Three data tables plus one observability table, `PRAGMA foreign_keys = ON`:

### `models` — one row per served model variant

| Column | Type | Notes |
|---|---|---|
| `model_id` | TEXT | Full catalog id (e.g. `glm-5.2-short-fast-flex`) |
| `provider` | TEXT | `neuralwatt` |
| `base_model_id` | TEXT | Model family (e.g. `glm-5.2`). Proficiency/leaderboard keys here. |
| `display_name` | TEXT | Human-readable name |
| `cost_per_1m_prompt` | REAL | Listed USD per 1M input tokens |
| `cost_per_1m_completion` | REAL | Listed USD per 1M output tokens |
| `cost_per_1m_prompt_cached` | REAL | Cached prefix price (null if no cache discount) |
| `context_window` | INTEGER | Advertised max tokens |
| `effective_context_window` | INTEGER | `advertised × safety_factor − reserve` |
| `max_output_tokens` | INTEGER | |
| `tier` | INTEGER | 1–3, set by `tier.py` pass |
| `supports_tools` | INTEGER | Boolean 0/1 |
| `supports_json_mode` | INTEGER | |
| `supports_vision` | INTEGER | |
| `supports_reasoning` | INTEGER | "API accepts reasoning param" — NOT a quality signal |
| `reasoning_default_enabled` | INTEGER | The actual tier-bearing signal |
| `latency_class` | TEXT | `standard` \| `flex` (`-flex`: discounted async, held during peak) |
| `reasoning_mode` | TEXT | `default` \| `reduced` (`-fast`: reasoning capped) |
| `context_variant` | TEXT | `full` \| `short` (`-short`: 200K pool bounded budget) |
| `access_level` | TEXT | `public` \| `preview` \| `canary` |
| `pricing_tbd` | INTEGER | |
| `deprecated` | INTEGER | |
| `availability` | TEXT | `active` \| `deprecated` \| `stale` |
| `last_updated` | TEXT | ISO8601 |

**Serving class:** Neuralwatt ships ~6 base models as 19 catalog rows. The id
suffixes are three **orthogonal** dimensions (`glm-5.2-short-fast-flex`), parsed
by `poller.parse_serving_class` into columns. Rows carry identical catalog
pricing, so routing would pick between them arbitrarily without these — the
`latency_tolerance` hard filter resolves it.

**Access gating:** 6 of 19 rows are prose-gated
("Private preview (grant-gated)", "(Canary)"). `poller.parse_access_level`
parses them into `access_level` and `routing.allowed_access_levels` (default
`[public]`) excludes them, so dispatch won't earn a 403.

### `proficiency` — one row per (model, provider, category)

| Column | Type | Notes |
|---|---|---|
| `model_id` | TEXT | |
| `provider` | TEXT | |
| `category` | TEXT | See category list below |
| `leaderboard_score` | REAL | 0–1, from external benchmarks |
| `self_eval_score` | REAL | 0–1, from self-eval harness |
| `self_eval_samples` | INTEGER | Evidence count for blending threshold |
| `blended_score` | REAL | Computed: `w_lb × lb + w_se × se` |
| `source` | TEXT | `blended` \| `self_eval` \| `self_eval_thin` \| `leaderboard` |
| `inherited_from` | TEXT | Model this row was copied from, NULL if measured directly |
| `last_updated` | TEXT | ISO8601 |

**Category set** (9 categories, defined in `config.yaml`):

| Category | Example | Scoring type |
|---|---|---|
| `coding_general` | Merge intervals, parse semver, word wrap | Code (execution) |
| `coding_refactor` | Remove repetition, refactor dispatch chain | Code (execution) |
| `debugging` | Fix closure leak, fix binary search, fix regex | Code (execution) |
| `reasoning_math` | Percent trap, rate trap, counting | Exact match |
| `tool_use_agentic` | Right tool / right args / no tool when empty | Structural |
| `docs_writing` | Docstring quality, must-mention gotchas | Judge |
| `summarization` | Root-cause isolation, buried-lede identification | Judge |
| `translation` | Technical register, hedging/informal tone | Judge |
| `general_chat` | Simple explanations, measured pushback | Judge |

**Scoring kinds** (4 types, objective wherever the category admits it):
- **`code`** — runs model-generated Python in a subprocess, scores fraction of checks passing
- **`exact`** — normalizes & compares a single answer
- **`tool`** — structural: which tool was called, which args matched
- **`judge`** — a strong model scores against a rubric (prose categories only)

**Blending:** Once `self_eval_samples ≥ self_eval_min_samples` (default 10):
`blended = 0.3 × leaderboard + 0.7 × self_eval`. Before that, falls back
to leaderboard alone. If neither exists, the row scores neutral 0.5.

Source labels distinguish confidence levels: `self_eval_thin` means
real measurement but below the sample threshold — a caller wanting to
exclude it can.

**Proficiency inheritance:** `propagate_to_variants` copies evaluated
`scores` to equivalent serving variants (same weights, same reasoning
setting, same context pool), but never over a row that was measured
directly. A `-fast` row is **not** equivalent to its `-standard` sibling.

"Measured directly" is `inherited_from IS NULL`, not `self_eval_samples > 0`
— inheritance copies the sample count too, so sample count alone cannot tell
an inherited row from a measured one, and using it meant a variant inherited
exactly once and then froze forever. `ensure_columns()` adds the column and
backfills provenance on databases that predate it.

### `energy_observations` — per-request telemetry

| Column | Type | Notes |
|---|---|---|
| `id` | INTEGER | Autoincrement |
| `model_id` / `provider` | TEXT | |
| `task_category` | TEXT | |
| `prompt_tokens` / `completion_tokens` | INTEGER | |
| `energy_kwh` | REAL | **Attributed** billed figure (noisy, 20× within-model) |
| `energy_btu` | REAL | `kwh × 3412.14`, dashboard value |
| `avg_power_watts` / `duration_seconds` | REAL | Pre-attribution product, ~1.8× within-model |
| `attribution_ratio` | REAL | Request's share of shared GPU pool (stable quantized: 0.001, 0.25, 0.5, 0.75) |
| `carbon_g_co2eq` | REAL | Reported by provider |
| `grid_carbon_intensity` | REAL | gCO2/kWh at call time |
| `grid_id` | TEXT | e.g. `FI` |
| `carbon_source` | TEXT | `static_fallback` (constant, excluded from routing) or live measurement |
| `cost_usd` | REAL | **Billed** figure, not tokens × list price |
| `allowance_remaining_usd` | REAL | |
| `service_tier` | TEXT | As billed |
| `observed_at` | TEXT | ISO8601 |

**Attribution noise:** Billed `energy_kwh = avg_power_watts × duration ×
attribution_ratio`. The attribution term looks like noise up close (8 identical
calls varied 20×), but ranks 750× between models while within-model spread is
1.8× — it's a stable per-model property reflecting serving concurrency. Routing
scores on the attributed figures with a **median** over all `seed_reference`
rows, so repeated sweeps accumulate into a median-across-time.

### `verifications` — response quality observations

| Column | Type | Notes |
|---|---|---|
| `id` | INTEGER | Autoincrement |
| `model_id` / `provider` | TEXT | |
| `task_category` | TEXT | Optional (prose answers may lack a category) |
| `kind` | TEXT | `structural` \| `local_llm` \| `client_outcome` |
| `verdict` | TEXT | `ok` \| `truncated` \| `malformed` \| `unverifiable` \| `succeeded` \| `failed` |
| `detail` | TEXT | Human-readable reason |
| `completion_tokens` | INTEGER | Wasted answer cost, for payoff sum |
| `observed_at` | TEXT | ISO8601 |
| `applied_at` | TEXT | Set by `feedback.py` when folded into proficiency |
| `model_attributable` | INTEGER | 1 = model's fault; 0 = client caused (e.g. tight cap) |

Verifications drive the feedback loop: `feedback.py` reads unanswered failures,
applies a 0.0 sample per failure to `proficiency`, and marks them
`applied_at` for idempotency. `unverifiable` is recorded but not treated as a
failure — it means the checker had nothing to say, not that the model failed.

### `route_decisions` — routing observability

| Column | Type | Notes |
|---|---|---|
| `id` | INTEGER | Autoincrement |
| `observed_at` | TEXT | ISO8601, UTC |
| `kind` | TEXT | `route` \| `dispatch` \| `chat` \| `passthrough` \| `local_vision` |
| `task_category` | TEXT | |
| `task_tier` | INTEGER | 1–3 |
| `required_context_tokens` | INTEGER | |
| `confidence` | REAL | Classifier confidence |
| `classifier_ms` | INTEGER | Classification latency |
| `classification_source` | TEXT | `classifier` \| `override` \| `fallback` |
| `latency_tolerance` | TEXT | `interactive` \| `batch` |
| `candidates_considered` | INTEGER | How many survived hard filters |
| `selected_model` | TEXT | Null when no model was selected |
| `selected_provider` | TEXT | `neuralwatt` or `local` (vision fallback) |
| `runner_up_models` | TEXT | JSON array of up to 3 runner-up candidates |
| `est_cost_usd` | REAL | Estimated cost of the selected model |
| `est_proficiency` | REAL | Estimated proficiency for the task category |
| `rejected_reason` | TEXT | Active filters when nothing was selected |
| `session_key` | TEXT | Hashed session fingerprint ONLY |
| `tools` | INTEGER | 0/1 — request carried a `tools` array |
| `images` | INTEGER | 0/1 — request carried image parts |
| `json_mode` | INTEGER | 0/1 — request required JSON mode |
| `streamed` | INTEGER | 0/1 — response was streamed |

`route_decisions` stores one row per routing decision so "how is routing
performing" is answerable: which model was picked, for what category/tier,
how long classification took, and — when nothing was selected — which hard
filter shut it out. It is an observability table: nothing in routing reads it.

It stores only a hashed session fingerprint in `session_key`; `session_dir`,
prompts, and answers are deliberately excluded. A test enforces that the write
path does not store prompt or answer text. The write is gated by
`logging.log_route_decisions` and is **best-effort**: a failed write is logged
at warning and swallowed so monitoring cannot slow or fail a request.

## Weighted Scoring

Six hard filters are applied **before** scoring (not weighted — outright disqualification):

1. `effective_context_window ≥ required_context_tokens`
2. `tier ≥ required_tier` (from classifier)
3. Serving class compatible with request's `latency_tolerance`; `access_level` reachable
4. `tool_use_agentic` proficiency ≥ `routing.min_tool_proficiency`, **only when
   the request carries a `tools` array** (ships disabled — `null`)
5. `supports_vision = 1` when the request carries `image_url` parts; `NULL`
   fails closed (an unknown flag means the capability cannot be confirmed)
6. `supports_json_mode = 1` when `response_format.type` is `json_object` or
   `json_schema`; `NULL` also fails closed

Filters 4–6 are read from the request body rather than inferred. A `tools`
array states whether tool definitions are on the table, `image_url` parts state
whether vision is needed, and `response_format` states whether JSON mode is
needed. Local classifiers identified unambiguous tool-use prompts only 1-2
times in 6, so inferring capability requirements does not work; the request
states them exactly and for free.

The tool gate is a **quality measurement** filter: a model with no measured
`tool_use_agentic` score is admitted, because "unproven" is not "proven bad".
Only a measured score below `routing.min_tool_proficiency` is dropped. The
vision and JSON-mode gates are **capability flag** filters. There every
routable catalog row has `supports_tools = 1`, so a flag gate would be inert;
vision and JSON mode are not universal, and an absent or `NULL` flag means
"cannot confirm the capability", so the model is dropped. A wrong guess on a
capability flag is a guaranteed provider 400.

The tool filter ships **off** (`null`), because agent clients send `tools` on
nearly every request, so enabling it excludes the cheapest model from ordinary
agent traffic. Whether that trade is worth it is an empirical question best
settled with `POST /outcome` data rather than a 3-task benchmark score. Set it
to `0.5` to turn it on. The vision and JSON-mode gates ship **on**, because a
wrong guess is a guaranteed 400.

### Local Vision Fallback

Cloud vision is not universal in the catalog, and the most economical coding
rows do not declare `supports_vision`. Routing an image request through them
would earn a provider-side 400, so when `routing.require_vision` is true only
rows with `supports_vision = 1` survive the hard filters. If **no** cloud
candidate survives, the router can fall back to a local vision model instead
of returning 422.

`local_vision:` in `config.yaml` controls this path:

| Key | Default | Purpose |
|---|---|---|
| `enabled` | `true` | Whether the fallback runs at all |
| `base_url` | `http://localhost:11434/v1` | OpenAI-compatible Ollama endpoint |
| `api_key_env` | `null` | Env var holding an API key, if the endpoint needs one |
| `model` | `qwen3-vl:4b` | Vision model on that Ollama |
| `timeout_seconds` | `60` | Request timeout |
| `max_images` | `4` | Refuse requests with more image parts |
| `max_image_bytes` | `9437184` (9 MiB) | Refuse requests whose image payload exceeds this |

The fallback is **enabled by default** both in `config.yaml` and in
`LocalVisionConfig`, so omitting the section still turns it on. Disable it
explicitly (`enabled: false`) on a host with no local Ollama or one that has
not pulled the vision model.

When enabled and no cloud candidate is selected, `_run_local_vision` sends the
original message list — with `image_url` parts intact — to the configured
Ollama model. The local answer then **replaces** the cloud completion:
`_local_vision_response` returns a normal OpenAI-shaped response, including a
stream-wrapped version when the client asked for `stream: true`. It does *not*
inject a caption into a cloud call, because the streaming proxy cannot rewrite
bytes mid-stream.

Security and budget guards:
- **Only inline `data:` URIs are accepted.** A remote `http(s)` image URL is
  declined, because pointing a local model at an arbitrary URL would let an
  unauthenticated caller make the router fetch internal resources (SSRF).
- Image count and total payload size are bounded by `max_images` and
  `max_image_bytes` before the local call is made.
- If the local call fails for any reason, the request falls through to the
  ordinary `422 No model satisfies the hard filters` rather than returning a
  silent empty response.

Pull the model on whichever Ollama the fallback points at:

```bash
ollama pull qwen3-vl:4b
```

Config is strict (`extra="forbid"`): a misspelled or misplaced key fails at
load instead of being silently ignored.

```
1. drop candidates whose measured energy exceeds objective.max_energy_per_request
2. rank by proficiency for the task's category
3. treat differences smaller than objective.quality_tolerance as equal
4. among equals, pick the cheapest
```

| Setting | Value | Notes |
|---|---|---|
| `quality_tolerance` | 0.10 | Measurement noise, not preference: scores rest on 2-3 samples, so smaller gaps are sampling variation |
| `assumed_cache_rate` | 0.917 | Share of prompt tokens served from the provider's prefix cache. Agent clients resend the conversation each turn, so most of it hits. Measured token-weighted over 40.7M tokens; measure your own against the provider's session view |
| `assumed_completion_tokens` | 500 | Completion length assumed when pricing a candidate |
| `max_energy_per_request` | null | Per-request kWh ceiling — a wall, not a bill. Null disables it |
| `plan_kwh_per_period` | 6.25 | **Set this to your own plan's quota.** Reported in `/health` as burn against the allowance; it does not gate anything |

This replaced a weighted blend (cost 0.4 / eco 0.2 / proficiency 0.4).
Measurement retired it: turning the cost weight from 0.4 to **zero** changed
the winner in only 2 of 6 categories, so the blend was never steering on
quality — while 60% of every decision adjudicated fractions of a cent.

**Cost is priced per request, from catalog token prices scaled to the
request's shape** — prompt size, `assumed_completion_tokens`,
`assumed_cache_rate`. It is deliberately *not* a benchmark average.

Neuralwatt bills flat per-kWh rather than per-token, so list price is not what
gets charged. Scoring used measured billed cost from a fixed 400-token
reference sweep for exactly that reason — and that was wrong for real traffic,
because the ranking depends on the workload's *shape*, not just the model. On
a 400-token prompt one model looked 3.2× cheaper than another; on a realistic
70k-token prompt the same pair inverted and the second was 5.0× cheaper. The
provider's attribution ratio moves with prompt size, so a fixed-shape
benchmark cannot rank models for a workload of a different shape.

List price is still not what is billed, but billing is capped at a multiple of
it, so it tracks the real ordering and bounds it — and it is free, needs no
sweep, and refreshes whenever the poller runs.

**Why cost ≠ eco:** Cost tracks energy (kWh), but carbon is energy × grid
intensity. Grid intensity spanned ~49 gCO2/kWh (`FI`) to ~442
(`US-MIDA-PJM`) when measured — and it moves with time of day. The models
disagree: `glm-5.2-fast` is
2nd cheapest but 6th cleanest; `kimi-k3-flex` draws 3.7× less energy than
`kimi-k2.7-code` while emitting 3.6× more carbon. Collapsing them picks a
side.

### What routing actually returns, and why it moves

Sweeping `/route` across every category and tier is the fastest way to see
whether your data is doing anything. On the deployment this was written
against, 9 categories × 3 tiers currently yields **5 distinct winners**
(`qwen3.6-35b`, `gemma-4-31b`, `deepseek-v4-flash`, `kimi-k3`,
`kimi-k3-fast`).

That number is a diagnostic, not a target, and it is worth knowing what each
outcome means:

- **One winner everywhere** is a legitimate answer, not a misconfiguration.
  It happened here: with cost, eco and proficiency all populated, one model
  was Pareto-dominant — cheapest *and* cleanest in the routable set while
  scoring within `quality_tolerance` of the best. No defensible weighting
  picks anything else. If you see this, check whether the leader really is
  dominant before reaching for the config.
- **Winners that change with context size** are the hard filters working.
  A model is dropped once `required_context_tokens` exceeds its window, so a
  long session can change model mid-conversation. Past the largest window,
  `/v1/chat/completions` returns 422 naming the constraint rather than
  silently truncating.
- **Winners that change by category** mean proficiency is live. That is the
  only category-dependent term, so until the `proficiency` table has data,
  `task_category` cannot change a decision at all — the classifier computes
  it and the router pays for it for nothing.

The spread here widened for two reasons worth copying: cost became a
per-request estimate rather than a benchmark average, and tier stopped being
inferred from price alone. Both had been quietly excluding a cheap
large-context model from every request above tier 1.

If you want a different balance, the levers are `objective.quality_tolerance`
(how big a quality gap must be before it outranks cost) and
`objective.max_energy_per_request` (a hard ceiling). There is no weight to
tune — quality is the objective and cost is the tiebreak, which replaced an
earlier weighted blend.

## API Endpoints

The dispatcher binds `127.0.0.1:8080`. **No auth of its own** — loopback
is the only thing standing between the open internet and your billing
allowance.

| Method | Path | Description |
|---|---|---|
| `GET` | `/health` | Catalog/reachability status, scoring coverage, warnings |
| `GET` | `/metrics` | Aggregated observability JSON: quota burn, coverage, recent decisions, per-model totals, verdict mix, top proficiency; loopback-only, no auth |
| `POST` | `/route` | Classify task, rank candidates, return selected model — **no provider call, no cost** |
| `POST` | `/dispatch` | Same as `/route`, plus complete the provider call, stream response, log observation |
| `GET` | `/v1/models` | OpenAI-compatible model list (router virtual models + catalog) |
| `POST` | `/v1/chat/completions` | OpenAI-compatible completions — routes then proxies, **streaming supported** |

`/metrics` returns a single JSON object with these top-level keys:

- `quota` — kWh metered in the last 30 days against `objective.plan_kwh_per_period`
- `coverage` — routable-model counts with energy/proficiency data plus warnings
- `recent_decisions` — last 50 rows from `route_decisions`
- `per_model` — per-model aggregates over the last 30 days of `energy_observations`
- `verdict_mix` — counts by verification verdict over the last 7 days
- `top_proficiency` — top models by `blended_score` for `coding_general`
- `generated_at` — ISO8601 timestamp

It exposes no conversation text, prompts, or `session_dir`; it is bound to
loopback and unauthenticated exactly like `/health`.

Input to `/route` and `/dispatch` can include `task_category`, `task_tier`,
and `required_context_tokens` overrides — these skip the classifier, useful
for testing routing without the classifier in the loop.

Two virtual router models:
- `auto` — normal routing, `-flex` rows excluded (interactive)
- `auto:batch` — admits `-flex` rows (overnight/async work)

Ask for **any real model id** in `/v1/chat/completions` and it dispatches
directly, still logged — routing is transparent, not opaque.

**Streaming** (chunk-by-chunk proxy): Tokens render as they arrive. Neuralwatt
emits energy and cost as SSE **comment** lines (`: energy {...}`) before
`data: [DONE]` — ordinary clients ignore comments, so the stream flows
untouched while the router scrapes telemetry on the way past. Without this,
streamed calls would log no energy at all.

**Verification headers** (non-streaming): When streaming is not used, the
structural verification verdict surfaces in the `X-Router-Verification` header
so a client can inspect it without parsing the response body. Valid values:
`ok`, `truncated`, `malformed`, `unverifiable`, `none`.

**Capability 422s**: When no model survives the hard filters, the 422 names the
active constraints. That now includes "vision-capable model" or
"json-mode-capable model" when the request carried images or a JSON-mode
`response_format`, alongside the existing context/tier/latency/tool reasons.

## Quick Usage

```bash
# Pick a model without spending anything
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
  -d '{"task":"Refactor this 800-line Django view into service objects."}'

# With explicit category/tier overrides (skips classifier)
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
  -d '{"task":"Refactor this 800-line Django view","task_category":"coding_refactor","task_tier":2}'

# Actually call the winner and log energy/cost
curl -s -X POST localhost:8080/dispatch -H 'content-type: application/json' \
  -d '{"task":"What is a Python context manager?"}'

# OpenAI-compatible — works with any SDK or agent client
curl -s localhost:8080/v1/models
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'

# Admit flex rows for overnight/async work
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
  -d '{"task":"nightly code review","latency_tolerance":"batch"}'

# Image URL request (routed only to vision-capable catalog rows)
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":[{"type":"text","text":"Describe this"},{"type":"image_url","image_url":{"url":"data:image/gif;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs="}}]}]}'

# JSON-mode request via response_format
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"Return a JSON object with field answer"}],"response_format":{"type":"json_object"}}'

# Monitoring: aggregate router health
curl -s localhost:8080/metrics | python -m json.tool

# Terminal dashboard foreground tool (requires `textual`; runs until you press q)
python tui.py

# One-shot routing probe with no spend
python router_cli.py "Refactor this Django view into service objects"
python router_cli.py "Summarize this diff" --category summarization --tier 2
```

### Monitoring

Three foreground tools read the running router without spending quota:

- **`GET /metrics`** — JSON summary of quota, coverage, recent routing
  decisions, per-model usage, verdict mix, and top proficiency. No auth;
  loopback only.
- **`tui.py`** — Textual terminal dashboard that polls `/metrics` every few
  seconds. Run it in a terminal with the service already up. It is a separate
  entrypoint, not a systemd unit.
- **`router_cli.py "<task>"`** — POSTs to `/route` once and prints the full
  decision tree, including candidates, selected model, estimated cost, and
  proficiency. Use `--category`, `--tier`, and `--context` to override the
  classifier deterministically, or `--json` for raw output.

`textual` is pinned in `requirements.txt` solely for `tui.py`. It is imported
only by that module; the FastAPI service dispatch path never touches it, so
the router itself has no UI dependency.

## Logging and Traceability

One `route` line per request says what was decided; one `dispatch` line says
what it cost. Both carry a trace id, and `rid`/`sess` are columns in
`energy_observations`, so a journal line pivots to its database row and back.

```
route id=r9116d9 cat=coding_refactor tier=2 ctx=500 src=classifier tools=0
  latency=interactive cand=8 pick=deepseek-v4-flash est_usd=0.00016296 ms=1868
dispatch id=r9116d9 model=deepseek-v4-flash rid=chatcmpl-... sess=c090d751
  p_tok=59511 c_tok=415 kwh=4.8e-05 usd=0.000783 verdict=ok total_ms=5692
```

At `debug` it also says *why* — the classifier's answer and timing, every
candidate that was dropped and by which filter, and the ranking with scores:

```
classify id=r9116d9 cat=coding_refactor tier=2 confidence=0.95 ms=1863
filter   id=r9116d9 model=gemma-4-31b reason=tier(1<2)
filter   id=r9116d9 model=glm-5.2-short reason=access_level(preview)
filter   id=r9116d9 model=kimi-k3-flex reason=latency_class(flex)
rank     id=r9116d9 pos=0 model=deepseek-v4-flash prof=1 est_usd=0.00016296
```

**No conversation text is logged at any level**, prompt or answer — prompts
here run 60k–150k tokens and the journal is on disk. A test enforces it.

Set the level in `config.yaml` (`logging.level`), or override it without
touching a tracked file:

```bash
systemctl --user edit llm-router      # Environment="LLM_ROUTER_LOG_LEVEL=debug"
systemctl --user restart llm-router
```

Reading it back:

```bash
journalctl --user -u 'llm-router*' -f            # quote the glob in zsh
journalctl --user -u llm-router -f -o cat        # message only
journalctl --user -u llm-router -p warning       # severity actually filters
journalctl --user -u llm-router --grep 'chatcmpl-abc123'   # one request's trail
journalctl --user -u llm-router --grep ' id=r9116d9'       # every stage of it
```

`-p` works because the service emits journald priority prefixes when — and only
when — systemd owns its stderr. Run uvicorn in a terminal and the lines come
out clean.

## Scheduled Jobs (systemd)

Five user units. See `deploy/README.md` for full install/operate instructions.

| Unit | Schedule | What it runs |
|---|---|---|
| `llm-router.service` | Continuous | FastAPI dispatcher |
| `llm-router-poller.timer` | 2 min after boot, then every 2 h | triggers the poller unit |
| `llm-router-poller.service` | oneshot | `poller.py` → `tier.py` |
| `llm-router-seed.timer` | Every 6 h | triggers the seed unit |
| `llm-router-seed.service` | oneshot | small `seed_energy.py` sweep |

**The poller timer is load-bearing, not optional.** `freshness.stale_after_days`
is 3 with `exclude_stale: true` — an unpolled catalog marks every row stale
in 3 days and the router returns zero candidates for everything.

**The seed timer spans time.** Energy attribution drifts with pool load
across hours (~50× for one model between sweeps), so a single sweep measures
one moment. The median has to span time — every 6 h sweep accumulates into a
median-across-time automatically.

This now feeds `eco` only. Cost is priced per request from catalog prices, so
routing no longer depends on the sweep at all; disabling this timer costs you
carbon figures, not routing quality.

## Self-Eval Harness (`eval_proficiency.py`)

```bash
python eval_proficiency.py                      # every routable model × every task
python eval_proficiency.py --models kimi-k3     # subset of models
python eval_proficiency.py --categories coding_general
python eval_proficiency.py --dry-run            # plan only
```

- Runs every task through the target provider, scores it, writes to `proficiency`.
- Scores **accumulate** (running mean), so repeated runs tighten estimates.
- `propagate_to_variants` copies evaluated scores to equivalent serving variants.
- Safety: code tasks run in a temp directory with a 15s wall-clock timeout —
  bounded isolation, not a container.
- Judge tasks skip the model being judged (avoids self-scoring bias).

## Classifier Reliability Notes

The classifier is the one blocking LLM call on the request path, so it sets
the latency floor for every routed request. `classifier.base_url` /
`api_key_env` / `model` accept any OpenAI-compatible endpoint — a local
Ollama, an Ollama on another machine across a VPN, or a cloud model.

**Pick a non-reasoning model.** The classifier's entire output is ~45 tokens
of JSON, and a reasoning model spends its budget getting there. Replacing
`qwen3.5` with `mistral-nemo:12b` on identical prompts, cold load excluded:

| | `qwen3.5` | `mistral-nemo:12b` |
|---|---|---|
| category correct | 10/14 | 9/14 |
| tier correct | 1/5 (2 crashed) | 3/5 |
| **hard failures** | **4 of 19 calls** | **0** |
| latency mean / max | 6.6s / **15.6s** | 1.7s / **1.9s** |

Accuracy is a wash; the failure column and the tail are not. Every failure was
an unbounded thinking trace producing no JSON, which degrades to
`source: "fallback"` after ~15s. End-to-end `/route` went ~10s → ~1.7s.

Several settings keep it from cascading failures:

- **`max_input_chars: 8000`** clamps what the classifier is shown to head +
  tail. It decides a category and a tier; it does not need the document, and
  feeding it one is harmful rather than wasteful — on a ~20k-token prompt one
  model returned empty after 28.7s and the other spent 41.8s echoing the input
  into its JSON. Clamped, the same prompts classify in ~2.2s.


- **`max_retries: 0`** on the classifier client. The OpenAI SDK retries twice
  by default, so `timeout_seconds` silently became a 3× wall-clock bound — a
  request could hang past 250s on a 120s setting with no log.
- **`max_output_tokens: 1024`** bounds a reasoning model's chain of thought.
  Without it, a reasoning model emits unbounded traces — Ollama keeps generating after
  the client gives up AND serializes per model, so one runaway request queues
  every later request behind it. 256 was tried first and was too tight; the
  trace would consume the whole budget and the model would be truncated before
  emitting JSON.
- **`temperature: 0`** prevents non-reproducible classifications. At the
  default, the same prompt classified tier 2 then tier 1 on consecutive calls,
  routing to two different models.
- **Graceful fallback** routes to `fallback_tier: 2` / `fallback_category:
  general_chat` when the classifier times out, errors, or returns
  unparseable output. A coding agent would rather have a mid-tier answer than
  an error. Escalation deliberately skips fallbacks so an unavailable local
  model doesn't silently promote every request to the frontier tier.

## Pointing a Coding Agent at It

The `/v1` endpoints are OpenAI-compatible, so any normal client works —
opencode, an SDK, plain curl. A repo-local `opencode.json` is included, so
running `opencode` from a clone of this repo routes through the router by
default. For global use, merge its `provider.llm-router` block into
`~/.config/opencode/opencode.json`.

Virtual model names:
- `auto` → router picks, interactive mode (flex rows excluded)
- `auto:batch` → router picks, admits flex rows for async work

**opencode image support**: The repo's `opencode.json` declares
`"modalities": {"input": ["text", "image"]}` for every `llm-router` model.
This is required: opencode strips image parts client-side unless the provider
model declares image input. Without it, images never reach the router at all.
For global use, make sure the merged `~/.config/opencode/opencode.json` entry
carries the same modality block.

## Testing

```bash
python -m pytest        # 356 tests
python -m pytest --cov  # with coverage
```

No test calls a provider or a local model — the pure modules take rows and
config as arguments, so the suite runs offline on a clean checkout.


| Test file | What it covers |
|---|---|
| `test_scoring.py` | `normalize_inverted`, `composite_score`, None-handling |
| `test_routing.py` | Hard filters, candidate selection & ranking across all 9 categories × 3 tiers |
| `test_tiering.py` | Tier resolver precedence: override → reasoning → cost + context window → mid |
| `test_apply_tiering.py` | DB tiering pass, sanity guards |
| `test_poller_parsing.py` | Serving class, base model, access level parsing |
| `test_proficiency.py` | Blending rule, accumulation |
| `test_load_candidates.py` | `load_candidates`: cost/eco/proficiency join |
| `test_eval_scoring.py` | Code, exact, tool, and judge scoring |
| `test_task_set.py` | Reference solutions validating every `code` task's checks & `exact` answers |
| `test_verification.py` | Structural checks on every language, verifier edge cases, local-LLM gating |
| `test_feedback.py` | Failure identification, idempotency, attribution filtering |
| `test_iteration.py` | Retry budget per tier, and matching the retry to the failure kind |
| `test_session_identity.py` | Outcome attribution: session matching, ambiguity refusal |
| `test_config_endpoints.py` | Classifier and verifier are separately addressable; guards on the split |

## Setup

**You need:** a Neuralwatt API key, Python 3.10+ (suite verified on 3.10 and
3.14), and an Ollama reachable from wherever this runs with a classifier model
pulled. Nothing else is assumed about the host — routing itself is SQLite and
arithmetic.

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
sqlite3 router.db < schema.sql
cp .env.example .env   # fill in NEURALWATT_API_KEY
python poller.py       # populate the catalog
python tier.py         # resolve tiers
python config.py       # sanity-check config loads
python -m uvicorn dispatcher:app --reload
```

Then edit `config.yaml` for your own setup — at minimum:

| Key | Why |
|---|---|
| `classifier.model` | must match a model `ollama list` reports |
| `classifier.base_url` | where that Ollama actually is |
| `objective.plan_kwh_per_period` | your plan's quota; `/health` reports burn against it |
| `objective.assumed_cache_rate` | 0.917 was measured from one client's traffic (40.7M tokens). Check yours against the provider's per-session cache-hit figures |

`python seed_energy.py` is optional. It sweeps a fixed reference workload to
populate `eco`, which is logged but is not an objective — routing works
without it. It costs real money and quota, so it is not in the path above.

### Where Ollama lives

```bash
ollama pull mistral-nemo:12b   # or whatever you set as classifier.model
```

It does not have to be on the machine running the router; the box with the
GPU usually isn't the laptop. To use one across a VPN, point **both**
endpoints at it:

```yaml
classifier:
  base_url: "http://<vpn-ip>:11434/v1"
verification:
  base_url: "http://<vpn-ip>:11434"    # same host, so `model` can stay null
```

and apply `deploy/ollama-over-vpn.conf` on the serving host — Ollama binds
`127.0.0.1` by default and will otherwise refuse. Bind it to the VPN address
rather than `0.0.0.0`: Ollama has no authentication, so anything reaching the
port can run inference and enumerate your models.

Both endpoints move together because the verifier speaks Ollama's *native*
API and cannot follow the classifier to a cloud provider. Config load refuses
the case where they are on different hosts and `verification.model` is null,
because that combination fails silently.

## Known Limitations & Open Items

- **Leaderboard priors unfilled** — `leaderboards.yaml` ships empty. `python leaderboard.py --check`
  names every active family lacking a prior. Must source figures from
  published benchmarks; inventing plausible numbers is the same failure as
  the provider's `static_fallback` carbon constant.
- **Three models unsettled** — split-half stability at 7 samples:
  `kimi-k2.7-code-fast` (29×), `kimi-k3` (14×), `glm-5.2-flex` (2.2×).
  More samples needed before their positioning is trustworthy.
- **Retry does not reach streaming** — a failed structural check now buys
  corrective attempts scaled by tier (`iteration.py`), but only on the
  non-streaming path. Once bytes have gone to the client there is nothing to
  take back, and buffering to allow correction would cost streaming itself.
  `POST /outcome` is the answer for streamed traffic: the client reports
  afterwards, so it works identically either way.
- **No auth** — the service holds a billable API key with no authentication
  of its own. It binds loopback for that reason; widening the bind needs an
  auth layer first. The same applies to an Ollama shared over a VPN, which
  also has no auth — bind it to the VPN address, never `0.0.0.0`.
- **Local energy is not accounted for** — the router measures what the cloud
  provider bills and reports, but the electricity its own classifier and
  verifier burn on your hardware is invisible to it. Since the whole premise
  is spending cheap local compute to avoid wasting expensive cloud compute,
  that half of the ledger is currently taken on faith. Closing it means
  metering local draw and pricing it against a real tariff rather than
  assuming local is free.
- **Context assembly (RAG)** is out of scope — the classifier sees the full
  conversation but does not perform document/code retrieval.