8b861f9267
README gains the Local Vision fallback section (the Ollama/qwen3-vl path for image requests with no cloud vision candidate — previously only the cloud capability gates were documented) plus route_decisions table, metrics.py/tui.py/router_cli.py in the module list, GET /metrics endpoint and keys, and monitoring usage snippets. CLAUDE gains metrics.py/tui.py/router_cli.py in what's-built and a Monitoring: route_decisions persistence section. Also corrects verifications.kind/verdict to match schema.sql.
977 lines
51 KiB
Markdown
977 lines
51 KiB
Markdown
# Local LLM Model Router
|
||
|
||
A router that uses a local model (served via Ollama) to classify incoming
|
||
coding/documentation tasks — category, tier, required context size — and
|
||
dispatch each task to the cheapest/best-fit open-weight model on **Neuralwatt
|
||
Cloud**, weighted by cost, per-category proficiency, and a per-request energy
|
||
ceiling.
|
||
|
||
The local model does the classifying, so it wants a GPU, but not necessarily
|
||
*your* GPU — `classifier.base_url` takes any OpenAI-compatible endpoint, so
|
||
the usual shape is the router on your laptop and Ollama on a workstation
|
||
across a VPN. See [Setup](#setup).
|
||
|
||
**Numbers in this README are measurements, not specifications.** They come
|
||
from one deployment against one provider account, and the catalog, prices,
|
||
grid intensity and pool load all move. They are here because the reasoning
|
||
behind a design choice is worth more than the choice, and re-running the
|
||
measurement is how you check whether it still holds for you.
|
||
|
||
Neuralwatt is the only provider. The `provider` column and the
|
||
`(model_id, provider)` primary key stay so a second provider can be added
|
||
later without a migration.
|
||
|
||
## At a Glance
|
||
|
||
| Dimension | Detail |
|
||
|---|---|
|
||
| **Cost model** | Per-kWh, not per-token. Flat $8.00/kWh measured across the catalog. |
|
||
| **Latency** | Classification is the floor: ~5-15 s warm on a local reasoning model, up to ~120 s cold. A cloud classifier measured ~1 s. Verification is async and never blocks the response. |
|
||
| **Quality** | Quality is the objective; cost is a per-request kWh ceiling plus a tiebreak. Eco is logged but no longer optimized. Proficiency blends external benchmarks with a self-run eval harness. |
|
||
| **Verification** | Every response structurally checked (free) + local LLM spot-check on long prose answers (~6 s, never blocks). Failures fold back into proficiency via `feedback.py`. |
|
||
| **Fault tolerance** | Classifier failure degrades to a mid-tier fallback rather than 502/503. SDK retry count is zero, to prevent `timeout_seconds` silently becoming a 3× wall-clock bound. |
|
||
| **API surface** | OpenAI-compatible `/v1` endpoints, streaming chunk proxy with SSE telemetry scraping. |
|
||
|
||
## Verification Pipeline
|
||
|
||
Every completion passes through a two-layer check **before** routing learns from
|
||
it and **without** adding to client-facing latency for the async layer.
|
||
|
||
### Structural Verification — `verification.py`
|
||
|
||
Free, exact checks on what a model just returned — no code execution.
|
||
Extracts fenced code blocks and validates them by **parsing**, not running:
|
||
|
||
| Language | Check | Safeguard |
|
||
|---|---|---|
|
||
| Python (`python`, `py`, `python3`) | `ast.parse` | Syntax tree, no evaluation |
|
||
| JSON (`json`, `jsonc`) | `json.loads` | Strict parse |
|
||
| YAML (`yaml`, `yml`) | `yaml.safe_load` | No arbitrary objects |
|
||
| Shell (`bash`, `sh`, `zsh`) | `bash -n` (caller's job) | Parse-only, never runs |
|
||
| Other | `unverifiable` (not a failure) | Prose answers land here |
|
||
|
||
Plus two universal checks:
|
||
- **`finish_reason == "length"`** — decisive truncation, even if parsed content
|
||
looks fine (the most dangerous case: a fragment that parses)
|
||
- **Unterminated code fences** — the response ran out of tokens mid-block
|
||
|
||
**Key design decisions:**
|
||
- Never executes model output. The eval harness (`eval_proficiency.py`) does
|
||
run generated code, but there the prompts are ones this project authored,
|
||
so what comes back is bounded. Here the code is whatever the user asked for
|
||
and could do anything.
|
||
- `unverifiable` is **not** a failure. Most prose answers land there (correctly
|
||
— there's nothing structural to check), and counting it as wrong would
|
||
penalize models for the checker's limits.
|
||
- Measured on real traffic: a wasted cloud completion costs about 52 local
|
||
checks at 1,500 tokens and 139 at 4,000. A zero-cost check pays trivially.
|
||
|
||
### Local LLM Verification — `verification.py` (cont.)
|
||
|
||
For answers nothing structural can judge (prose, reasoning, refusals), a local
|
||
model spot-checks the response **after** it has gone back to the client, so its
|
||
~6 s never lands on anyone's latency.
|
||
|
||
The prompt frames the local model as a judge: "Does this answer actually
|
||
address the user's request?" and returns `{"ok": true|false, "reason": "..."}`.
|
||
|
||
Gating and safeguards:
|
||
- **Size threshold** — only responses ≥ 600 completion tokens trigger a local
|
||
check (configurable via `verification.min_completion_tokens`). A check on
|
||
a 193-token answer costs ~15% of the answer; at 1,500 tokens the break-even
|
||
failure rate drops to ~1.9%.
|
||
- **Middle elision** — if the answer is longer than the limit (default 4,000),
|
||
the **middle** is elided, not the head, so the checker judges the real ending.
|
||
An elision marker tells the checker not to flag it as a defect.
|
||
- **Thinking disabled** — the local model is a reasoning model and will otherwise
|
||
emit unbounded chain-of-thought. Verification uses Ollama's native endpoint
|
||
with `think=False`, so it answers in ~25 tokens instead of burning through
|
||
the budget on reasoning.
|
||
- **Malfunction safety** — an empty or unparseable verdict produces no sample
|
||
rather than a false failure. An empty response is settled by an if-statement,
|
||
never asked of a model what code can decide.
|
||
|
||
### Feedback Loop — `feedback.py`
|
||
|
||
Observation → learning. `feedback.py` folds verification failures into
|
||
`proficiency` so routing improves on **your traffic**, not just the fixed
|
||
23-task benchmark:
|
||
|
||
```bash
|
||
python feedback.py --dry-run # preview what would change
|
||
python feedback.py # apply
|
||
```
|
||
|
||
Key behaviors:
|
||
- **Only failures** are folded in. A structural 'ok' means the code parsed,
|
||
not that it was correct — scoring a parse-success as a proficiency 1.0 would
|
||
flatten every score toward the ceiling. The coding categories already sit
|
||
at 1.00 for every model; this would spread that flatness everywhere.
|
||
- **One 0.0 sample per failure**, added to the running mean so the penalty
|
||
scales with failure rate rather than replacing the benchmark score outright.
|
||
- **Idempotent** — `applied_at` marks consumed rows so re-running cannot
|
||
penalize a model repeatedly for the same response.
|
||
- **Not the model's fault** — a client that sets `max_tokens=40` and gets a
|
||
truncated answer caused that itself. Those failures are recorded in the
|
||
verifications table but excluded from proficiency feedback.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌─────────────────────┐
|
||
incoming task ───▶│ Local Classifier │ Ollama (mistral-nemo:12b)
|
||
│ - task_category │ ~4s warm, ~120s cold cap
|
||
│ - task_tier │ temperature: 0, max 1024 tokens
|
||
│ - required_context │ max_retries: 0 (silent 3× cap guard)
|
||
│ - confidence │
|
||
└──────────┬───────────┘
|
||
▼
|
||
┌─────────────────────┐
|
||
│ Classifier Fallback │ tier 2 / general_chat on failure
|
||
│ (graceful degrade) │ — not a 502/503
|
||
└──────────┬───────────┘
|
||
▼
|
||
┌─────────────────┐
|
||
│ Escalation │ low-confidence tier bump
|
||
│ (optional) │ threshold: 0.6 confidence
|
||
└────────┬─────────┘
|
||
▼
|
||
┌─────────────────────┐
|
||
│ Hard Filters │ context window ≥ required
|
||
│ (routing.py) │ tier ≥ required
|
||
│ │ freshness (active, not stale)
|
||
│ │ access_level allowed
|
||
│ │ latency_class compatible
|
||
└──────────┬───────────┘
|
||
▼
|
||
┌─────────────────────┐
|
||
│ Quality-first select │ max proficiency, cheapest
|
||
│ (routing.py) │ among equals, under a
|
||
│ │ per-request kWh ceiling
|
||
└──────────┬───────────┘
|
||
▼
|
||
┌──────────────────────┐
|
||
│ Dispatcher / Provider │──▶ Neuralwatt Cloud
|
||
│ (FastAPI API) │──▶ OpenAI-compatible /v1
|
||
│ │ Streaming SSE + energy scrape
|
||
└───────┬──────────────┘
|
||
▼
|
||
┌─────────────────────────┐
|
||
│ Verification Pipeline │
|
||
│ Layer 1: Structural │ ast.parse, json.load, yaml.safe_load
|
||
│ Layer 2: Local LLM │ async, ~6s, >600 token gate
|
||
└──────┬──────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────┐
|
||
│ feedback.py │ failures → proficiency updates
|
||
│ (on-demand agent) │ idempotent, attributed-only
|
||
└─────────────────────┘
|
||
```
|
||
|
||
## Tech Stack
|
||
|
||
| Layer | Technology |
|
||
|---|---|
|
||
| **Language** | Python 3.10+ (IO-bound provider APIs; iteration speed matters more than raw speed) |
|
||
| **Framework** | FastAPI + uvicorn |
|
||
| **Database** | SQLite (`router.db`) — decision table, energy observations, proficiency, verifications |
|
||
| **Local Classification** | Ollama, OpenAI-compatible — `localhost:11434/v1` or an Ollama across your VPN |
|
||
| **Local Model** | `classifier.model` — `mistral-nemo:12b` by default; any Ollama model works |
|
||
| **Cloud Provider** | Neuralwatt only |
|
||
| **Config** | `config.yaml` loaded & validated by Pydantic (`config.py`) |
|
||
| **OpenAI Client** | `openai==3.0.0` (official SDK) |
|
||
| **HTTP** | `requests` for poller, `httpx` (via openai/uvicorn) |
|
||
| **Testing** | `pytest` — 356 tests across 19 files, all offline |
|
||
| **Config Files** | `config.yaml`, `leaderboards.yaml`, `evals/tasks.yaml` |
|
||
| **Deployment** | systemd user units (`.service` + `.timer` files in `deploy/`) |
|
||
| **Integration** | `opencode.json` in the repo routes through it by default; any OpenAI-compatible client works |
|
||
|
||
## Pinned Dependencies (requirements.txt)
|
||
|
||
```
|
||
pyyaml==6.0.3
|
||
pydantic==2.13.4
|
||
requests==2.34.2
|
||
fastapi==0.141.1
|
||
uvicorn[standard]==0.52.1
|
||
openai==3.0.0
|
||
python-dotenv==1.2.2
|
||
pytest==9.1.1
|
||
pytest-cov==7.1.0
|
||
```
|
||
|
||
**Pinned intentionally.** Recreating the venv with `>=` constraints silently jumped
|
||
`openai 2.53 → 3.0` and `httpx → httpx2` without warning; a service that
|
||
restarts on boot shouldn't change its dependency tree underneath itself.
|
||
|
||
## Modules
|
||
|
||
| Module | Role | I/O? |
|
||
|---|---|---|
|
||
| **`dispatcher.py`** | FastAPI service: routes, calls providers, logs, streams | Yes (DB, network) |
|
||
| **`poller.py`** | Fetches Neuralwatt catalog, normalizes, upserts `models` table | Yes (network, DB) |
|
||
| **`scoring.py`** | `normalize_inverted` + `composite_score` weighted formula | Pure |
|
||
| **`routing.py`** | Hard filters (`select_candidates`) + ranking (`rank_candidates`) | Pure |
|
||
| **`tiering.py`** | Pure tier resolver: 1=cheap, 2=mid, 3=frontier | Pure |
|
||
| **`tier.py`** | DB tiering pass: reads models, resolves, writes `tier` column | Yes (DB) |
|
||
| **`proficiency.py`** | Score blending: leaderboard + self-eval → weighted composite | Pure |
|
||
| **`proficiency_store.py`** | DB access for `proficiency` table; single write path ensuring `blended_score` never drifts | Yes (DB) |
|
||
| **`seed_energy.py`** | Reference workload sweep: fixed prompt × N runs per model | Yes (network, DB) |
|
||
| **`eval_proficiency.py`** | Self-eval harness: 4 scoring kinds against N models | Yes (network, DB, subprocess) |
|
||
| **`verification.py`** | Two-layer check: structural parse (always) + local LLM spot-check (async, size-gated). Never executes model output | Pure |
|
||
| **`feedback.py`** | Folds observed verification failures into `proficiency`; routing learns from real traffic | Yes (DB) |
|
||
| **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) |
|
||
| **`iteration.py`** | Retry budget per tier, and matching the retry to the failure kind | Pure |
|
||
| **`config.py`** | YAML loader + Pydantic validators (blend weights sum to 1, valid tiers, endpoints separately addressable) | Yes (file) |
|
||
| **`metrics.py`** | Read-only aggregations for `/health` and `GET /metrics`: quota burn, coverage, recent decisions, per-model totals, verdict mix, top proficiency | Yes (DB) |
|
||
| **`tui.py`** | Textual terminal dashboard over `GET /metrics`; foreground tool, not a service | Yes (network) |
|
||
| **`router_cli.py`** | One-shot routing probe: POSTs to `/route` and prints the decision tree | Yes (network) |
|
||
|
||
## Decision Table Schema (SQLite)
|
||
|
||
Three data tables plus one observability table, `PRAGMA foreign_keys = ON`:
|
||
|
||
### `models` — one row per served model variant
|
||
|
||
| Column | Type | Notes |
|
||
|---|---|---|
|
||
| `model_id` | TEXT | Full catalog id (e.g. `glm-5.2-short-fast-flex`) |
|
||
| `provider` | TEXT | `neuralwatt` |
|
||
| `base_model_id` | TEXT | Model family (e.g. `glm-5.2`). Proficiency/leaderboard keys here. |
|
||
| `display_name` | TEXT | Human-readable name |
|
||
| `cost_per_1m_prompt` | REAL | Listed USD per 1M input tokens |
|
||
| `cost_per_1m_completion` | REAL | Listed USD per 1M output tokens |
|
||
| `cost_per_1m_prompt_cached` | REAL | Cached prefix price (null if no cache discount) |
|
||
| `context_window` | INTEGER | Advertised max tokens |
|
||
| `effective_context_window` | INTEGER | `advertised × safety_factor − reserve` |
|
||
| `max_output_tokens` | INTEGER | |
|
||
| `tier` | INTEGER | 1–3, set by `tier.py` pass |
|
||
| `supports_tools` | INTEGER | Boolean 0/1 |
|
||
| `supports_json_mode` | INTEGER | |
|
||
| `supports_vision` | INTEGER | |
|
||
| `supports_reasoning` | INTEGER | "API accepts reasoning param" — NOT a quality signal |
|
||
| `reasoning_default_enabled` | INTEGER | The actual tier-bearing signal |
|
||
| `latency_class` | TEXT | `standard` \| `flex` (`-flex`: discounted async, held during peak) |
|
||
| `reasoning_mode` | TEXT | `default` \| `reduced` (`-fast`: reasoning capped) |
|
||
| `context_variant` | TEXT | `full` \| `short` (`-short`: 200K pool bounded budget) |
|
||
| `access_level` | TEXT | `public` \| `preview` \| `canary` |
|
||
| `pricing_tbd` | INTEGER | |
|
||
| `deprecated` | INTEGER | |
|
||
| `availability` | TEXT | `active` \| `deprecated` \| `stale` |
|
||
| `last_updated` | TEXT | ISO8601 |
|
||
|
||
**Serving class:** Neuralwatt ships ~6 base models as 19 catalog rows. The id
|
||
suffixes are three **orthogonal** dimensions (`glm-5.2-short-fast-flex`), parsed
|
||
by `poller.parse_serving_class` into columns. Rows carry identical catalog
|
||
pricing, so routing would pick between them arbitrarily without these — the
|
||
`latency_tolerance` hard filter resolves it.
|
||
|
||
**Access gating:** 6 of 19 rows are prose-gated
|
||
("Private preview (grant-gated)", "(Canary)"). `poller.parse_access_level`
|
||
parses them into `access_level` and `routing.allowed_access_levels` (default
|
||
`[public]`) excludes them, so dispatch won't earn a 403.
|
||
|
||
### `proficiency` — one row per (model, provider, category)
|
||
|
||
| Column | Type | Notes |
|
||
|---|---|---|
|
||
| `model_id` | TEXT | |
|
||
| `provider` | TEXT | |
|
||
| `category` | TEXT | See category list below |
|
||
| `leaderboard_score` | REAL | 0–1, from external benchmarks |
|
||
| `self_eval_score` | REAL | 0–1, from self-eval harness |
|
||
| `self_eval_samples` | INTEGER | Evidence count for blending threshold |
|
||
| `blended_score` | REAL | Computed: `w_lb × lb + w_se × se` |
|
||
| `source` | TEXT | `blended` \| `self_eval` \| `self_eval_thin` \| `leaderboard` |
|
||
| `inherited_from` | TEXT | Model this row was copied from, NULL if measured directly |
|
||
| `last_updated` | TEXT | ISO8601 |
|
||
|
||
**Category set** (9 categories, defined in `config.yaml`):
|
||
|
||
| Category | Example | Scoring type |
|
||
|---|---|---|
|
||
| `coding_general` | Merge intervals, parse semver, word wrap | Code (execution) |
|
||
| `coding_refactor` | Remove repetition, refactor dispatch chain | Code (execution) |
|
||
| `debugging` | Fix closure leak, fix binary search, fix regex | Code (execution) |
|
||
| `reasoning_math` | Percent trap, rate trap, counting | Exact match |
|
||
| `tool_use_agentic` | Right tool / right args / no tool when empty | Structural |
|
||
| `docs_writing` | Docstring quality, must-mention gotchas | Judge |
|
||
| `summarization` | Root-cause isolation, buried-lede identification | Judge |
|
||
| `translation` | Technical register, hedging/informal tone | Judge |
|
||
| `general_chat` | Simple explanations, measured pushback | Judge |
|
||
|
||
**Scoring kinds** (4 types, objective wherever the category admits it):
|
||
- **`code`** — runs model-generated Python in a subprocess, scores fraction of checks passing
|
||
- **`exact`** — normalizes & compares a single answer
|
||
- **`tool`** — structural: which tool was called, which args matched
|
||
- **`judge`** — a strong model scores against a rubric (prose categories only)
|
||
|
||
**Blending:** Once `self_eval_samples ≥ self_eval_min_samples` (default 10):
|
||
`blended = 0.3 × leaderboard + 0.7 × self_eval`. Before that, falls back
|
||
to leaderboard alone. If neither exists, the row scores neutral 0.5.
|
||
|
||
Source labels distinguish confidence levels: `self_eval_thin` means
|
||
real measurement but below the sample threshold — a caller wanting to
|
||
exclude it can.
|
||
|
||
**Proficiency inheritance:** `propagate_to_variants` copies evaluated
|
||
`scores` to equivalent serving variants (same weights, same reasoning
|
||
setting, same context pool), but never over a row that was measured
|
||
directly. A `-fast` row is **not** equivalent to its `-standard` sibling.
|
||
|
||
"Measured directly" is `inherited_from IS NULL`, not `self_eval_samples > 0`
|
||
— inheritance copies the sample count too, so sample count alone cannot tell
|
||
an inherited row from a measured one, and using it meant a variant inherited
|
||
exactly once and then froze forever. `ensure_columns()` adds the column and
|
||
backfills provenance on databases that predate it.
|
||
|
||
### `energy_observations` — per-request telemetry
|
||
|
||
| Column | Type | Notes |
|
||
|---|---|---|
|
||
| `id` | INTEGER | Autoincrement |
|
||
| `model_id` / `provider` | TEXT | |
|
||
| `task_category` | TEXT | |
|
||
| `prompt_tokens` / `completion_tokens` | INTEGER | |
|
||
| `energy_kwh` | REAL | **Attributed** billed figure (noisy, 20× within-model) |
|
||
| `energy_btu` | REAL | `kwh × 3412.14`, dashboard value |
|
||
| `avg_power_watts` / `duration_seconds` | REAL | Pre-attribution product, ~1.8× within-model |
|
||
| `attribution_ratio` | REAL | Request's share of shared GPU pool (stable quantized: 0.001, 0.25, 0.5, 0.75) |
|
||
| `carbon_g_co2eq` | REAL | Reported by provider |
|
||
| `grid_carbon_intensity` | REAL | gCO2/kWh at call time |
|
||
| `grid_id` | TEXT | e.g. `FI` |
|
||
| `carbon_source` | TEXT | `static_fallback` (constant, excluded from routing) or live measurement |
|
||
| `cost_usd` | REAL | **Billed** figure, not tokens × list price |
|
||
| `allowance_remaining_usd` | REAL | |
|
||
| `service_tier` | TEXT | As billed |
|
||
| `observed_at` | TEXT | ISO8601 |
|
||
|
||
**Attribution noise:** Billed `energy_kwh = avg_power_watts × duration ×
|
||
attribution_ratio`. The attribution term looks like noise up close (8 identical
|
||
calls varied 20×), but ranks 750× between models while within-model spread is
|
||
1.8× — it's a stable per-model property reflecting serving concurrency. Routing
|
||
scores on the attributed figures with a **median** over all `seed_reference`
|
||
rows, so repeated sweeps accumulate into a median-across-time.
|
||
|
||
### `verifications` — response quality observations
|
||
|
||
| Column | Type | Notes |
|
||
|---|---|---|
|
||
| `id` | INTEGER | Autoincrement |
|
||
| `model_id` / `provider` | TEXT | |
|
||
| `task_category` | TEXT | Optional (prose answers may lack a category) |
|
||
| `kind` | TEXT | `structural` \| `local_llm` \| `client_outcome` |
|
||
| `verdict` | TEXT | `ok` \| `truncated` \| `malformed` \| `unverifiable` \| `succeeded` \| `failed` |
|
||
| `detail` | TEXT | Human-readable reason |
|
||
| `completion_tokens` | INTEGER | Wasted answer cost, for payoff sum |
|
||
| `observed_at` | TEXT | ISO8601 |
|
||
| `applied_at` | TEXT | Set by `feedback.py` when folded into proficiency |
|
||
| `model_attributable` | INTEGER | 1 = model's fault; 0 = client caused (e.g. tight cap) |
|
||
|
||
Verifications drive the feedback loop: `feedback.py` reads unanswered failures,
|
||
applies a 0.0 sample per failure to `proficiency`, and marks them
|
||
`applied_at` for idempotency. `unverifiable` is recorded but not treated as a
|
||
failure — it means the checker had nothing to say, not that the model failed.
|
||
|
||
### `route_decisions` — routing observability
|
||
|
||
| Column | Type | Notes |
|
||
|---|---|---|
|
||
| `id` | INTEGER | Autoincrement |
|
||
| `observed_at` | TEXT | ISO8601, UTC |
|
||
| `kind` | TEXT | `route` \| `dispatch` \| `chat` \| `passthrough` \| `local_vision` |
|
||
| `task_category` | TEXT | |
|
||
| `task_tier` | INTEGER | 1–3 |
|
||
| `required_context_tokens` | INTEGER | |
|
||
| `confidence` | REAL | Classifier confidence |
|
||
| `classifier_ms` | INTEGER | Classification latency |
|
||
| `classification_source` | TEXT | `classifier` \| `override` \| `fallback` |
|
||
| `latency_tolerance` | TEXT | `interactive` \| `batch` |
|
||
| `candidates_considered` | INTEGER | How many survived hard filters |
|
||
| `selected_model` | TEXT | Null when no model was selected |
|
||
| `selected_provider` | TEXT | `neuralwatt` or `local` (vision fallback) |
|
||
| `runner_up_models` | TEXT | JSON array of up to 3 runner-up candidates |
|
||
| `est_cost_usd` | REAL | Estimated cost of the selected model |
|
||
| `est_proficiency` | REAL | Estimated proficiency for the task category |
|
||
| `rejected_reason` | TEXT | Active filters when nothing was selected |
|
||
| `session_key` | TEXT | Hashed session fingerprint ONLY |
|
||
| `tools` | INTEGER | 0/1 — request carried a `tools` array |
|
||
| `images` | INTEGER | 0/1 — request carried image parts |
|
||
| `json_mode` | INTEGER | 0/1 — request required JSON mode |
|
||
| `streamed` | INTEGER | 0/1 — response was streamed |
|
||
|
||
`route_decisions` stores one row per routing decision so "how is routing
|
||
performing" is answerable: which model was picked, for what category/tier,
|
||
how long classification took, and — when nothing was selected — which hard
|
||
filter shut it out. It is an observability table: nothing in routing reads it.
|
||
|
||
It stores only a hashed session fingerprint in `session_key`; `session_dir`,
|
||
prompts, and answers are deliberately excluded. A test enforces that the write
|
||
path does not store prompt or answer text. The write is gated by
|
||
`logging.log_route_decisions` and is **best-effort**: a failed write is logged
|
||
at warning and swallowed so monitoring cannot slow or fail a request.
|
||
|
||
## Weighted Scoring
|
||
|
||
Six hard filters are applied **before** scoring (not weighted — outright disqualification):
|
||
|
||
1. `effective_context_window ≥ required_context_tokens`
|
||
2. `tier ≥ required_tier` (from classifier)
|
||
3. Serving class compatible with request's `latency_tolerance`; `access_level` reachable
|
||
4. `tool_use_agentic` proficiency ≥ `routing.min_tool_proficiency`, **only when
|
||
the request carries a `tools` array** (ships disabled — `null`)
|
||
5. `supports_vision = 1` when the request carries `image_url` parts; `NULL`
|
||
fails closed (an unknown flag means the capability cannot be confirmed)
|
||
6. `supports_json_mode = 1` when `response_format.type` is `json_object` or
|
||
`json_schema`; `NULL` also fails closed
|
||
|
||
Filters 4–6 are read from the request body rather than inferred. A `tools`
|
||
array states whether tool definitions are on the table, `image_url` parts state
|
||
whether vision is needed, and `response_format` states whether JSON mode is
|
||
needed. Local classifiers identified unambiguous tool-use prompts only 1-2
|
||
times in 6, so inferring capability requirements does not work; the request
|
||
states them exactly and for free.
|
||
|
||
The tool gate is a **quality measurement** filter: a model with no measured
|
||
`tool_use_agentic` score is admitted, because "unproven" is not "proven bad".
|
||
Only a measured score below `routing.min_tool_proficiency` is dropped. The
|
||
vision and JSON-mode gates are **capability flag** filters. There every
|
||
routable catalog row has `supports_tools = 1`, so a flag gate would be inert;
|
||
vision and JSON mode are not universal, and an absent or `NULL` flag means
|
||
"cannot confirm the capability", so the model is dropped. A wrong guess on a
|
||
capability flag is a guaranteed provider 400.
|
||
|
||
The tool filter ships **off** (`null`), because agent clients send `tools` on
|
||
nearly every request, so enabling it excludes the cheapest model from ordinary
|
||
agent traffic. Whether that trade is worth it is an empirical question best
|
||
settled with `POST /outcome` data rather than a 3-task benchmark score. Set it
|
||
to `0.5` to turn it on. The vision and JSON-mode gates ship **on**, because a
|
||
wrong guess is a guaranteed 400.
|
||
|
||
### Local Vision Fallback
|
||
|
||
Cloud vision is not universal in the catalog, and the most economical coding
|
||
rows do not declare `supports_vision`. Routing an image request through them
|
||
would earn a provider-side 400, so when `routing.require_vision` is true only
|
||
rows with `supports_vision = 1` survive the hard filters. If **no** cloud
|
||
candidate survives, the router can fall back to a local vision model instead
|
||
of returning 422.
|
||
|
||
`local_vision:` in `config.yaml` controls this path:
|
||
|
||
| Key | Default | Purpose |
|
||
|---|---|---|
|
||
| `enabled` | `true` | Whether the fallback runs at all |
|
||
| `base_url` | `http://localhost:11434/v1` | OpenAI-compatible Ollama endpoint |
|
||
| `api_key_env` | `null` | Env var holding an API key, if the endpoint needs one |
|
||
| `model` | `qwen3-vl:4b` | Vision model on that Ollama |
|
||
| `timeout_seconds` | `60` | Request timeout |
|
||
| `max_images` | `4` | Refuse requests with more image parts |
|
||
| `max_image_bytes` | `9437184` (9 MiB) | Refuse requests whose image payload exceeds this |
|
||
|
||
The fallback is **enabled by default** both in `config.yaml` and in
|
||
`LocalVisionConfig`, so omitting the section still turns it on. Disable it
|
||
explicitly (`enabled: false`) on a host with no local Ollama or one that has
|
||
not pulled the vision model.
|
||
|
||
When enabled and no cloud candidate is selected, `_run_local_vision` sends the
|
||
original message list — with `image_url` parts intact — to the configured
|
||
Ollama model. The local answer then **replaces** the cloud completion:
|
||
`_local_vision_response` returns a normal OpenAI-shaped response, including a
|
||
stream-wrapped version when the client asked for `stream: true`. It does *not*
|
||
inject a caption into a cloud call, because the streaming proxy cannot rewrite
|
||
bytes mid-stream.
|
||
|
||
Security and budget guards:
|
||
- **Only inline `data:` URIs are accepted.** A remote `http(s)` image URL is
|
||
declined, because pointing a local model at an arbitrary URL would let an
|
||
unauthenticated caller make the router fetch internal resources (SSRF).
|
||
- Image count and total payload size are bounded by `max_images` and
|
||
`max_image_bytes` before the local call is made.
|
||
- If the local call fails for any reason, the request falls through to the
|
||
ordinary `422 No model satisfies the hard filters` rather than returning a
|
||
silent empty response.
|
||
|
||
Pull the model on whichever Ollama the fallback points at:
|
||
|
||
```bash
|
||
ollama pull qwen3-vl:4b
|
||
```
|
||
|
||
Config is strict (`extra="forbid"`): a misspelled or misplaced key fails at
|
||
load instead of being silently ignored.
|
||
|
||
```
|
||
1. drop candidates whose measured energy exceeds objective.max_energy_per_request
|
||
2. rank by proficiency for the task's category
|
||
3. treat differences smaller than objective.quality_tolerance as equal
|
||
4. among equals, pick the cheapest
|
||
```
|
||
|
||
| Setting | Value | Notes |
|
||
|---|---|---|
|
||
| `quality_tolerance` | 0.10 | Measurement noise, not preference: scores rest on 2-3 samples, so smaller gaps are sampling variation |
|
||
| `assumed_cache_rate` | 0.917 | Share of prompt tokens served from the provider's prefix cache. Agent clients resend the conversation each turn, so most of it hits. Measured token-weighted over 40.7M tokens; measure your own against the provider's session view |
|
||
| `assumed_completion_tokens` | 500 | Completion length assumed when pricing a candidate |
|
||
| `max_energy_per_request` | null | Per-request kWh ceiling — a wall, not a bill. Null disables it |
|
||
| `plan_kwh_per_period` | 6.25 | **Set this to your own plan's quota.** Reported in `/health` as burn against the allowance; it does not gate anything |
|
||
|
||
This replaced a weighted blend (cost 0.4 / eco 0.2 / proficiency 0.4).
|
||
Measurement retired it: turning the cost weight from 0.4 to **zero** changed
|
||
the winner in only 2 of 6 categories, so the blend was never steering on
|
||
quality — while 60% of every decision adjudicated fractions of a cent.
|
||
|
||
**Cost is priced per request, from catalog token prices scaled to the
|
||
request's shape** — prompt size, `assumed_completion_tokens`,
|
||
`assumed_cache_rate`. It is deliberately *not* a benchmark average.
|
||
|
||
Neuralwatt bills flat per-kWh rather than per-token, so list price is not what
|
||
gets charged. Scoring used measured billed cost from a fixed 400-token
|
||
reference sweep for exactly that reason — and that was wrong for real traffic,
|
||
because the ranking depends on the workload's *shape*, not just the model. On
|
||
a 400-token prompt one model looked 3.2× cheaper than another; on a realistic
|
||
70k-token prompt the same pair inverted and the second was 5.0× cheaper. The
|
||
provider's attribution ratio moves with prompt size, so a fixed-shape
|
||
benchmark cannot rank models for a workload of a different shape.
|
||
|
||
List price is still not what is billed, but billing is capped at a multiple of
|
||
it, so it tracks the real ordering and bounds it — and it is free, needs no
|
||
sweep, and refreshes whenever the poller runs.
|
||
|
||
**Why cost ≠ eco:** Cost tracks energy (kWh), but carbon is energy × grid
|
||
intensity. Grid intensity spanned ~49 gCO2/kWh (`FI`) to ~442
|
||
(`US-MIDA-PJM`) when measured — and it moves with time of day. The models
|
||
disagree: `glm-5.2-fast` is
|
||
2nd cheapest but 6th cleanest; `kimi-k3-flex` draws 3.7× less energy than
|
||
`kimi-k2.7-code` while emitting 3.6× more carbon. Collapsing them picks a
|
||
side.
|
||
|
||
### What routing actually returns, and why it moves
|
||
|
||
Sweeping `/route` across every category and tier is the fastest way to see
|
||
whether your data is doing anything. On the deployment this was written
|
||
against, 9 categories × 3 tiers currently yields **5 distinct winners**
|
||
(`qwen3.6-35b`, `gemma-4-31b`, `deepseek-v4-flash`, `kimi-k3`,
|
||
`kimi-k3-fast`).
|
||
|
||
That number is a diagnostic, not a target, and it is worth knowing what each
|
||
outcome means:
|
||
|
||
- **One winner everywhere** is a legitimate answer, not a misconfiguration.
|
||
It happened here: with cost, eco and proficiency all populated, one model
|
||
was Pareto-dominant — cheapest *and* cleanest in the routable set while
|
||
scoring within `quality_tolerance` of the best. No defensible weighting
|
||
picks anything else. If you see this, check whether the leader really is
|
||
dominant before reaching for the config.
|
||
- **Winners that change with context size** are the hard filters working.
|
||
A model is dropped once `required_context_tokens` exceeds its window, so a
|
||
long session can change model mid-conversation. Past the largest window,
|
||
`/v1/chat/completions` returns 422 naming the constraint rather than
|
||
silently truncating.
|
||
- **Winners that change by category** mean proficiency is live. That is the
|
||
only category-dependent term, so until the `proficiency` table has data,
|
||
`task_category` cannot change a decision at all — the classifier computes
|
||
it and the router pays for it for nothing.
|
||
|
||
The spread here widened for two reasons worth copying: cost became a
|
||
per-request estimate rather than a benchmark average, and tier stopped being
|
||
inferred from price alone. Both had been quietly excluding a cheap
|
||
large-context model from every request above tier 1.
|
||
|
||
If you want a different balance, the levers are `objective.quality_tolerance`
|
||
(how big a quality gap must be before it outranks cost) and
|
||
`objective.max_energy_per_request` (a hard ceiling). There is no weight to
|
||
tune — quality is the objective and cost is the tiebreak, which replaced an
|
||
earlier weighted blend.
|
||
|
||
## API Endpoints
|
||
|
||
The dispatcher binds `127.0.0.1:8080`. **No auth of its own** — loopback
|
||
is the only thing standing between the open internet and your billing
|
||
allowance.
|
||
|
||
| Method | Path | Description |
|
||
|---|---|---|
|
||
| `GET` | `/health` | Catalog/reachability status, scoring coverage, warnings |
|
||
| `GET` | `/metrics` | Aggregated observability JSON: quota burn, coverage, recent decisions, per-model totals, verdict mix, top proficiency; loopback-only, no auth |
|
||
| `POST` | `/route` | Classify task, rank candidates, return selected model — **no provider call, no cost** |
|
||
| `POST` | `/dispatch` | Same as `/route`, plus complete the provider call, stream response, log observation |
|
||
| `GET` | `/v1/models` | OpenAI-compatible model list (router virtual models + catalog) |
|
||
| `POST` | `/v1/chat/completions` | OpenAI-compatible completions — routes then proxies, **streaming supported** |
|
||
|
||
`/metrics` returns a single JSON object with these top-level keys:
|
||
|
||
- `quota` — kWh metered in the last 30 days against `objective.plan_kwh_per_period`
|
||
- `coverage` — routable-model counts with energy/proficiency data plus warnings
|
||
- `recent_decisions` — last 50 rows from `route_decisions`
|
||
- `per_model` — per-model aggregates over the last 30 days of `energy_observations`
|
||
- `verdict_mix` — counts by verification verdict over the last 7 days
|
||
- `top_proficiency` — top models by `blended_score` for `coding_general`
|
||
- `generated_at` — ISO8601 timestamp
|
||
|
||
It exposes no conversation text, prompts, or `session_dir`; it is bound to
|
||
loopback and unauthenticated exactly like `/health`.
|
||
|
||
Input to `/route` and `/dispatch` can include `task_category`, `task_tier`,
|
||
and `required_context_tokens` overrides — these skip the classifier, useful
|
||
for testing routing without the classifier in the loop.
|
||
|
||
Two virtual router models:
|
||
- `auto` — normal routing, `-flex` rows excluded (interactive)
|
||
- `auto:batch` — admits `-flex` rows (overnight/async work)
|
||
|
||
Ask for **any real model id** in `/v1/chat/completions` and it dispatches
|
||
directly, still logged — routing is transparent, not opaque.
|
||
|
||
**Streaming** (chunk-by-chunk proxy): Tokens render as they arrive. Neuralwatt
|
||
emits energy and cost as SSE **comment** lines (`: energy {...}`) before
|
||
`data: [DONE]` — ordinary clients ignore comments, so the stream flows
|
||
untouched while the router scrapes telemetry on the way past. Without this,
|
||
streamed calls would log no energy at all.
|
||
|
||
**Verification headers** (non-streaming): When streaming is not used, the
|
||
structural verification verdict surfaces in the `X-Router-Verification` header
|
||
so a client can inspect it without parsing the response body. Valid values:
|
||
`ok`, `truncated`, `malformed`, `unverifiable`, `none`.
|
||
|
||
**Capability 422s**: When no model survives the hard filters, the 422 names the
|
||
active constraints. That now includes "vision-capable model" or
|
||
"json-mode-capable model" when the request carried images or a JSON-mode
|
||
`response_format`, alongside the existing context/tier/latency/tool reasons.
|
||
|
||
## Quick Usage
|
||
|
||
```bash
|
||
# Pick a model without spending anything
|
||
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
|
||
-d '{"task":"Refactor this 800-line Django view into service objects."}'
|
||
|
||
# With explicit category/tier overrides (skips classifier)
|
||
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
|
||
-d '{"task":"Refactor this 800-line Django view","task_category":"coding_refactor","task_tier":2}'
|
||
|
||
# Actually call the winner and log energy/cost
|
||
curl -s -X POST localhost:8080/dispatch -H 'content-type: application/json' \
|
||
-d '{"task":"What is a Python context manager?"}'
|
||
|
||
# OpenAI-compatible — works with any SDK or agent client
|
||
curl -s localhost:8080/v1/models
|
||
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
||
-d '{"model":"auto","messages":[{"role":"user","content":"hello"}]}'
|
||
|
||
# Admit flex rows for overnight/async work
|
||
curl -s -X POST localhost:8080/route -H 'content-type: application/json' \
|
||
-d '{"task":"nightly code review","latency_tolerance":"batch"}'
|
||
|
||
# Image URL request (routed only to vision-capable catalog rows)
|
||
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
||
-d '{"model":"auto","messages":[{"role":"user","content":[{"type":"text","text":"Describe this"},{"type":"image_url","image_url":{"url":"data:image/gif;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs="}}]}]}'
|
||
|
||
# JSON-mode request via response_format
|
||
curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
|
||
-d '{"model":"auto","messages":[{"role":"user","content":"Return a JSON object with field answer"}],"response_format":{"type":"json_object"}}'
|
||
|
||
# Monitoring: aggregate router health
|
||
curl -s localhost:8080/metrics | python -m json.tool
|
||
|
||
# Terminal dashboard foreground tool (requires `textual`; runs until you press q)
|
||
python tui.py
|
||
|
||
# One-shot routing probe with no spend
|
||
python router_cli.py "Refactor this Django view into service objects"
|
||
python router_cli.py "Summarize this diff" --category summarization --tier 2
|
||
```
|
||
|
||
### Monitoring
|
||
|
||
Three foreground tools read the running router without spending quota:
|
||
|
||
- **`GET /metrics`** — JSON summary of quota, coverage, recent routing
|
||
decisions, per-model usage, verdict mix, and top proficiency. No auth;
|
||
loopback only.
|
||
- **`tui.py`** — Textual terminal dashboard that polls `/metrics` every few
|
||
seconds. Run it in a terminal with the service already up. It is a separate
|
||
entrypoint, not a systemd unit.
|
||
- **`router_cli.py "<task>"`** — POSTs to `/route` once and prints the full
|
||
decision tree, including candidates, selected model, estimated cost, and
|
||
proficiency. Use `--category`, `--tier`, and `--context` to override the
|
||
classifier deterministically, or `--json` for raw output.
|
||
|
||
`textual` is pinned in `requirements.txt` solely for `tui.py`. It is imported
|
||
only by that module; the FastAPI service dispatch path never touches it, so
|
||
the router itself has no UI dependency.
|
||
|
||
## Logging and Traceability
|
||
|
||
One `route` line per request says what was decided; one `dispatch` line says
|
||
what it cost. Both carry a trace id, and `rid`/`sess` are columns in
|
||
`energy_observations`, so a journal line pivots to its database row and back.
|
||
|
||
```
|
||
route id=r9116d9 cat=coding_refactor tier=2 ctx=500 src=classifier tools=0
|
||
latency=interactive cand=8 pick=deepseek-v4-flash est_usd=0.00016296 ms=1868
|
||
dispatch id=r9116d9 model=deepseek-v4-flash rid=chatcmpl-... sess=c090d751
|
||
p_tok=59511 c_tok=415 kwh=4.8e-05 usd=0.000783 verdict=ok total_ms=5692
|
||
```
|
||
|
||
At `debug` it also says *why* — the classifier's answer and timing, every
|
||
candidate that was dropped and by which filter, and the ranking with scores:
|
||
|
||
```
|
||
classify id=r9116d9 cat=coding_refactor tier=2 confidence=0.95 ms=1863
|
||
filter id=r9116d9 model=gemma-4-31b reason=tier(1<2)
|
||
filter id=r9116d9 model=glm-5.2-short reason=access_level(preview)
|
||
filter id=r9116d9 model=kimi-k3-flex reason=latency_class(flex)
|
||
rank id=r9116d9 pos=0 model=deepseek-v4-flash prof=1 est_usd=0.00016296
|
||
```
|
||
|
||
**No conversation text is logged at any level**, prompt or answer — prompts
|
||
here run 60k–150k tokens and the journal is on disk. A test enforces it.
|
||
|
||
Set the level in `config.yaml` (`logging.level`), or override it without
|
||
touching a tracked file:
|
||
|
||
```bash
|
||
systemctl --user edit llm-router # Environment="LLM_ROUTER_LOG_LEVEL=debug"
|
||
systemctl --user restart llm-router
|
||
```
|
||
|
||
Reading it back:
|
||
|
||
```bash
|
||
journalctl --user -u 'llm-router*' -f # quote the glob in zsh
|
||
journalctl --user -u llm-router -f -o cat # message only
|
||
journalctl --user -u llm-router -p warning # severity actually filters
|
||
journalctl --user -u llm-router --grep 'chatcmpl-abc123' # one request's trail
|
||
journalctl --user -u llm-router --grep ' id=r9116d9' # every stage of it
|
||
```
|
||
|
||
`-p` works because the service emits journald priority prefixes when — and only
|
||
when — systemd owns its stderr. Run uvicorn in a terminal and the lines come
|
||
out clean.
|
||
|
||
## Scheduled Jobs (systemd)
|
||
|
||
Five user units. See `deploy/README.md` for full install/operate instructions.
|
||
|
||
| Unit | Schedule | What it runs |
|
||
|---|---|---|
|
||
| `llm-router.service` | Continuous | FastAPI dispatcher |
|
||
| `llm-router-poller.timer` | 2 min after boot, then every 2 h | triggers the poller unit |
|
||
| `llm-router-poller.service` | oneshot | `poller.py` → `tier.py` |
|
||
| `llm-router-seed.timer` | Every 6 h | triggers the seed unit |
|
||
| `llm-router-seed.service` | oneshot | small `seed_energy.py` sweep |
|
||
|
||
**The poller timer is load-bearing, not optional.** `freshness.stale_after_days`
|
||
is 3 with `exclude_stale: true` — an unpolled catalog marks every row stale
|
||
in 3 days and the router returns zero candidates for everything.
|
||
|
||
**The seed timer spans time.** Energy attribution drifts with pool load
|
||
across hours (~50× for one model between sweeps), so a single sweep measures
|
||
one moment. The median has to span time — every 6 h sweep accumulates into a
|
||
median-across-time automatically.
|
||
|
||
This now feeds `eco` only. Cost is priced per request from catalog prices, so
|
||
routing no longer depends on the sweep at all; disabling this timer costs you
|
||
carbon figures, not routing quality.
|
||
|
||
## Self-Eval Harness (`eval_proficiency.py`)
|
||
|
||
```bash
|
||
python eval_proficiency.py # every routable model × every task
|
||
python eval_proficiency.py --models kimi-k3 # subset of models
|
||
python eval_proficiency.py --categories coding_general
|
||
python eval_proficiency.py --dry-run # plan only
|
||
```
|
||
|
||
- Runs every task through the target provider, scores it, writes to `proficiency`.
|
||
- Scores **accumulate** (running mean), so repeated runs tighten estimates.
|
||
- `propagate_to_variants` copies evaluated scores to equivalent serving variants.
|
||
- Safety: code tasks run in a temp directory with a 15s wall-clock timeout —
|
||
bounded isolation, not a container.
|
||
- Judge tasks skip the model being judged (avoids self-scoring bias).
|
||
|
||
## Classifier Reliability Notes
|
||
|
||
The classifier is the one blocking LLM call on the request path, so it sets
|
||
the latency floor for every routed request. `classifier.base_url` /
|
||
`api_key_env` / `model` accept any OpenAI-compatible endpoint — a local
|
||
Ollama, an Ollama on another machine across a VPN, or a cloud model.
|
||
|
||
**Pick a non-reasoning model.** The classifier's entire output is ~45 tokens
|
||
of JSON, and a reasoning model spends its budget getting there. Replacing
|
||
`qwen3.5` with `mistral-nemo:12b` on identical prompts, cold load excluded:
|
||
|
||
| | `qwen3.5` | `mistral-nemo:12b` |
|
||
|---|---|---|
|
||
| category correct | 10/14 | 9/14 |
|
||
| tier correct | 1/5 (2 crashed) | 3/5 |
|
||
| **hard failures** | **4 of 19 calls** | **0** |
|
||
| latency mean / max | 6.6s / **15.6s** | 1.7s / **1.9s** |
|
||
|
||
Accuracy is a wash; the failure column and the tail are not. Every failure was
|
||
an unbounded thinking trace producing no JSON, which degrades to
|
||
`source: "fallback"` after ~15s. End-to-end `/route` went ~10s → ~1.7s.
|
||
|
||
Several settings keep it from cascading failures:
|
||
|
||
- **`max_input_chars: 8000`** clamps what the classifier is shown to head +
|
||
tail. It decides a category and a tier; it does not need the document, and
|
||
feeding it one is harmful rather than wasteful — on a ~20k-token prompt one
|
||
model returned empty after 28.7s and the other spent 41.8s echoing the input
|
||
into its JSON. Clamped, the same prompts classify in ~2.2s.
|
||
|
||
|
||
- **`max_retries: 0`** on the classifier client. The OpenAI SDK retries twice
|
||
by default, so `timeout_seconds` silently became a 3× wall-clock bound — a
|
||
request could hang past 250s on a 120s setting with no log.
|
||
- **`max_output_tokens: 1024`** bounds a reasoning model's chain of thought.
|
||
Without it, a reasoning model emits unbounded traces — Ollama keeps generating after
|
||
the client gives up AND serializes per model, so one runaway request queues
|
||
every later request behind it. 256 was tried first and was too tight; the
|
||
trace would consume the whole budget and the model would be truncated before
|
||
emitting JSON.
|
||
- **`temperature: 0`** prevents non-reproducible classifications. At the
|
||
default, the same prompt classified tier 2 then tier 1 on consecutive calls,
|
||
routing to two different models.
|
||
- **Graceful fallback** routes to `fallback_tier: 2` / `fallback_category:
|
||
general_chat` when the classifier times out, errors, or returns
|
||
unparseable output. A coding agent would rather have a mid-tier answer than
|
||
an error. Escalation deliberately skips fallbacks so an unavailable local
|
||
model doesn't silently promote every request to the frontier tier.
|
||
|
||
## Pointing a Coding Agent at It
|
||
|
||
The `/v1` endpoints are OpenAI-compatible, so any normal client works —
|
||
opencode, an SDK, plain curl. A repo-local `opencode.json` is included, so
|
||
running `opencode` from a clone of this repo routes through the router by
|
||
default. For global use, merge its `provider.llm-router` block into
|
||
`~/.config/opencode/opencode.json`.
|
||
|
||
Virtual model names:
|
||
- `auto` → router picks, interactive mode (flex rows excluded)
|
||
- `auto:batch` → router picks, admits flex rows for async work
|
||
|
||
**opencode image support**: The repo's `opencode.json` declares
|
||
`"modalities": {"input": ["text", "image"]}` for every `llm-router` model.
|
||
This is required: opencode strips image parts client-side unless the provider
|
||
model declares image input. Without it, images never reach the router at all.
|
||
For global use, make sure the merged `~/.config/opencode/opencode.json` entry
|
||
carries the same modality block.
|
||
|
||
## Testing
|
||
|
||
```bash
|
||
python -m pytest # 356 tests
|
||
python -m pytest --cov # with coverage
|
||
```
|
||
|
||
No test calls a provider or a local model — the pure modules take rows and
|
||
config as arguments, so the suite runs offline on a clean checkout.
|
||
|
||
|
||
| Test file | What it covers |
|
||
|---|---|
|
||
| `test_scoring.py` | `normalize_inverted`, `composite_score`, None-handling |
|
||
| `test_routing.py` | Hard filters, candidate selection & ranking across all 9 categories × 3 tiers |
|
||
| `test_tiering.py` | Tier resolver precedence: override → reasoning → cost + context window → mid |
|
||
| `test_apply_tiering.py` | DB tiering pass, sanity guards |
|
||
| `test_poller_parsing.py` | Serving class, base model, access level parsing |
|
||
| `test_proficiency.py` | Blending rule, accumulation |
|
||
| `test_load_candidates.py` | `load_candidates`: cost/eco/proficiency join |
|
||
| `test_eval_scoring.py` | Code, exact, tool, and judge scoring |
|
||
| `test_task_set.py` | Reference solutions validating every `code` task's checks & `exact` answers |
|
||
| `test_verification.py` | Structural checks on every language, verifier edge cases, local-LLM gating |
|
||
| `test_feedback.py` | Failure identification, idempotency, attribution filtering |
|
||
| `test_iteration.py` | Retry budget per tier, and matching the retry to the failure kind |
|
||
| `test_session_identity.py` | Outcome attribution: session matching, ambiguity refusal |
|
||
| `test_config_endpoints.py` | Classifier and verifier are separately addressable; guards on the split |
|
||
|
||
## Setup
|
||
|
||
**You need:** a Neuralwatt API key, Python 3.10+ (suite verified on 3.10 and
|
||
3.14), and an Ollama reachable from wherever this runs with a classifier model
|
||
pulled. Nothing else is assumed about the host — routing itself is SQLite and
|
||
arithmetic.
|
||
|
||
```bash
|
||
python -m venv .venv && source .venv/bin/activate
|
||
pip install -r requirements.txt
|
||
sqlite3 router.db < schema.sql
|
||
cp .env.example .env # fill in NEURALWATT_API_KEY
|
||
python poller.py # populate the catalog
|
||
python tier.py # resolve tiers
|
||
python config.py # sanity-check config loads
|
||
python -m uvicorn dispatcher:app --reload
|
||
```
|
||
|
||
Then edit `config.yaml` for your own setup — at minimum:
|
||
|
||
| Key | Why |
|
||
|---|---|
|
||
| `classifier.model` | must match a model `ollama list` reports |
|
||
| `classifier.base_url` | where that Ollama actually is |
|
||
| `objective.plan_kwh_per_period` | your plan's quota; `/health` reports burn against it |
|
||
| `objective.assumed_cache_rate` | 0.917 was measured from one client's traffic (40.7M tokens). Check yours against the provider's per-session cache-hit figures |
|
||
|
||
`python seed_energy.py` is optional. It sweeps a fixed reference workload to
|
||
populate `eco`, which is logged but is not an objective — routing works
|
||
without it. It costs real money and quota, so it is not in the path above.
|
||
|
||
### Where Ollama lives
|
||
|
||
```bash
|
||
ollama pull mistral-nemo:12b # or whatever you set as classifier.model
|
||
```
|
||
|
||
It does not have to be on the machine running the router; the box with the
|
||
GPU usually isn't the laptop. To use one across a VPN, point **both**
|
||
endpoints at it:
|
||
|
||
```yaml
|
||
classifier:
|
||
base_url: "http://<vpn-ip>:11434/v1"
|
||
verification:
|
||
base_url: "http://<vpn-ip>:11434" # same host, so `model` can stay null
|
||
```
|
||
|
||
and apply `deploy/ollama-over-vpn.conf` on the serving host — Ollama binds
|
||
`127.0.0.1` by default and will otherwise refuse. Bind it to the VPN address
|
||
rather than `0.0.0.0`: Ollama has no authentication, so anything reaching the
|
||
port can run inference and enumerate your models.
|
||
|
||
Both endpoints move together because the verifier speaks Ollama's *native*
|
||
API and cannot follow the classifier to a cloud provider. Config load refuses
|
||
the case where they are on different hosts and `verification.model` is null,
|
||
because that combination fails silently.
|
||
|
||
## Known Limitations & Open Items
|
||
|
||
- **Leaderboard priors unfilled** — `leaderboards.yaml` ships empty. `python leaderboard.py --check`
|
||
names every active family lacking a prior. Must source figures from
|
||
published benchmarks; inventing plausible numbers is the same failure as
|
||
the provider's `static_fallback` carbon constant.
|
||
- **Three models unsettled** — split-half stability at 7 samples:
|
||
`kimi-k2.7-code-fast` (29×), `kimi-k3` (14×), `glm-5.2-flex` (2.2×).
|
||
More samples needed before their positioning is trustworthy.
|
||
- **Retry does not reach streaming** — a failed structural check now buys
|
||
corrective attempts scaled by tier (`iteration.py`), but only on the
|
||
non-streaming path. Once bytes have gone to the client there is nothing to
|
||
take back, and buffering to allow correction would cost streaming itself.
|
||
`POST /outcome` is the answer for streamed traffic: the client reports
|
||
afterwards, so it works identically either way.
|
||
- **No auth** — the service holds a billable API key with no authentication
|
||
of its own. It binds loopback for that reason; widening the bind needs an
|
||
auth layer first. The same applies to an Ollama shared over a VPN, which
|
||
also has no auth — bind it to the VPN address, never `0.0.0.0`.
|
||
- **Local energy is not accounted for** — the router measures what the cloud
|
||
provider bills and reports, but the electricity its own classifier and
|
||
verifier burn on your hardware is invisible to it. Since the whole premise
|
||
is spending cheap local compute to avoid wasting expensive cloud compute,
|
||
that half of the ledger is currently taken on faith. Closing it means
|
||
metering local draw and pricing it against a real tariff rather than
|
||
assuming local is free.
|
||
- **Context assembly (RAG)** is out of scope — the classifier sees the full
|
||
conversation but does not perform document/code retrieval.
|