CLAUDE.md

@@ -242,10 +242,43 @@ rather than from months of history.
   StreamingResponse's generator is resumed in a fresh copy of the caller's
   context, so the ContextVar cannot reach it — the path all agent traffic
   takes. Level from `logging.level`, overridden by `LLM_ROUTER_LOG_LEVEL`.
+- `metrics.py` / `GET /metrics` — read-only observability aggregation. Keeps
+  the dashboard helpers outside `dispatcher.py` to avoid an import cycle:
+  `dispatcher` imports `metrics` for `/health` and `/metrics`, so `metrics`
+  takes `(conn, cfg)` arguments and never imports `dispatcher`. Returns quota
+  burn against `plan_kwh_per_period`, scoring coverage warnings, recent
+  `route_decisions`, per-model aggregates over `energy_observations`,
+  verification verdict mix, and top proficiency by category.
+- `tui.py` — Textual terminal dashboard over `GET /metrics`. A foreground
+  entrypoint (`python tui.py`), not a service. `textual` is imported only
+  here, so the router's dispatch path has no UI dependency.
+- `router_cli.py` — one-shot `/route` probe. Posts a task to the running router
+  and prints the decision tree, or emits raw JSON with `--json`. Spends no
+  quota because it only routes.
 - `tests/` — 356 tests across 19 files, all passing, all offline. Verified on
   Python 3.10 and 3.14; nothing declares `requires-python`, so 3.10 is the
   tested floor rather than a promised one.
 
+### Monitoring: route_decisions persistence
+
+`route_decisions` is the newest observability table (not a scoring input). It
+records one row per routing decision — `route` \| `dispatch` \| `chat` |
+`passthrough` \| `local_vision` — with category, tier, selected model, runners
+up, estimated cost/proficiency, classifier latency, rejection reason, and
+request feature flags (`tools`, `images`, `json_mode`, `streamed`). It stores
+only a hashed `session_key`; `session_dir`, prompts, and answers are excluded,
+and a test enforces that the write path never stores conversation text.
+
+The write is gated by `logging.log_route_decisions` and is **best-effort**: a
+failed write is logged and swallowed, because a decision record is worth
+having but never worth failing or slowing a request for. The table is created
+from code at module load (`_ensure_route_decisions_table`) and again on every
+write (`ensure_route_decisions` inside `persist_route_decision`), mirroring
+the `proficiency_store.ensure_columns` migration pattern: `schema.sql` is
+`CREATE TABLE IF NOT EXISTS`, but a live `router.db` predating this table
+needs the code-side migration. Both the module-load hook and the write-path
+guarantee are idempotent and leave existing rows intact.
+
 ### Request-side capability gates
 
 The router treats some capabilities as hard filters, read directly from the

README.md

@@ -224,6 +224,9 @@ restarts on boot shouldn't change its dependency tree underneath itself.
 | **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) |
 | **`iteration.py`** | Retry budget per tier, and matching the retry to the failure kind | Pure |
 | **`config.py`** | YAML loader + Pydantic validators (blend weights sum to 1, valid tiers, endpoints separately addressable) | Yes (file) |
+| **`metrics.py`** | Read-only aggregations for `/health` and `GET /metrics`: quota burn, coverage, recent decisions, per-model totals, verdict mix, top proficiency | Yes (DB) |
+| **`tui.py`** | Textual terminal dashboard over `GET /metrics`; foreground tool, not a service | Yes (network) |
+| **`router_cli.py`** | One-shot routing probe: POSTs to `/route` and prints the decision tree | Yes (network) |
 
 ## Decision Table Schema (SQLite)
 
@@ -358,8 +361,8 @@ rows, so repeated sweeps accumulate into a median-across-time.
 | `id` | INTEGER | Autoincrement |
 | `model_id` / `provider` | TEXT | |
 | `task_category` | TEXT | Optional (prose answers may lack a category) |
-| `kind` | TEXT | `structural` \| `local_llm` |
+| `kind` | TEXT | `structural` \| `local_llm` \| `client_outcome` |
-| `verdict` | TEXT | `ok` \| `truncated` \| `malformed` \| `unverifiable` |
+| `verdict` | TEXT | `ok` \| `truncated` \| `malformed` \| `unverifiable` \| `succeeded` \| `failed` |
 | `detail` | TEXT | Human-readable reason |
 | `completion_tokens` | INTEGER | Wasted answer cost, for payoff sum |
 | `observed_at` | TEXT | ISO8601 |
@@ -371,6 +374,44 @@ applies a 0.0 sample per failure to `proficiency`, and marks them
 `applied_at` for idempotency. `unverifiable` is recorded but not treated as a
 failure — it means the checker had nothing to say, not that the model failed.
 
+### `route_decisions` — routing observability
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER | Autoincrement |
+| `observed_at` | TEXT | ISO8601, UTC |
+| `kind` | TEXT | `route` \| `dispatch` \| `chat` \| `passthrough` \| `local_vision` |
+| `task_category` | TEXT | |
+| `task_tier` | INTEGER | 1–3 |
+| `required_context_tokens` | INTEGER | |
+| `confidence` | REAL | Classifier confidence |
+| `classifier_ms` | INTEGER | Classification latency |
+| `classification_source` | TEXT | `classifier` \| `override` \| `fallback` |
+| `latency_tolerance` | TEXT | `interactive` \| `batch` |
+| `candidates_considered` | INTEGER | How many survived hard filters |
+| `selected_model` | TEXT | Null when no model was selected |
+| `selected_provider` | TEXT | `neuralwatt` or `local` (vision fallback) |
+| `runner_up_models` | TEXT | JSON array of up to 3 runner-up candidates |
+| `est_cost_usd` | REAL | Estimated cost of the selected model |
+| `est_proficiency` | REAL | Estimated proficiency for the task category |
+| `rejected_reason` | TEXT | Active filters when nothing was selected |
+| `session_key` | TEXT | Hashed session fingerprint ONLY |
+| `tools` | INTEGER | 0/1 — request carried a `tools` array |
+| `images` | INTEGER | 0/1 — request carried image parts |
+| `json_mode` | INTEGER | 0/1 — request required JSON mode |
+| `streamed` | INTEGER | 0/1 — response was streamed |
+
+`route_decisions` stores one row per routing decision so "how is routing
+performing" is answerable: which model was picked, for what category/tier,
+how long classification took, and — when nothing was selected — which hard
+filter shut it out. It is an observability table: nothing in routing reads it.
+
+It stores only a hashed session fingerprint in `session_key`; `session_dir`,
+prompts, and answers are deliberately excluded. A test enforces that the write
+path does not store prompt or answer text. The write is gated by
+`logging.log_route_decisions` and is **best-effort**: a failed write is logged
+at warning and swallowed so monitoring cannot slow or fail a request.
+
 ## Weighted Scoring
 
 Six hard filters are applied **before** scoring (not weighted — outright disqualification):
@@ -408,6 +449,56 @@ settled with `POST /outcome` data rather than a 3-task benchmark score. Set it
 to `0.5` to turn it on. The vision and JSON-mode gates ship **on**, because a
 wrong guess is a guaranteed 400.
 
+### Local Vision Fallback
+
+Cloud vision is not universal in the catalog, and the most economical coding
+rows do not declare `supports_vision`. Routing an image request through them
+would earn a provider-side 400, so when `routing.require_vision` is true only
+rows with `supports_vision = 1` survive the hard filters. If **no** cloud
+candidate survives, the router can fall back to a local vision model instead
+of returning 422.
+
+`local_vision:` in `config.yaml` controls this path:
+
+| Key | Default | Purpose |
+|---|---|---|
+| `enabled` | `true` | Whether the fallback runs at all |
+| `base_url` | `http://localhost:11434/v1` | OpenAI-compatible Ollama endpoint |
+| `api_key_env` | `null` | Env var holding an API key, if the endpoint needs one |
+| `model` | `qwen3-vl:4b` | Vision model on that Ollama |
+| `timeout_seconds` | `60` | Request timeout |
+| `max_images` | `4` | Refuse requests with more image parts |
+| `max_image_bytes` | `9437184` (9 MiB) | Refuse requests whose image payload exceeds this |
+
+The fallback is **enabled by default** both in `config.yaml` and in
+`LocalVisionConfig`, so omitting the section still turns it on. Disable it
+explicitly (`enabled: false`) on a host with no local Ollama or one that has
+not pulled the vision model.
+
+When enabled and no cloud candidate is selected, `_run_local_vision` sends the
+original message list — with `image_url` parts intact — to the configured
+Ollama model. The local answer then **replaces** the cloud completion:
+`_local_vision_response` returns a normal OpenAI-shaped response, including a
+stream-wrapped version when the client asked for `stream: true`. It does *not*
+inject a caption into a cloud call, because the streaming proxy cannot rewrite
+bytes mid-stream.
+
+Security and budget guards:
+- **Only inline `data:` URIs are accepted.** A remote `http(s)` image URL is
+  declined, because pointing a local model at an arbitrary URL would let an
+  unauthenticated caller make the router fetch internal resources (SSRF).
+- Image count and total payload size are bounded by `max_images` and
+  `max_image_bytes` before the local call is made.
+- If the local call fails for any reason, the request falls through to the
+  ordinary `422 No model satisfies the hard filters` rather than returning a
+  silent empty response.
+
+Pull the model on whichever Ollama the fallback points at:
+
+```bash
+ollama pull qwen3-vl:4b
+```
+
 Config is strict (`extra="forbid"`): a misspelled or misplaced key fails at
 load instead of being silently ignored.
 
@@ -503,11 +594,25 @@ allowance.
 | Method | Path | Description |
 |---|---|---|
 | `GET` | `/health` | Catalog/reachability status, scoring coverage, warnings |
+| `GET` | `/metrics` | Aggregated observability JSON: quota burn, coverage, recent decisions, per-model totals, verdict mix, top proficiency; loopback-only, no auth |
 | `POST` | `/route` | Classify task, rank candidates, return selected model — **no provider call, no cost** |
 | `POST` | `/dispatch` | Same as `/route`, plus complete the provider call, stream response, log observation |
 | `GET` | `/v1/models` | OpenAI-compatible model list (router virtual models + catalog) |
 | `POST` | `/v1/chat/completions` | OpenAI-compatible completions — routes then proxies, **streaming supported** |
 
+`/metrics` returns a single JSON object with these top-level keys:
+
+- `quota` — kWh metered in the last 30 days against `objective.plan_kwh_per_period`
+- `coverage` — routable-model counts with energy/proficiency data plus warnings
+- `recent_decisions` — last 50 rows from `route_decisions`
+- `per_model` — per-model aggregates over the last 30 days of `energy_observations`
+- `verdict_mix` — counts by verification verdict over the last 7 days
+- `top_proficiency` — top models by `blended_score` for `coding_general`
+- `generated_at` — ISO8601 timestamp
+
+It exposes no conversation text, prompts, or `session_dir`; it is bound to
+loopback and unauthenticated exactly like `/health`.
+
 Input to `/route` and `/dispatch` can include `task_category`, `task_tier`,
 and `required_context_tokens` overrides — these skip the classifier, useful
 for testing routing without the classifier in the loop.
@@ -566,8 +671,37 @@ curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application
 # JSON-mode request via response_format
 curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \
   -d '{"model":"auto","messages":[{"role":"user","content":"Return a JSON object with field answer"}],"response_format":{"type":"json_object"}}'
+
+# Monitoring: aggregate router health
+curl -s localhost:8080/metrics | python -m json.tool
+
+# Terminal dashboard foreground tool (requires `textual`; runs until you press q)
+python tui.py
+
+# One-shot routing probe with no spend
+python router_cli.py "Refactor this Django view into service objects"
+python router_cli.py "Summarize this diff" --category summarization --tier 2
 ```
 
+### Monitoring
+
+Three foreground tools read the running router without spending quota:
+
+- **`GET /metrics`** — JSON summary of quota, coverage, recent routing
+  decisions, per-model usage, verdict mix, and top proficiency. No auth;
+  loopback only.
+- **`tui.py`** — Textual terminal dashboard that polls `/metrics` every few
+  seconds. Run it in a terminal with the service already up. It is a separate
+  entrypoint, not a systemd unit.
+- **`router_cli.py "<task>"`** — POSTs to `/route` once and prints the full
+  decision tree, including candidates, selected model, estimated cost, and
+  proficiency. Use `--category`, `--tier`, and `--context` to override the
+  classifier deterministically, or `--json` for raw output.
+
+`textual` is pinned in `requirements.txt` solely for `tui.py`. It is imported
+only by that module; the FastAPI service dispatch path never touches it, so
+the router itself has no UI dependency.
+
 ## Logging and Traceability
 
 One `route` line per request says what was decided; one `dispatch` line says