From 8b861f9267ee1bbc0f0313a4f113116d4bd7f6e9 Mon Sep 17 00:00:00 2001 From: adlee-was-taken Date: Sun, 23 Aug 2026 21:00:51 -0400 Subject: [PATCH] docs: document local vision fallback and the monitoring stack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README gains the Local Vision fallback section (the Ollama/qwen3-vl path for image requests with no cloud vision candidate — previously only the cloud capability gates were documented) plus route_decisions table, metrics.py/tui.py/router_cli.py in the module list, GET /metrics endpoint and keys, and monitoring usage snippets. CLAUDE gains metrics.py/tui.py/router_cli.py in what's-built and a Monitoring: route_decisions persistence section. Also corrects verifications.kind/verdict to match schema.sql. --- CLAUDE.md | 33 +++++++++++++ README.md | 138 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 169 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 75c9618..e3c9e7c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -242,10 +242,43 @@ rather than from months of history. StreamingResponse's generator is resumed in a fresh copy of the caller's context, so the ContextVar cannot reach it — the path all agent traffic takes. Level from `logging.level`, overridden by `LLM_ROUTER_LOG_LEVEL`. +- `metrics.py` / `GET /metrics` — read-only observability aggregation. Keeps + the dashboard helpers outside `dispatcher.py` to avoid an import cycle: + `dispatcher` imports `metrics` for `/health` and `/metrics`, so `metrics` + takes `(conn, cfg)` arguments and never imports `dispatcher`. Returns quota + burn against `plan_kwh_per_period`, scoring coverage warnings, recent + `route_decisions`, per-model aggregates over `energy_observations`, + verification verdict mix, and top proficiency by category. +- `tui.py` — Textual terminal dashboard over `GET /metrics`. A foreground + entrypoint (`python tui.py`), not a service. `textual` is imported only + here, so the router's dispatch path has no UI dependency. +- `router_cli.py` — one-shot `/route` probe. Posts a task to the running router + and prints the decision tree, or emits raw JSON with `--json`. Spends no + quota because it only routes. - `tests/` — 356 tests across 19 files, all passing, all offline. Verified on Python 3.10 and 3.14; nothing declares `requires-python`, so 3.10 is the tested floor rather than a promised one. +### Monitoring: route_decisions persistence + +`route_decisions` is the newest observability table (not a scoring input). It +records one row per routing decision — `route` \| `dispatch` \| `chat` | +`passthrough` \| `local_vision` — with category, tier, selected model, runners +up, estimated cost/proficiency, classifier latency, rejection reason, and +request feature flags (`tools`, `images`, `json_mode`, `streamed`). It stores +only a hashed `session_key`; `session_dir`, prompts, and answers are excluded, +and a test enforces that the write path never stores conversation text. + +The write is gated by `logging.log_route_decisions` and is **best-effort**: a +failed write is logged and swallowed, because a decision record is worth +having but never worth failing or slowing a request for. The table is created +from code at module load (`_ensure_route_decisions_table`) and again on every +write (`ensure_route_decisions` inside `persist_route_decision`), mirroring +the `proficiency_store.ensure_columns` migration pattern: `schema.sql` is +`CREATE TABLE IF NOT EXISTS`, but a live `router.db` predating this table +needs the code-side migration. Both the module-load hook and the write-path +guarantee are idempotent and leave existing rows intact. + ### Request-side capability gates The router treats some capabilities as hard filters, read directly from the diff --git a/README.md b/README.md index 3d75a2e..8d138a7 100644 --- a/README.md +++ b/README.md @@ -224,6 +224,9 @@ restarts on boot shouldn't change its dependency tree underneath itself. | **`leaderboard.py`** | Imports `leaderboards.yaml` priors into `proficiency`; `--check` reports gaps | Yes (DB) | | **`iteration.py`** | Retry budget per tier, and matching the retry to the failure kind | Pure | | **`config.py`** | YAML loader + Pydantic validators (blend weights sum to 1, valid tiers, endpoints separately addressable) | Yes (file) | +| **`metrics.py`** | Read-only aggregations for `/health` and `GET /metrics`: quota burn, coverage, recent decisions, per-model totals, verdict mix, top proficiency | Yes (DB) | +| **`tui.py`** | Textual terminal dashboard over `GET /metrics`; foreground tool, not a service | Yes (network) | +| **`router_cli.py`** | One-shot routing probe: POSTs to `/route` and prints the decision tree | Yes (network) | ## Decision Table Schema (SQLite) @@ -358,8 +361,8 @@ rows, so repeated sweeps accumulate into a median-across-time. | `id` | INTEGER | Autoincrement | | `model_id` / `provider` | TEXT | | | `task_category` | TEXT | Optional (prose answers may lack a category) | -| `kind` | TEXT | `structural` \| `local_llm` | -| `verdict` | TEXT | `ok` \| `truncated` \| `malformed` \| `unverifiable` | +| `kind` | TEXT | `structural` \| `local_llm` \| `client_outcome` | +| `verdict` | TEXT | `ok` \| `truncated` \| `malformed` \| `unverifiable` \| `succeeded` \| `failed` | | `detail` | TEXT | Human-readable reason | | `completion_tokens` | INTEGER | Wasted answer cost, for payoff sum | | `observed_at` | TEXT | ISO8601 | @@ -371,6 +374,44 @@ applies a 0.0 sample per failure to `proficiency`, and marks them `applied_at` for idempotency. `unverifiable` is recorded but not treated as a failure — it means the checker had nothing to say, not that the model failed. +### `route_decisions` — routing observability + +| Column | Type | Notes | +|---|---|---| +| `id` | INTEGER | Autoincrement | +| `observed_at` | TEXT | ISO8601, UTC | +| `kind` | TEXT | `route` \| `dispatch` \| `chat` \| `passthrough` \| `local_vision` | +| `task_category` | TEXT | | +| `task_tier` | INTEGER | 1–3 | +| `required_context_tokens` | INTEGER | | +| `confidence` | REAL | Classifier confidence | +| `classifier_ms` | INTEGER | Classification latency | +| `classification_source` | TEXT | `classifier` \| `override` \| `fallback` | +| `latency_tolerance` | TEXT | `interactive` \| `batch` | +| `candidates_considered` | INTEGER | How many survived hard filters | +| `selected_model` | TEXT | Null when no model was selected | +| `selected_provider` | TEXT | `neuralwatt` or `local` (vision fallback) | +| `runner_up_models` | TEXT | JSON array of up to 3 runner-up candidates | +| `est_cost_usd` | REAL | Estimated cost of the selected model | +| `est_proficiency` | REAL | Estimated proficiency for the task category | +| `rejected_reason` | TEXT | Active filters when nothing was selected | +| `session_key` | TEXT | Hashed session fingerprint ONLY | +| `tools` | INTEGER | 0/1 — request carried a `tools` array | +| `images` | INTEGER | 0/1 — request carried image parts | +| `json_mode` | INTEGER | 0/1 — request required JSON mode | +| `streamed` | INTEGER | 0/1 — response was streamed | + +`route_decisions` stores one row per routing decision so "how is routing +performing" is answerable: which model was picked, for what category/tier, +how long classification took, and — when nothing was selected — which hard +filter shut it out. It is an observability table: nothing in routing reads it. + +It stores only a hashed session fingerprint in `session_key`; `session_dir`, +prompts, and answers are deliberately excluded. A test enforces that the write +path does not store prompt or answer text. The write is gated by +`logging.log_route_decisions` and is **best-effort**: a failed write is logged +at warning and swallowed so monitoring cannot slow or fail a request. + ## Weighted Scoring Six hard filters are applied **before** scoring (not weighted — outright disqualification): @@ -408,6 +449,56 @@ settled with `POST /outcome` data rather than a 3-task benchmark score. Set it to `0.5` to turn it on. The vision and JSON-mode gates ship **on**, because a wrong guess is a guaranteed 400. +### Local Vision Fallback + +Cloud vision is not universal in the catalog, and the most economical coding +rows do not declare `supports_vision`. Routing an image request through them +would earn a provider-side 400, so when `routing.require_vision` is true only +rows with `supports_vision = 1` survive the hard filters. If **no** cloud +candidate survives, the router can fall back to a local vision model instead +of returning 422. + +`local_vision:` in `config.yaml` controls this path: + +| Key | Default | Purpose | +|---|---|---| +| `enabled` | `true` | Whether the fallback runs at all | +| `base_url` | `http://localhost:11434/v1` | OpenAI-compatible Ollama endpoint | +| `api_key_env` | `null` | Env var holding an API key, if the endpoint needs one | +| `model` | `qwen3-vl:4b` | Vision model on that Ollama | +| `timeout_seconds` | `60` | Request timeout | +| `max_images` | `4` | Refuse requests with more image parts | +| `max_image_bytes` | `9437184` (9 MiB) | Refuse requests whose image payload exceeds this | + +The fallback is **enabled by default** both in `config.yaml` and in +`LocalVisionConfig`, so omitting the section still turns it on. Disable it +explicitly (`enabled: false`) on a host with no local Ollama or one that has +not pulled the vision model. + +When enabled and no cloud candidate is selected, `_run_local_vision` sends the +original message list — with `image_url` parts intact — to the configured +Ollama model. The local answer then **replaces** the cloud completion: +`_local_vision_response` returns a normal OpenAI-shaped response, including a +stream-wrapped version when the client asked for `stream: true`. It does *not* +inject a caption into a cloud call, because the streaming proxy cannot rewrite +bytes mid-stream. + +Security and budget guards: +- **Only inline `data:` URIs are accepted.** A remote `http(s)` image URL is + declined, because pointing a local model at an arbitrary URL would let an + unauthenticated caller make the router fetch internal resources (SSRF). +- Image count and total payload size are bounded by `max_images` and + `max_image_bytes` before the local call is made. +- If the local call fails for any reason, the request falls through to the + ordinary `422 No model satisfies the hard filters` rather than returning a + silent empty response. + +Pull the model on whichever Ollama the fallback points at: + +```bash +ollama pull qwen3-vl:4b +``` + Config is strict (`extra="forbid"`): a misspelled or misplaced key fails at load instead of being silently ignored. @@ -503,11 +594,25 @@ allowance. | Method | Path | Description | |---|---|---| | `GET` | `/health` | Catalog/reachability status, scoring coverage, warnings | +| `GET` | `/metrics` | Aggregated observability JSON: quota burn, coverage, recent decisions, per-model totals, verdict mix, top proficiency; loopback-only, no auth | | `POST` | `/route` | Classify task, rank candidates, return selected model — **no provider call, no cost** | | `POST` | `/dispatch` | Same as `/route`, plus complete the provider call, stream response, log observation | | `GET` | `/v1/models` | OpenAI-compatible model list (router virtual models + catalog) | | `POST` | `/v1/chat/completions` | OpenAI-compatible completions — routes then proxies, **streaming supported** | +`/metrics` returns a single JSON object with these top-level keys: + +- `quota` — kWh metered in the last 30 days against `objective.plan_kwh_per_period` +- `coverage` — routable-model counts with energy/proficiency data plus warnings +- `recent_decisions` — last 50 rows from `route_decisions` +- `per_model` — per-model aggregates over the last 30 days of `energy_observations` +- `verdict_mix` — counts by verification verdict over the last 7 days +- `top_proficiency` — top models by `blended_score` for `coding_general` +- `generated_at` — ISO8601 timestamp + +It exposes no conversation text, prompts, or `session_dir`; it is bound to +loopback and unauthenticated exactly like `/health`. + Input to `/route` and `/dispatch` can include `task_category`, `task_tier`, and `required_context_tokens` overrides — these skip the classifier, useful for testing routing without the classifier in the loop. @@ -566,8 +671,37 @@ curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application # JSON-mode request via response_format curl -s -X POST localhost:8080/v1/chat/completions -H 'content-type: application/json' \ -d '{"model":"auto","messages":[{"role":"user","content":"Return a JSON object with field answer"}],"response_format":{"type":"json_object"}}' + +# Monitoring: aggregate router health +curl -s localhost:8080/metrics | python -m json.tool + +# Terminal dashboard foreground tool (requires `textual`; runs until you press q) +python tui.py + +# One-shot routing probe with no spend +python router_cli.py "Refactor this Django view into service objects" +python router_cli.py "Summarize this diff" --category summarization --tier 2 ``` +### Monitoring + +Three foreground tools read the running router without spending quota: + +- **`GET /metrics`** — JSON summary of quota, coverage, recent routing + decisions, per-model usage, verdict mix, and top proficiency. No auth; + loopback only. +- **`tui.py`** — Textual terminal dashboard that polls `/metrics` every few + seconds. Run it in a terminal with the service already up. It is a separate + entrypoint, not a systemd unit. +- **`router_cli.py ""`** — POSTs to `/route` once and prints the full + decision tree, including candidates, selected model, estimated cost, and + proficiency. Use `--category`, `--tier`, and `--context` to override the + classifier deterministically, or `--json` for raw output. + +`textual` is pinned in `requirements.txt` solely for `tui.py`. It is imported +only by that module; the FastAPI service dispatch path never touches it, so +the router itself has no UI dependency. + ## Logging and Traceability One `route` line per request says what was decided; one `dispatch` line says -- 2.49.1