6krrt/plans/cockpit-brainstorm.md

# Cockpit brainstorm: the router as an operator's instrument panel

Status: reference -- brainstorm, not a queue item; its six quick wins shipped in PR #101

**Date:** 2026-09-23
**Read against:** the repo tarball as of this date (code, config, plans,
screenshots). **Not** read against the live `router.db`, so every "gap" below
is a gap in what the code surfaces, not a claim about what the data shows.
Section 5 is the exception: it was read against the live portal.

## The framing

6krrt as a local LLM router with a cockpit. The operator:

1. adjusts dials and knobs in flight,
2. sees where tokens are wasted,
3. watches proficiency, and
4. sees poor results from a carrier or model surface, so they can decide
   whether to preclude it (with the circuit breaker as the automatic version
   of that decision for outages).

The router makes the per-request decision. The cockpit is where the operator
makes the slower decisions: which models and carriers are trusted, for what,
and at what settings. Under this framing, the question for every feature is
whether it helps the operator notice something and act on it.

---

## What the cockpit already has

| job | what exists | where |
|---|---|---|
| in-flight dials | ~11 runtime knobs (bool/float/int tables in `admin.py`), persisted config writes via `/api/config/{key}`, classifier card, profiles CRUD, gaming mode | Controls, Profiles |
| knob coverage | North Star rule #1 + `test_admin_knob_coverage.py` | tests |
| token waste | pinch savings card; `cache_rate_series` + warnings; incumbent cache pricing dial; `cost_estimate_calibration` in `/metrics` | Dashboard, Controls, `/metrics` |
| proficiency | model × category matrix with evidence status (measured / inherited / thin, n per cell); client outcome log with attributable + applied flags | Proficiency |
| poor results | `content_fault_warnings` (malformed output rate); `rejection_warnings`; availability override (active / deprecated / stale); allowlist editor | warnings bell, Models, Providers |
| breaker | passive per-(model, provider) breaker on 5xx; on/off knob | Controls (toggle only) |

The foundation is good. What's mostly missing is the link between a signal
and the action it should prompt, plus any record of which actions were taken.

---

## Cross-cutting ideas (these help all four jobs)

### X1. Flight recorder: a change log

**Gap.** `admin_schema.sql` has only `admin_model_overrides`. Nothing records
when a knob, override, allowlist entry, profile or feedback fold changed, what
it changed from, or why. Runtime knobs are "NEVER persisted", so a restart
silently reverts them, and nothing notes that the revert happened.

**Idea.**
- An `admin_changes` table: `(at, surface, key, old, new, reason, source)`,
  where `source` is `runtime | persisted | override | allowlist | profile |
  feedback_fold | restart`. Every admin write path appends a row.
- An `operator_epoch` id stamped on each `route_decisions` row, incremented
  on each **operator action** (a row in `admin_changes`). Every metric can then
  be split before and after a change without joining on timestamps. Scope it
  to operator actions on purpose: "any change that affects routing" would also
  cover catalog price polls and every feedback fold, so the epoch would tick
  constantly and split nothing.
- Change markers drawn on the dashboard history charts.
- A drift badge when a runtime value differs from its persisted twin
  ("this will revert on restart").
- Restart reverts are loggable without persisting runtime knobs: the change
  log already holds each knob's last runtime value, so at startup, compare it
  with the loaded config and write one `restart` row per value discarded.

**Why first.** Adjusting knobs in flight produces no learning if you can't
later tell what a change did. The other ideas here lean on this one.

### X2. Structured warnings with actions attached

**Gap.** `/metrics` warnings are plain strings. The dashboard decides which
page fixes a warning, and how severe it is, by regex on the text
(`warningTarget` and `warningSeverity` in `index.html`). Rewording a warning
silently breaks its link and its severity.

**Idea.** Emit `{class, severity, subject: {model, provider, category?},
evidence_url, actions: [...]}`. The warning registry in
`test_tui_warnings.py` already lists every class, so it can serve as the
schema. The bell can then offer an action in place: "exclude from
`tool_use_agentic`", "quarantine 24h", "open decisions filtered to this
model".

### X3. Preview before apply (replay)

**Gap.** The reviews replayed thousands of real decisions to test a setting,
but only as one-off scripts pasted into plan documents.

**Idea.** For any change that affects ranking (`quality_tolerance`, the
incumbent dial, profile edits, availability overrides, graded exclusions
from P2 below), the portal replays the last N decisions through
`select_candidates` and `rank_candidates` with the proposed value, then shows:
- a winner-shift matrix: old winner → new winner, with counts,
- estimated cost delta,
- decisions that would become unroutable (the 2026-09-01 and 2026-09-04
  incidents were both deprecations that emptied a candidate set).

This is cheap because the ranking modules are pure. Its output is an estimate
of routing change, not quality change, and should be labeled that way.

---

## 1. Dials in flight

- **D1. Timed changes.** "Apply for 2h, then revert." Turns a knob change into
  a bounded experiment and lowers the risk of forgetting a test setting.
  Needs X1 to record both the apply and the revert.
- **D2. Blast radius on hover.** How many of the last 24h of decisions this
  knob would have touched. It's the X3 replay reduced to one number.
- **D3. Session-scoped A/B.** Assign new sessions to setting A or B and
  compare cost and outcome rate per arm. It has to be per session, not per
  turn, because a mid-session switch costs cache (token-waste Wave 2
  measured 0.919 → 0.348). Larger build; only worth it once X1 exists and
  outcome volume can support a comparison.
- **D4. Grouping by effect.** Group controls by what they move (cost,
  quality, latency, safety, measurement-only) rather than by config section,
  and mark the few that actually change routing. Knob coverage guarantees
  every control exists; this makes them usable.

## 2. Token waste

**Gap.** Waste signals are spread across pinch, cache rate, calibration and
the verifications table, and none is in dollars in one place.
`cost_estimate_calibration` is in `/metrics` but not in the portal.

- **W1. Waste ledger.** One panel with dollars per waste class over a window,
  each with a trend:
  | class | how it's computed |
  |---|---|
  | switch cache loss | billed on switch turns − same prompt at the session's same-model cache rate |
  | retries | re-billed prompt tokens from `iteration.py` attempts |
  | paid for failure | billed cost of responses later reported `ok:false` via `/outcome` |
  | empty-200 / malformed | billed cost of responses with a structural `malformed` verdict |
  | truncation | responses stopped by `finish_reason: length` with no client cap |
  | pinch (negative) | dollars saved, from `pinch_summary` |

  "Paid for failure" counts only decisions that received an outcome, so it
  shows n and outcome coverage (the share of billed decisions with a report)
  next to the dollar figure.
- **W2. Why did it switch?** Record a switch reason on each decision: category
  changed, breaker open, exploration, override removed the incumbent, profile
  change. Then rank sessions by switch cost and drill into the turns. Without
  a reason, a switch is only a cost; with one, it points to a knob.
- **W3. Calibration panel.** Surface `cost_estimate_calibration` per model,
  headlining the **spread** (as its docstring argues), and flag pairs where
  the estimator and the bill disagree on order. Where that happens, the cost
  tiebreak picks the wrong model.
- **W4. Cost per successful outcome.** Per (model, provider, category):
  billed dollars ÷ `ok:true` outcomes. A cheap model that fails often isn't
  cheap. This is the one waste figure that includes quality.

  Show n and outcome coverage next to it. Models that carry more traffic
  collect more outcomes (the same exposure bias the feedback fold had to
  correct), and if coverage stays hidden, a thinly reported cheap model will
  look better than it is.

## 3. Proficiency monitoring

**Gap.** The matrix shows current state well. It doesn't show movement, and
it doesn't show which thin cells actually matter.

- **P1. Trend and drift per cell.** A sparkline of the outcome rate over time.
  Compare a recent window against the long-run rate with an interval, and
  flag drops that fall outside it. Carriers change serving setups (quant,
  engine, hardware) without notice, and a drift flag is how that shows up.
- **P2. Evidence priority.** Rank cells by `traffic share × uncertainty`. A
  thin cell that routing never consults doesn't matter; a thin cell deciding
  30% of traffic does. The ranking tells you where to spend `eval_proficiency`
  runs or exploration budget.
- **P3. The tolerance band per category.** Show which models sit within
  `quality_tolerance` of the leader, so it's clear where cost is deciding and
  where quality is.
- **P4. Label provenance per cell.** The share of each cell's outcomes whose
  category came from a fresh classification versus a cached or borrowed one.
  North Star rule #2 exists because borrowed labels trained the matrix;
  this makes it visible cell by cell.

## 4. Poor results → operator preclusion (and the breaker)

**Gap.** The operator's only preclusion tools are binary: an availability
override (active / deprecated / stale) or removing a model from the
allowlist. The breaker is in-memory, trips only on 5xx, forgets everything
on restart, and shows nothing but an on/off toggle.

- **Q1. Carrier/model scorecard.** One sortable table per (model, provider)
  over a window: outcome fail rate (with n), malformed / empty-200 rate,
  5xx and timeout count, breaker trips, p95 latency, estimate-vs-bill error,
  cache rate, cost per success (W4). This is the "who's misbehaving" view the
  preclusion decision needs.
- **Q2. Same model, different carriers.** Group scorecard rows by base model,
  e.g. `qwen3.6-35b` on NeuralWatt vs OpenRouter. This separates "the model is
  bad at this" from "this carrier serves it badly", which calls for a
  different fix: drop the carrier's row, or restrict the model.
- **Q3. Graded actions** in place of deprecate-or-not. Each action records a
  reason and links its evidence in X1:
  1. exclude from category X. This **cannot** reuse `eligible_categories`:
     that field is restrict-only (`admin.py`, the probe docstring), and NULL
     means "every category", so a deny needs its own column,
  2. exclude when the request carries tools (the `deepseek-v4-flash` case).
     This may be the way out of the frozen `tool_use_agentic` data:
     `min_tool_proficiency` can only read scores that stopped moving on
     2026-09-15, while a per-model manual rule lets the operator decide from
     Q1 scorecard evidence instead,
  3. restrict to batch / `-flex` only,
  4. quarantine: exploration-only, so it keeps collecting evidence without
     carrying traffic (see the open question below; deferred until Q1 shows
     it is needed),
  5. timed ban, e.g. 24h, auto-expiring and logged,
  6. deprecate (what exists today).

  Each action goes through at least the minimal X3 check (would this empty a
  candidate set for any recent decision?) before committing. That check alone
  would have caught both the 2026-09-01 and 2026-09-04 incidents; the full
  winner-shift replay can come later.
- **Q4. Breaker visibility.** Show open circuits with `down_until`, current
  cooldown and trip count; keep a trip history (persist it, since `_store`
  is process-lifetime); add manual force-open (drain a model on purpose) and
  force-close (reset after a known fix).
- **Q5. A quality breaker, separate from availability.** Trip on content
  failures (empty-200, mangled output, a burst of `ok:false`) using the
  novelty-or-rate rule already in `rejection_warnings`. This is roughly what
  `plans/mangled-output-detection.md` specs (status: planned). Keep it a
  separate breaker and panel so a quality trip doesn't read as an outage.
  **One doc owns it:** fold Q5 into `mangled-output-detection.md` rather than
  specifying it twice, and let this file point there.
- **Q6. Carrier-level breaker.** When several models on one provider trip
  inside a short window, open the provider rather than walking its models one
  cooldown at a time. Account exhaustion already has its own path; this is for
  partial carrier outages.

---

## 5. Quality of life: ergonomics and readouts

Unlike the sections above, this one **was** read against the live portal
(8080, view-only, 2026-09-23), plus the frontend source. Each item names the
thing observed.

### Readouts that mislead today

- **R1. Home tiles use different windows.** Quota is "this period",
  Decisions 7d, Models and Pinch 30d. Busiest models shows `kimi-k2.7-code`
  at 10,028 while the Decisions tile says 2,887 for everything, and both are
  correct. Label each tile's window on its face, or drive all of them from
  the Activity card's 24h / 7d / 30d selector.
- **R2. The Decisions tile counts verifications, and mixes diagnostics with
  ground truth.** It reads "2,887 verified in 7 days: 132 ok, 93 failed", but
  `index.html` sums `verdict_mix`, which counts `verifications` rows, not
  decisions. Checked read-only against the live DB on 2026-09-23:
  | | tile | actually |
  |---|---|---|
  | headline | 2,887 | 2,678 route decisions; 2,887 is verification rows, 2,660 of them `unverifiable` |
  | "ok" | 132 | 96 client `succeeded` + 29 `local_llm` ok + 7 structural ok |
  | "failed" | 93 | 82 client `failed` + 11 `malformed` (local_llm + structural) |
  The tile adds structural and `local_llm` verdicts, which CLAUDE.md calls
  diagnostics only, to `/outcome` reports, the only ground truth. Show route
  decisions as the headline, then client outcomes on their own: "178 client
  reports (6.6%): 96 ok, 82 failed (46%)". Diagnostics, if shown at all, go on
  a separate line.
- **R3. The Activity chart smooths across gaps.** The decisions series is a
  spline through sparse points, so it draws continuous traffic across hours
  that had none. Break the line at empty buckets or use bars. Separately,
  billed requests exceed decisions at several peaks; the chart should say what
  the excess is (cloud classifier calls? retries? unrouted pins?) rather than
  leave two lines that disagree unexplained.
- **R4. Sparklines have no values.** The tile sparklines carry no axis and no
  hover. Add a hover value, or min and max labels.
- **R5. The Classifier card flashes a wrong value.** On first paint the mode
  select reads `local_llm` and only switches to `local_encoder` once config
  loads. For a second, the card shows a value that isn't configured. Render
  a loading state instead of the default.
- **R6. Config echo vs live state.** The Classifier card shows what the
  overlay says (`local_encoder`, device `cpu`), not what the process actually
  loaded. Show the resolved model, device and recent p50 latency from the
  running classifier. (It also exposed doc drift: CLAUDE.md says the live
  deployment runs `device: cuda`; the overlay says `cpu`.)
- **R7. Proficiency colour encodes evidence, not score.** Green / amber mean
  measured / thin, so the best model in a column isn't visible without reading
  every number. Mark each column's leader, outline the `quality_tolerance`
  band (P3), make columns sortable, and add a "routable only" toggle so rows
  that are deprecated or not allowlisted stop padding the matrix.

### Decisions page ergonomics

- **E1. Collapse runs.** The top 40 rows are one session: same category,
  profile, tier, source and model, and only ctx and cost move. Fold
  consecutive same-session, same-model rows into one expandable row: "38 turns,
  ctx 75k to 98k, $0.27 total". Switches then stand out as row boundaries,
  which is what W2 wants to surface.
- **E2. Session view.** A per-session rollup: turns, total cost, context
  growth, switches, outcome count. Context climbing about 1k per turn is
  visible in the raw table and invisible everywhere else.
- **E3. Filters in the URL.** `decisions.html` never reads `URLSearchParams`,
  so no other page can link to "decisions for this model" or "this session".
  X2 actions, the Q1 scorecard and the model modal all need that link.
- **E4. Row drill-down.** A row click does nothing. Open a panel with the
  decision's rejected candidates, verification verdict, `/outcome` report,
  and estimated vs billed cost from its energy observation.
- **E5. Server-side filtering.** The page loads 1,000 of 32,540 rows and
  filters and searches only those, with a footnote saying so. Push filters to
  the API so that "All" means all.
- **E6. Model, provider and source as filters.** Today they're reachable only
  through free-text search.

### Controls page

- **C1. One knob table, not two.** Runtime Knobs and Persisted Config list
  largely the same knobs in two columns, in different orders, with persisted
  labels truncated (`objective.incumbent_cache_pri…`). Checking drift means
  matching rows by eye. Use one table: knob, live value, persisted value,
  layer (base / overlay), drift badge. That table is also X1's natural home.
- **C2. Separate Restart Service.** It sits in the same button row as Refresh
  Catalog and Seed Energy. Move it apart, and have its confirmation list which
  runtime values differ from persisted and will revert.
- **C3. Prose in cards.** Local Compute carries three paragraphs; the portal
  style rule is no paragraphs. Keep one line plus a docs link or tooltip.
- **C4. Default and last change per knob.** Show the code default beside
  each value, plus "changed 2h ago from 0.3" once X1 exists.

### Portal-wide

- **G1. Nav lives in eight files.** Each page hardcodes the same `<ul>` of
  nav links. `navbar.js` already notes that adding Quota was an eight-file
  edit; have it render the links too.
- **G2. Width.** Most pages sit in `container-xl` and use well under half of
  a wide screen, while Decisions, the densest table, is the most cramped.
  Proficiency already goes full-width. Go fluid on the data-heavy pages.
- **G3. Dismissals don't stick on rate warnings.** Dismissal is keyed on the
  warning's exact text, on purpose (a changed condition resurfaces it). But
  warnings that embed a live rate ("3.2x pace") change text every poll, so a
  dismissal never holds. X2's `class` + `subject` is the right key; resurface
  on a severity change, not a digit change.
- **G4. Keyboard.** `/` to focus search, `Esc` to close modals, and `j`/`k`
  on the Decisions table. Cheap, and this is a page someone lives in.

---

## One thing the cockpit framing makes more urgent

The portal is loopback-only with no auth, and it already includes
`/api/restart-service`, `/api/apply-feedback` and provider deletes. The more
it becomes the place decisions are made, the stronger the pull to reach it
from other machines, and moving the router onto a Proxmox box on the LAN is
exactly that move. Add auth before binding anything but `127.0.0.1`. Same
warning CLAUDE.md already gives for the API, with more at stake.

---

## A possible order

| # | item | why here |
|---|---|---|
| 0 | portal auth | **only if** the router is leaving `127.0.0.1` (the Proxmox move); blocks that move, not this list |
| 1 | X1 change log + operator epoch | everything else reads it |
| 2 | Q1 scorecard + Q4 breaker visibility | the preclusion decision needs one view first |
| 3 | Q3 graded actions (via X1) + minimal X3 (empties-a-candidate-set check) | turns the scorecard into decisions without repeating 09-01 / 09-04 |
| 4 | X2 structured warnings | lets warnings trigger Q3 actions directly |
| 5 | W1 waste ledger + W2 switch reasons | token waste in dollars, with causes |
| 6 | X3 full replay preview (winner shift, cost delta) | makes knob changes safe to try |
| 7 | P1 drift, P2 evidence priority | proficiency monitoring over time |
| 8 | Q5 quality breaker (owned by `mangled-output-detection.md`), D1 timed changes | automate what the operator has been doing by hand |
| 9 | D3 session A/B, Q6 carrier breaker | only after the above prove out |

Section 5 sits outside this order: most of it is small and independent.
Quick wins that also unblock the numbered items: **E3** (URL filters, needed
by X2 and Q1), **R2** (outcome coverage, the same honesty W1 and W4 need),
**C1** (one knob table, X1's home), **G3** (fixed by X2's class keys). R5 and
G1 are fixes of a few lines each.

**North Star #1 applies to every item here.** Q5 thresholds, D1 durations,
timed-ban lengths and any quarantine settings are new config knobs, so each
ships with its admin control or `test_admin_knob_coverage.py` fails. Scope the
control into the item, not a follow-up.

### Open questions, with proposed answers

- **Is the change log append-only history, or also an undo stack?**
  Proposed: append-only, plus a per-row "revert to old value" that writes one
  key and appends its own row. That is not the shape CLAUDE.md rejected for
  gaming mode: that concern was one flag writing five keys and drifting apart.
  A one-key revert can't drift, and the history stays intact.
- **Should graded exclusions live in the DB or in `config.local.yaml`?**
  Proposed: the DB, next to `admin_model_overrides` (extend it or add a
  sibling table). Timed bans need an expiry, and an expiry belongs in a DB
  column. The overlay is per-machine and not in git, so it's the wrong home
  for judgments about a carrier. And availability overrides already live in
  the DB. The category exclusion still needs a new deny column (see Q3.1).
- **Does quarantine conflict with session-scoped exploration?** Yes, and the
  weight goes up: a quarantined model would carry whole sessions, not single
  turns. There's a second problem too. `exploration.py` only chooses among
  models that already passed the hard filters and were ranked, so quarantine
  needs a new state, "eligible to explore, barred from winning", rather than
  reusing anything existing. Defer it until Q1 shows it's needed.