6krrt/plans/admin-portal-functional-sweep.md

# Admin portal functional sweep -- 2026-09-08

Status: in progress -- 12 findings; #1-#8, #10-#12 fixed, #9 was not a defect

Every page, control, and API endpoint under `/admin` exercised end to end
against real data, on a throwaway instance bound to 8081.

## How it was tested

```
sqlite3 /home/alee/Sources/6krrt/router.db \
  "VACUUM INTO '<worktree>/router.db'"          # real catalog + 23,844 decisions
PYTHONPATH=<worktree>/src HF_HOME=<repo>/.hf-cache \
  .venv/bin/uvicorn dispatcher:app --host 127.0.0.1 --port 8081
```

cwd = the worktree, so `config/config.yaml`, `router.db` and the admin
overlay all resolve inside it. Production 8080 and
`/home/alee/Sources/6krrt/config/config.local.yaml` were untouched --
verified by md5 before and after (`1ec1734...`, unchanged) and by 8080
answering 200 throughout.

**Not run, deliberately.** `Seed Energy` spends real NeuralWatt credits
(`?samples=5`, 65+ billed completions) and `Restart Service` shells
`systemctl --user restart llm-router.service`, which is the *production*
unit regardless of which instance served the click. Both were read in
source instead.

## Findings

Ordered by consequence. Every item was reproduced, not inferred.

### 1. Profile edit silently drops an `allowed_model_ids` entry that is not in the current catalog

`admin/frontend/profiles.html`

The editor's `allowed_model_ids` control is a `<select multiple>` populated
from the 446-row catalog. An id stored in the profile that has no matching
`<option>` is simply not selected -- and Save posts the selection, so the
stored value is gone.

Reproduced: a profile holding `allowed_model_ids: ["totally-not-a-model"]`
was opened with Edit and saved **with no changes**. It went from
`admits 0 models` (warning triangle) to `no filters / admits 40 models`.

This fails **open**. The realistic trigger is a model that left the catalog
-- deprecated by the poller, renamed upstream, or pruned by the OpenRouter
allowlist -- at which point an unrelated edit to `min_tier` quietly converts
a locked-down profile into one that admits everything. Nothing warns.

Ids that *are* in the catalog round-trip correctly (verified: `gemma-4-31b`,
`kimi-k3` came back preselected), so this only bites on drift.

### 2. Admin request bodies are plain `BaseModel`, so unknown keys are ignored

`src/admin.py` -- `_CloudFallbackBody`, `_ProfileCreateBody`,
`_ProfileUpdateBody`, `_ProviderUpdateBody`, `_AvailabilityBody`,
`_AllowlistAddBody`, `_ValueBody`

CLAUDE.md already documents this exact hazard and closed it for config
models via `StrictModel(extra="forbid")`. The admin API surface never got
the same treatment.

Worst instance is destructive:

```
POST /admin/api/cloud-fallback-config
{"base_url":"https://api.neuralwatt.com/v1","model":"deepseek-v4-flash", ...}
-> 200 {"message":"A restart is required for this change to take effect"}
```

The body is nested (`{"cloud_fallback": {...}}`), so every flat key is
discarded, `cloud_fallback` defaults to `None`, and `None` is the documented
*remove the block* signal. A configured cloud fallback is **deleted** and the
caller is told the save succeeded. Confirmed: overlay went from a populated
`classifier.cloud_fallback` to no such key, with a 200 both times.

Note the asymmetry -- the inner dict *is* validated strictly
(`{"timeout_secondz":20}` -> 422 "Extra inputs are not permitted"); only the
outer wrapper is loose.

Same class, non-destructive: `POST /admin/api/profiles/`
`{"name":"typoprof","min_teir":3}` returns 200 and creates a profile with
every filter null.

### 3. The `stale` availability override does nothing

`admin/frontend/models.html` offers active / deprecated / stale. Only
`deprecated` is ever read -- `metrics._admin_deprecated_models` and
`dispatcher._admin_deprecated_models` both query
`WHERE availability = 'deprecated'`.

Measured on `/route` (`coding_refactor`, tier 1, 5k ctx):

| state | `candidates_considered` |
|---|---|
| `glm-5.2-fast` overridden to **stale** | 27 |
| stale override cleared | 27 |
| `gemma-4-31b` overridden to **deprecated** | 27 |
| deprecated override cleared | **28** |

Deprecated works. Stale is a control that persists a row and changes
nothing.

### 4. `/v1/models` advertises admin-deprecated models, and pins against them are not refused

`src/dispatcher.py:3645` reads `models.availability = 'active'` straight from
the table and never consults `admin_model_overrides`.
`_resolve_pinned_provider` has no override check either.

With `gemma-4-31b` admin-deprecated, it stayed in `/v1/models` (51 entries)
while `/route` correctly excluded it. So a client enumerating the endpoint
sees a model the operator has switched off, pins it, and gets it dispatched.

The same function already hides `ollama-local` rows under gaming mode, with
a comment giving the reason: listing one would "advertise a model every pin
against it is about to be refused for". The override case has neither the
hiding nor the refusal.

### 5. Decisions search is silently scoped to the newest 1000 rows

`admin/frontend/decisions.html`. Subtitle: "Full routing decision history --
filter, search, and inspect".

The page loads a fixed 1000-row window; every filter and the search box
operate on that client-side slice. The counter reads "N of 1000", which
looks like a total.

- `route_decisions` holds **23,844** rows
- **10,394** of them name a kimi model
- searching `kimi` on the page reports **"0 of 1000"**

A confident zero on a page that claims to hold the full history. The API
itself is fine (`/admin/api/decisions?limit=5000` returned 2000 rows), so
this is a frontend windowing choice, not a backend limit.

Also on this page: rows are not clickable and there is no detail affordance,
though the subtitle promises "inspect"; the FLAGS column has no legend.

### 6. Two reciprocal navigation gaps

- `admin/frontend/providers.html` has no `href="/admin/proficiency"`
- `admin/frontend/proficiency.html` has no `href="/admin/providers"`

Every other page carries both. Net effect: you cannot get from Providers to
Proficiency or back without editing the URL.

### 7. `POST /admin/api/providers/{name}` has no read-only guard, and the result can be unremovable

`src/admin.py:1378`. `admin_provider_update` validates the URL and the model,
then persists -- with no provenance check at all. Compare
`admin_provider_delete` (403 for a base-config provider) and
`admin_profile_update` (403, "add an overlay profile with the same name to
override it").

Editing `neuralwatt` -- shown in the UI as `base config` / `read-only` --
returned 200 and wrote a full `dispatch_providers.neuralwatt` block into
`config.local.yaml`.

The trap is what happens next: `DELETE /admin/api/providers/neuralwatt`
refuses with 422 because it is `dispatch_settings.default_provider`. The
overlay entry is now **unremovable through the portal**; the only way out is
hand-editing `config.local.yaml`, the one file in this deployment that is
not in git.

The UI hides the edit control for base-config providers, so this is
API-reachable only -- but the portal is the only thing that writes that file.

### 8. `cloud_fallback.base_url` is not URL-validated

```
POST /admin/api/cloud-fallback-config
{"cloud_fallback":{"base_url":"notaurl","model":"x", ...}}   -> 200
```

and `base_url: notaurl` lands in the overlay. `POST /api/providers/{name}`
rejects `ftp://nope` with 422 via `_validate_provider_url`; the same check is
absent here, and this block is what the router falls back to when the local
classifier dies.

### 9. Allowlist delete always reports success

`DELETE /admin/api/providers/openrouter/allowlist/nope/nope` -> 200
`{"deleted":"nope/nope"}`. Bare `DELETE`, no rowcount check. Profile delete
and model-override delete both 404 correctly on a missing target.

### 10. Allowlist add cannot update a note

`INSERT OR IGNORE`. Re-adding `sandbox/test-model` with `note:"dup"`
returned 200 carrying the **original** note. There is no way to correct a
note through the API or the UI.

## Smaller things, all reproduced

- **"Apply Feedback" only previews.** `controls.html:541` sends
  `?dry_run=true`. The button says Apply; there is no way to actually apply
  from the portal.
- **"Seed Energy" spends money with no confirmation** and is styled
  identically (`btn-success`) to the free Refresh Catalog next to it.
  `?samples=5` across the catalog is 65+ billed completions.
- **"Restart Service" has no confirmation.** It is `btn-danger`, which
  helps, but it drops in-flight requests on one click -- while deleting a
  profile requires a two-step confirm.
- **`Manage` on a provider that has no allowlist is a dead end.**
  neuralwatt's panel opens, says "No models on the allowlist" (the truth is
  "this provider has no allowlist"), and Browse catalog preview fails with a
  generic "Could not load catalog preview." The real reason -- 409
  "allowlist is not required for this provider" -- never reaches the user.
  The summary line "0 allowed" reads as *zero permitted* when it means
  *unrestricted*.
- **The profile editor's model picker offers all 446 rows**, including
  allowlist-pruned and deprecated OpenRouter models, with no filter. The
  Models page has a "Show deprecated / stale" toggle defaulting to off;
  this picker has none, so picking one silently yields a zero-admit profile.
- **Allowlist catalog search resets after every Add**, so adding several
  models from one search means retyping it each time.
- **Allowlist Remove has no confirm step**, unlike profile Delete.
- **Gaming-mode 409 after saving cloud fallback in the portal** tells you to
  "uncomment the classifier.cloud_fallback example in config/config.yaml"
  rather than "restart to pick up the block you just saved". `cfg` binds at
  import, so the refusal itself is correct; the message points at the wrong
  remedy.
- **Switching classifier mode away from `local_encoder` leaves the
  `encoder:` block in the overlay.** Only `cloud_primary` is cleaned
  (`deletions`). Harmless -- validators accept it -- but it is stale state.
- Overlay YAML writes nulls as bare `provider:` keys (ruamel). Parses fine,
  reads oddly.

## What works

Stated because it is most of the surface, and because the validation in
particular is unusually good.

**Pages.** All seven render and populate from real data. Dashboard
Calls/Cost/Energy toggle, all five history ranges, the warnings popover, the
quota strip, verdict mix, category breakdown. Models list + "Show deprecated
/ stale" (45 -> 446). Decisions filters (kind, category, tier, search),
pagination, page-size. Proficiency matrix, category column-narrowing,
model search, only-measured, sort, and the source control's cell dimming
(413/495 cells at `outcome_blended`) with a legend that distinguishes
measured from inherited.

**Profiles.** Create, duplicate-with-prefill (`onlycheaps` ->
`onlycheaps-copy`), inline edit, two-step delete, live admission counts that
match `select_candidates`, the zero-admit warning chip, and the
restart-required banner. Built-ins correctly read-only.

**Providers.** Create, delete, allowlist add/remove, live catalog preview
(422 models), search filter, "Added" state on entries already allowlisted.

**Model overrides.** Set and clear, including slash-bearing ids
(`deepseek/deepseek-v4-flash`), with `effective_availability` /
`is_overridden` reported correctly and deprecation genuinely removing the
model from routing.

**Controls.** All nine runtime knobs flip and read back. All twelve
persisted config keys write to the overlay with correct `base` / `overlay`
provenance.

**Error paths** -- every one of these returned the right status with a
message that explains itself:

| probe | result |
|---|---|
| `min_tier: 3, max_tier: 1` | 422, names both values |
| `latency_tolerance: "turbo"` | 422 |
| `logging.level: "screaming"` | 422, lists the four valid values |
| `quality_tolerance: -5` | 422, "must be in [0, 1)" |
| `objective.credit_attenuation.enabled` | 403, not editable (matches CLAUDE.md) |
| `default_profile: "ghostprofile"` | 422, lists the five real profiles |
| `confidence_threshold: 80` | 422, "returns a 0-1 probability, not a percent" |
| `encoder.device: "tpu"` | 422 |
| `cloud_llm` with neither primary nor auto | 422 |
| gaming mode without `cloud_fallback` | 409 (runtime) **and** 422 (persisted) |
| duplicate profile / builtin profile / missing profile | 409 / 403 / 404 |
| delete the current `default_profile` | 422, says to repoint it first |
| `history?range=bogus` | 400 |
| allowlist ops on a non-allowlist provider | 409 |
| unknown provider | 404 |

**Triggers.** `Refresh Catalog` ran clean (19 NeuralWatt + 29 OpenRouter
upserted, 393 pruned by the allowlist) and surfaced the catalog sanity-floor
warning CLAUDE.md asks for: "[openrouter] fetched only 29 models (current DB
has 425); proceeding but catalog may be truncated". `Apply Feedback`
(dry-run) produced a correct per-model failure/success tally.

## Two more, found while fixing #1

### 11. The model picker offered every deprecated row

The `allowed_model_ids` control was populated from an unfiltered
`/admin/api/models`: 446 options, of which 393 were OpenRouter rows the
provider allowlist had already pruned and the poller had marked
`deprecated`. Picking one produced a profile that admits nothing, with
nothing on screen saying why. models.html filtered the same payload behind a
"Show deprecated / stale" toggle defaulting off; the picker did not.

### 12. `config/config.local.yaml` breaks 9 tests

Not a portal bug, but it bites anyone running the suite on a configured
machine. Reproduced both directions on the same tree:

| overlay present | result |
|---|---|
| no | 1523 passed |
| yes (`classifier.mode: local_encoder`) | 9 failed, 1514 passed |

The nine are in `test_classifier_backoff.py` (5), `test_gaming_mode.py` (3)
and `test_classifier_modes_dispatch.py` (1) -- e.g.
`assert 'local_encoder' == 'local_llm'`. They load the repo's real
`config/config.yaml` and pick up the machine-local overlay beside it, so a
deployment that has configured `classifier.mode` cannot run its own tests
clean. This deployment has exactly that overlay, so `pytest` fails 9 today
regardless of any change.

## Fixed on `fix/admin-profile-model-visibility`

Findings **#1**, **#11**, and the picker half of the smaller items:

- `GET /admin/api/models` now defaults to routable rows only
  (50 here, of which the 30 OpenRouter ones are exactly the 30 allowlist
  entries). `?include_unroutable=true` returns all 446, and models.html --
  the availability table, the one page that legitimately wants them -- is
  the only caller that passes it. Every present and future picker gets the
  filtered default. Allowlist a model, poll, and it appears.
- The profile detail pane renders `admitted_models` as chips instead of
  hiding it in a `title=` tooltip truncated to 8 of 40.
- The `allowed_model_ids` control is a two-pane Available / Selected picker.
  What is chosen is always on screen, filtering narrows Available only,
  "Add all" takes the whole filtered set, and a stored id that is no longer
  routable stays in Selected flagged `not routable` -- which is what closes
  the silent-drop in #1: opening Edit and saving untouched now preserves it.
- The list summary says `2 models` rather than `1 filter`.

### The rest, in the order they were done

- **#2** `6b9826c` -- every admin body inherits `_AdminBody`
  (`extra="forbid"`). Immediately found a real client/server mismatch: the
  profiles UI sent `name` on the update path, which the body model has no
  field for. It had been silently dropped.
- **#7** `6b9826c` -- `POST /api/providers/{name}` refuses a base-only
  definition, as delete already did. Also corrected both profile 403s,
  which advised "add an overlay profile with the same name to override it"
  -- something `admin_profile_create` rejects with a 409.
- **#3 / #4** `43645d4` -- both off-states exclude, and the override binds
  `/v1/models` and pinned requests as well as `/route`.
  `_admin_deprecated_models` renamed `_admin_excluded_models`, since
  "deprecated" no longer describes what it returns.
- **#8 / #10** `281f4fb` -- `_validate_provider_url` applied to
  `cloud_fallback` and `cloud_primary`; allowlist add upserts so a note can
  be corrected.
- **#5 / #6** `48df93b` -- `X-Total-Count` on the decisions endpoint, and
  the page names the window and the history separately. Both nav gaps
  closed, with a test that derives the page list so a new admin page fails
  until it is linked.
- **#12** `80d0e6a` -- `ROUTER_IGNORE_LOCAL_CONFIG`, set in
  `tests/conftest.py`. 1613 tests now pass identically with and without the
  machine's overlay in place.

### #9 was not a defect

The sweep flagged the allowlist DELETE returning 200 for an absent entry as
a false confirmation, and the fix was written -- then reverted, because it
broke `test_allowlist_delete_idempotent`, a test that pins that exact
behaviour with a docstring saying so. That is a decision, not an oversight.

The inconsistency it noticed is still real and is the one open question
left here: profile delete and model-override delete both 404 on a missing
target while this one does not. Which way that resolves is a call to make,
not a bug to fix.

### Smaller items still open

The nine UX items listed above are unaddressed except the picker ones: the
"Apply Feedback" button still only previews, "Seed Energy" still spends
money with no confirmation, `Manage` on a non-allowlist provider still
opens a dead-end panel, and the decisions FLAGS column still has no legend.