6krrt/plans/admin-portal-functional-sweep.md

Admin portal functional sweep -- 2026-09-08

Status: in progress -- 12 findings; #1-#8, #10-#12 fixed, #9 was not a defect

Every page, control, and API endpoint under /admin exercised end to end against real data, on a throwaway instance bound to 8081.

How it was tested

sqlite3 /home/alee/Sources/6krrt/router.db \
  "VACUUM INTO '<worktree>/router.db'"          # real catalog + 23,844 decisions
PYTHONPATH=<worktree>/src HF_HOME=<repo>/.hf-cache \
  .venv/bin/uvicorn dispatcher:app --host 127.0.0.1 --port 8081

cwd = the worktree, so config/config.yaml, router.db and the admin overlay all resolve inside it. Production 8080 and /home/alee/Sources/6krrt/config/config.local.yaml were untouched -- verified by md5 before and after (1ec1734..., unchanged) and by 8080 answering 200 throughout.

Not run, deliberately. Seed Energy spends real NeuralWatt credits (?samples=5, 65+ billed completions) and Restart Service shells systemctl --user restart llm-router.service, which is the production unit regardless of which instance served the click. Both were read in source instead.

Findings

Ordered by consequence. Every item was reproduced, not inferred.

1. Profile edit silently drops an allowed_model_ids entry that is not in the current catalog

admin/frontend/profiles.html

The editor's allowed_model_ids control is a <select multiple> populated from the 446-row catalog. An id stored in the profile that has no matching <option> is simply not selected -- and Save posts the selection, so the stored value is gone.

Reproduced: a profile holding allowed_model_ids: ["totally-not-a-model"] was opened with Edit and saved with no changes. It went from admits 0 models (warning triangle) to no filters / admits 40 models.

This fails open. The realistic trigger is a model that left the catalog -- deprecated by the poller, renamed upstream, or pruned by the OpenRouter allowlist -- at which point an unrelated edit to min_tier quietly converts a locked-down profile into one that admits everything. Nothing warns.

Ids that are in the catalog round-trip correctly (verified: gemma-4-31b, kimi-k3 came back preselected), so this only bites on drift.

2. Admin request bodies are plain BaseModel, so unknown keys are ignored

src/admin.py -- _CloudFallbackBody, _ProfileCreateBody, _ProfileUpdateBody, _ProviderUpdateBody, _AvailabilityBody, _AllowlistAddBody, _ValueBody

CLAUDE.md already documents this exact hazard and closed it for config models via StrictModel(extra="forbid"). The admin API surface never got the same treatment.

Worst instance is destructive:

POST /admin/api/cloud-fallback-config
{"base_url":"https://api.neuralwatt.com/v1","model":"deepseek-v4-flash", ...}
-> 200 {"message":"A restart is required for this change to take effect"}

The body is nested ({"cloud_fallback": {...}}), so every flat key is discarded, cloud_fallback defaults to None, and None is the documented remove the block signal. A configured cloud fallback is deleted and the caller is told the save succeeded. Confirmed: overlay went from a populated classifier.cloud_fallback to no such key, with a 200 both times.

Note the asymmetry -- the inner dict is validated strictly ({"timeout_secondz":20} -> 422 "Extra inputs are not permitted"); only the outer wrapper is loose.

Same class, non-destructive: POST /admin/api/profiles/ {"name":"typoprof","min_teir":3} returns 200 and creates a profile with every filter null.

3. The stale availability override does nothing

admin/frontend/models.html offers active / deprecated / stale. Only deprecated is ever read -- metrics._admin_deprecated_models and dispatcher._admin_deprecated_models both query WHERE availability = 'deprecated'.

Measured on /route (coding_refactor, tier 1, 5k ctx):

state	candidates_considered
glm-5.2-fast overridden to stale	27
stale override cleared	27
gemma-4-31b overridden to deprecated	27
deprecated override cleared	28

Deprecated works. Stale is a control that persists a row and changes nothing.

4. /v1/models advertises admin-deprecated models, and pins against them are not refused

src/dispatcher.py:3645 reads models.availability = 'active' straight from the table and never consults admin_model_overrides. _resolve_pinned_provider has no override check either.

With gemma-4-31b admin-deprecated, it stayed in /v1/models (51 entries) while /route correctly excluded it. So a client enumerating the endpoint sees a model the operator has switched off, pins it, and gets it dispatched.

The same function already hides ollama-local rows under gaming mode, with a comment giving the reason: listing one would "advertise a model every pin against it is about to be refused for". The override case has neither the hiding nor the refusal.

5. Decisions search is silently scoped to the newest 1000 rows

admin/frontend/decisions.html. Subtitle: "Full routing decision history -- filter, search, and inspect".

The page loads a fixed 1000-row window; every filter and the search box operate on that client-side slice. The counter reads "N of 1000", which looks like a total.

• route_decisions holds 23,844 rows
• 10,394 of them name a kimi model
• searching kimi on the page reports "0 of 1000"

A confident zero on a page that claims to hold the full history. The API itself is fine (/admin/api/decisions?limit=5000 returned 2000 rows), so this is a frontend windowing choice, not a backend limit.

Also on this page: rows are not clickable and there is no detail affordance, though the subtitle promises "inspect"; the FLAGS column has no legend.

6. Two reciprocal navigation gaps

• admin/frontend/providers.html has no href="/admin/proficiency"
• admin/frontend/proficiency.html has no href="/admin/providers"

Every other page carries both. Net effect: you cannot get from Providers to Proficiency or back without editing the URL.

7. POST /admin/api/providers/{name} has no read-only guard, and the result can be unremovable

src/admin.py:1378. admin_provider_update validates the URL and the model, then persists -- with no provenance check at all. Compare admin_provider_delete (403 for a base-config provider) and admin_profile_update (403, "add an overlay profile with the same name to override it").

Editing neuralwatt -- shown in the UI as base config / read-only -- returned 200 and wrote a full dispatch_providers.neuralwatt block into config.local.yaml.

The trap is what happens next: DELETE /admin/api/providers/neuralwatt refuses with 422 because it is dispatch_settings.default_provider. The overlay entry is now unremovable through the portal; the only way out is hand-editing config.local.yaml, the one file in this deployment that is not in git.

The UI hides the edit control for base-config providers, so this is API-reachable only -- but the portal is the only thing that writes that file.

8. cloud_fallback.base_url is not URL-validated

POST /admin/api/cloud-fallback-config
{"cloud_fallback":{"base_url":"notaurl","model":"x", ...}}   -> 200

and base_url: notaurl lands in the overlay. POST /api/providers/{name} rejects ftp://nope with 422 via _validate_provider_url; the same check is absent here, and this block is what the router falls back to when the local classifier dies.

9. Allowlist delete always reports success

DELETE /admin/api/providers/openrouter/allowlist/nope/nope -> 200 {"deleted":"nope/nope"}. Bare DELETE, no rowcount check. Profile delete and model-override delete both 404 correctly on a missing target.

10. Allowlist add cannot update a note

INSERT OR IGNORE. Re-adding sandbox/test-model with note:"dup" returned 200 carrying the original note. There is no way to correct a note through the API or the UI.

Smaller things, all reproduced

• "Apply Feedback" only previews. controls.html:541 sends ?dry_run=true. The button says Apply; there is no way to actually apply from the portal.
• "Seed Energy" spends money with no confirmation and is styled identically (btn-success) to the free Refresh Catalog next to it. ?samples=5 across the catalog is 65+ billed completions.
• "Restart Service" has no confirmation. It is btn-danger, which helps, but it drops in-flight requests on one click -- while deleting a profile requires a two-step confirm.
• Manage on a provider that has no allowlist is a dead end. neuralwatt's panel opens, says "No models on the allowlist" (the truth is "this provider has no allowlist"), and Browse catalog preview fails with a generic "Could not load catalog preview." The real reason -- 409 "allowlist is not required for this provider" -- never reaches the user. The summary line "0 allowed" reads as zero permitted when it means unrestricted.
• The profile editor's model picker offers all 446 rows, including allowlist-pruned and deprecated OpenRouter models, with no filter. The Models page has a "Show deprecated / stale" toggle defaulting to off; this picker has none, so picking one silently yields a zero-admit profile.
• Allowlist catalog search resets after every Add, so adding several models from one search means retyping it each time.
• Allowlist Remove has no confirm step, unlike profile Delete.
• Gaming-mode 409 after saving cloud fallback in the portal tells you to "uncomment the classifier.cloud_fallback example in config/config.yaml" rather than "restart to pick up the block you just saved". cfg binds at import, so the refusal itself is correct; the message points at the wrong remedy.
• Switching classifier mode away from local_encoder leaves the encoder: block in the overlay. Only cloud_primary is cleaned (deletions). Harmless -- validators accept it -- but it is stale state.
• Overlay YAML writes nulls as bare provider: keys (ruamel). Parses fine, reads oddly.

What works

Stated because it is most of the surface, and because the validation in particular is unusually good.

Pages. All seven render and populate from real data. Dashboard Calls/Cost/Energy toggle, all five history ranges, the warnings popover, the quota strip, verdict mix, category breakdown. Models list + "Show deprecated / stale" (45 -> 446). Decisions filters (kind, category, tier, search), pagination, page-size. Proficiency matrix, category column-narrowing, model search, only-measured, sort, and the source control's cell dimming (413/495 cells at outcome_blended) with a legend that distinguishes measured from inherited.

Profiles. Create, duplicate-with-prefill (onlycheaps -> onlycheaps-copy), inline edit, two-step delete, live admission counts that match select_candidates, the zero-admit warning chip, and the restart-required banner. Built-ins correctly read-only.

Providers. Create, delete, allowlist add/remove, live catalog preview (422 models), search filter, "Added" state on entries already allowlisted.

Model overrides. Set and clear, including slash-bearing ids (deepseek/deepseek-v4-flash), with effective_availability / is_overridden reported correctly and deprecation genuinely removing the model from routing.

Controls. All nine runtime knobs flip and read back. All twelve persisted config keys write to the overlay with correct base / overlay provenance.

Error paths -- every one of these returned the right status with a message that explains itself:

probe	result
min_tier: 3, max_tier: 1	422, names both values
latency_tolerance: "turbo"	422
logging.level: "screaming"	422, lists the four valid values
quality_tolerance: -5	422, "must be in [0, 1)"
objective.credit_attenuation.enabled	403, not editable (matches CLAUDE.md)
default_profile: "ghostprofile"	422, lists the five real profiles
confidence_threshold: 80	422, "returns a 0-1 probability, not a percent"
encoder.device: "tpu"	422
cloud_llm with neither primary nor auto	422
gaming mode without cloud_fallback	409 (runtime) and 422 (persisted)
duplicate profile / builtin profile / missing profile	409 / 403 / 404
delete the current default_profile	422, says to repoint it first
history?range=bogus	400
allowlist ops on a non-allowlist provider	409
unknown provider	404

Triggers. Refresh Catalog ran clean (19 NeuralWatt + 29 OpenRouter upserted, 393 pruned by the allowlist) and surfaced the catalog sanity-floor warning CLAUDE.md asks for: "[openrouter] fetched only 29 models (current DB has 425); proceeding but catalog may be truncated". Apply Feedback (dry-run) produced a correct per-model failure/success tally.

Two more, found while fixing #1

11. The model picker offered every deprecated row

The allowed_model_ids control was populated from an unfiltered /admin/api/models: 446 options, of which 393 were OpenRouter rows the provider allowlist had already pruned and the poller had marked deprecated. Picking one produced a profile that admits nothing, with nothing on screen saying why. models.html filtered the same payload behind a "Show deprecated / stale" toggle defaulting off; the picker did not.

12. config/config.local.yaml breaks 9 tests

Not a portal bug, but it bites anyone running the suite on a configured machine. Reproduced both directions on the same tree:

overlay present	result
no	1523 passed
yes (classifier.mode: local_encoder)	9 failed, 1514 passed

The nine are in test_classifier_backoff.py (5), test_gaming_mode.py (3) and test_classifier_modes_dispatch.py (1) -- e.g. assert 'local_encoder' == 'local_llm'. They load the repo's real config/config.yaml and pick up the machine-local overlay beside it, so a deployment that has configured classifier.mode cannot run its own tests clean. This deployment has exactly that overlay, so pytest fails 9 today regardless of any change.

Fixed on fix/admin-profile-model-visibility

Findings #1, #11, and the picker half of the smaller items:

• GET /admin/api/models now defaults to routable rows only (50 here, of which the 30 OpenRouter ones are exactly the 30 allowlist entries). ?include_unroutable=true returns all 446, and models.html -- the availability table, the one page that legitimately wants them -- is the only caller that passes it. Every present and future picker gets the filtered default. Allowlist a model, poll, and it appears.
• The profile detail pane renders admitted_models as chips instead of hiding it in a title= tooltip truncated to 8 of 40.
• The allowed_model_ids control is a two-pane Available / Selected picker. What is chosen is always on screen, filtering narrows Available only, "Add all" takes the whole filtered set, and a stored id that is no longer routable stays in Selected flagged not routable -- which is what closes the silent-drop in #1: opening Edit and saving untouched now preserves it.
• The list summary says 2 models rather than 1 filter.

The rest, in the order they were done

• #2 6b9826c -- every admin body inherits _AdminBody (extra="forbid"). Immediately found a real client/server mismatch: the profiles UI sent name on the update path, which the body model has no field for. It had been silently dropped.
• #7 6b9826c -- POST /api/providers/{name} refuses a base-only definition, as delete already did. Also corrected both profile 403s, which advised "add an overlay profile with the same name to override it" -- something admin_profile_create rejects with a 409.
• #3 / #4 43645d4 -- both off-states exclude, and the override binds /v1/models and pinned requests as well as /route. _admin_deprecated_models renamed _admin_excluded_models, since "deprecated" no longer describes what it returns.
• #8 / #10 281f4fb -- _validate_provider_url applied to cloud_fallback and cloud_primary; allowlist add upserts so a note can be corrected.
• #5 / #6 48df93b -- X-Total-Count on the decisions endpoint, and the page names the window and the history separately. Both nav gaps closed, with a test that derives the page list so a new admin page fails until it is linked.
• #12 80d0e6a -- ROUTER_IGNORE_LOCAL_CONFIG, set in tests/conftest.py. 1613 tests now pass identically with and without the machine's overlay in place.

#9 was not a defect

The sweep flagged the allowlist DELETE returning 200 for an absent entry as a false confirmation, and the fix was written -- then reverted, because it broke test_allowlist_delete_idempotent, a test that pins that exact behaviour with a docstring saying so. That is a decision, not an oversight.

The inconsistency it noticed is still real and is the one open question left here: profile delete and model-override delete both 404 on a missing target while this one does not. Which way that resolves is a call to make, not a bug to fix.

Smaller items still open

The nine UX items listed above are unaddressed except the picker ones: the "Apply Feedback" button still only previews, "Seed Energy" still spends money with no confirmation, Manage on a non-allowlist provider still opens a dead-end panel, and the decisions FLAGS column still has no legend.