6krrt/plans/catalog-staleness-and-poller-failure-modes.md

# Catalog staleness: the fuse is real, the timer is not what arms it

Status: done -- rejection_warnings and the demand-ceiling detector in metrics.py

**Date:** 2026-09-01
**Status:** **FINAL — ready to implement.** Small; one commit.
**Prompted by:** "is `stale_after_days: 3` too tight?"
**Answer:** no — and it is also not the knob that governs the failure the docs
warn about. Both `CLAUDE.md` and `README.md` described this backwards and have
been corrected in the same pass.

---

## 1. What was actually checked

| | |
|---|---|
| poll cadence | **every 2 hours**, `llm-router-poller.timer`, healthy |
| poller errors, last 7 days | **0** |
| last run | upserted 19/19 models, 22 min before this was written |
| rows currently stale | **1** — `deepseek-ai/DeepSeek-V4-Flash`, genuinely gone from the catalog since 2026-08-24. Correct. |
| tests covering `fetch_neuralwatt` or `mark_stale` | **none** |

Against a 2-hourly poll, `stale_after_days: 3` is **36 consecutive successful
polls** of margin. Nothing has expired that should not have.

Recovery is automatic as well: `upsert` writes
`availability = excluded.availability` (`poller.py:251`), so a single good poll
flips every stale row back to `active`. The blast radius of a false stale-out is
one 2-hour cycle, not manual intervention.

---

## 2. The documented failure mode is not reachable

Both docs said, in substance: *an unpolled catalog marks every row stale and the
router returns zero candidates.*

It cannot. `mark_stale` (`poller.py:283`) runs only **inside** `poller.main()`,
and `main()` returns early on `requests.RequestException` — **before** `upsert`
and **before** `mark_stale`. So:

| event | what actually happens |
|---|---|
| timer stopped / disabled | nothing runs, so nothing is marked. Catalog freezes at last-known-good; every row still reads `active`. |
| provider API down, DNS fails, network out | early return at the fetch. Same: catalog frozen, nothing marked. |

The real behaviour is **silent and fail-open** — the router keeps routing on
prices, context windows and capability flags that may be weeks out of date, and
nothing anywhere says so. That is arguably the better default (a soft failure
beats a hard one), but it is the *opposite* of what the docs told you to watch
for, so attention was pointed at the wrong risk.

---

## 3. The path that can empty the candidate set

`fetch_neuralwatt` (`poller.py:161`):

```python
payload = resp.json()
rows = []
for m in payload.get("data", []):
```

No floor on row count. A **200 response carrying an empty or truncated `data`
array** — a partial provider outage, a schema change, an auth path degrading to
an empty list — clears `raise_for_status()`, returns `[]`, upserts nothing, and
then `mark_stale` runs anyway because `main()` never saw an exception.

Three days of that and every row is stale; `exclude_stale: true` then leaves
zero candidates for everything.

So the fuse exists — but `stale_after_days` only sets its **length**. It neither
arms nor disarms it. **Lengthening it buys a longer fuse on something that
should not be armed at all.**

Note this also breaks `mark_stale`'s own docstring, which claims it flags "rows
that weren't touched by this poll run". After a zero-row fetch that is
vacuously every row, which is not what the sentence means.

---

## 4. The fix

### 4.1 Sanity floor in `fetch_neuralwatt`

Refuse an implausible catalog by **raising**, so `main()` takes its existing
early-return path — which is already the correct behaviour for a failed fetch.

- **Zero rows → raise.** Unconditional. An empty catalog is never a real answer
  from a provider that has 19 models.
- **Below ~50% of the current row count → log a warning and proceed.** A
  legitimate mass deprecation should not be blocked outright, but it should not
  pass unremarked either.

Raise a `RequestException` subclass (or let `main()` catch a new
`CatalogTooSmall` alongside it) so the existing handler covers it and the exit
code stays 1 — systemd then reports the unit as failed, which is the loud signal
this path currently lacks.

The row count to compare against is `SELECT COUNT(*) FROM models WHERE provider
= 'neuralwatt'`, so this needs the connection. Either pass it in, or move the
check into `main()` between `fetch_neuralwatt` and `upsert` — the latter keeps
`fetch_neuralwatt` free of DB access, which matches the module's existing shape.
**Prefer the check in `main()`.**

### 4.2 Then leave `stale_after_days` at 3 — or shorten it

Once the floor exists, `availability = 'stale'` means exactly one thing: *this
row was absent from a catalog we successfully fetched.* That is a direct
observation, not something needing a three-day timer to establish. A genuinely
removed public model currently stays routable for three days after it vanishes;
with the floor in place, 1 day (or two consecutive polls) is defensible and
tighter.

This is a judgement call with no measurement behind it either way, so the
recommendation is: **land the floor, leave the value at 3, revisit only if a
removed model actually causes a dispatch failure.** Changing both at once would
make the next incident ambiguous.

### 4.3 Surface catalog age, since the real failure is silence

`metrics.py` already builds a `warnings` list (`metrics.py:131`) that the admin
dashboard's warnings bell and the TUI both render. Add one:

```
catalog last polled 4.2 days ago — prices, context windows and capability
flags may be out of date. Check: systemctl --user status llm-router-poller.timer
```

Threshold at `max(1 day, 2 x stale_after_days / 3)` or simply 1 day — the poll
is 2-hourly, so anything past a day means the timer is not running. This is the
change that actually addresses the fail-open behaviour in §2: it converts a
silent freeze into something visible without changing the (correct) decision to
keep serving.

### 4.4 Surface context-ceiling collapse — the other path that empties the candidate set

**Added 2026-09-01, after this exact failure took the router down mid-task.**

§3 covers one way to reach zero candidates. Admin availability overrides are
another, and unlike the poller path it fires instantly, silently, and by a
deliberate operator action that gives no indication of what it just cost.

**What happened.** Seven models were deprecated through `/admin` at 03:11-03:12
on 2026-09-01 — the expensive ones, a reasonable cost decision in isolation. The
`models` table still read `active` (the poller refreshes it every 2h), but
`admin_model_overrides` overlays `deprecated` and `_admin_deprecated_models`
(`dispatcher.py:1977`) feeds that into routing's hard filters. Measured effect on
the maximum servable `required_context_tokens`, interactive:

| tier | ceiling before | ceiling during | eligible models |
|---|---|---|---|
| 1 | 782,324 | 782,324 | 6 |
| 2 | 782,324 | 782,324 | 5 |
| 3 | 782,324 | **94,196** | **1** |

Tier 3's ceiling fell **8.3x** while tiers 1 and 2 were untouched. Observed
tier-3 traffic has a max of 268,168 tokens and a p95 of 179,356, so every
tier-3 request above 94,196 returned `422 No model satisfies the hard filters`.
Nothing warned, nothing logged above INFO, and the admin portal reported the
change as a plain success. It surfaced ~19 hours later as an agent failing
mid-task with an opaque "Unprocessable Content", and took a `route_decisions`
query to diagnose.

**CORRECTION, 2026-09-01 — an earlier draft of this section proposed an
"inversion" check. It was wrong, and the error is worth keeping because it is
easy to make again.** That draft said: warn when tier *T*'s ceiling is below
tier *T-1*'s. But `ceiling(T) = max(effective_context_window)` over models with
`tier >= T`, and tier is a capability FLOOR, so as *T* rises the eligible set
**shrinks monotonically**. Therefore `ceiling(1) >= ceiling(2) >= ceiling(3)`
holds for *every possible catalog*. A "higher tier has a lower ceiling" warning
is not a fault detector — it is a theorem, and it would fire on every healthy
catalog while never indicating anything. Verified live: 782,324 / 782,324 /
782,324 today, and 782,324 / 782,324 / 94,196 during the outage — descending in
both cases, exactly as the monotonicity predicts.

The real harm has nothing to do with ordering. It is that **the ceiling at some
tier fell below what that tier is actually asked to serve**, and the only way to
know that is to compare against traffic.

**Add two warnings.** Both compute the ceiling with the same filters routing
uses (tier floor, `access_level`, `availability`, admin overrides, `latency_class`
vs `latency_tolerance`), so they cannot drift from real routing behaviour:

1. **Ceiling below observed demand** — the primary check. Tier *T*'s ceiling is
   below the max (or p95) `required_context_tokens` among `route_decisions` rows
   with `kind = 'chat'` and `task_tier = T` over the last N days.
   Self-calibrating: it measures the router against what it is actually asked
   for rather than an invented constant. Verified against live data — silent on
   all three tiers today, and it fires on the outage state (tier 3 ceiling
   94,196 vs observed max 268,168).
2. **Zero eligible models** for any `(tier, latency_tolerance)`. Cheap, and it
   catches total collapse — but note it would *not* have fired on 2026-09-01,
   because tier 3 still had one eligible model. It is a backstop, not the
   detector.

**Optionally, an escalation-hazard warning.** `escalation.enabled` is true with
`max_tier: 3`, so a request can be bumped a tier at runtime. When
`ceiling(T+1)` is below the p95 demand at tier *T*, escalating a servable
request can make it unservable. That is the real hazard the word "inversion"
was reaching for, stated in a form that is actually checkable.

**Computing the ceiling correctly matters too.** Call `select_candidates` with
`required_context_tokens=0` — i.e. apply every hard filter *except* context —
then take `ceiling = max(effective_context_window)` and
`count = len(candidates)`. Filtering by the max window first (an easy mistake)
yields the right ceiling by luck but leaves `count` holding only the models tied
at the top, which silently breaks the zero-eligible check.

Both belong in the existing `metrics.py` `warnings` list (`metrics.py:131`),
next to §4.3's catalog-age warning — same rationale, same render path (admin
warnings bell, TUI).

**And warn at the moment of causation, which matters more than the dashboard.**
`admin.py`'s `POST /api/models/{model_id}/{provider}/availability`
(`admin.py:502`) is where the damage is done. After the write, recompute the
ceilings and, if the change introduced an inversion or dropped a ceiling below
observed demand, (a) log at WARNING and (b) return the warning in the response
body so the portal can render it inline against the toggle that caused it. The
operator should learn that deprecating `kimi-k2.7-code` just cost tier 3 its
context headroom **while looking at the switch**, not nineteen hours later.

Do **not** block the change — deprecating a model is a legitimate operator
action and there are good reasons to accept a lower ceiling temporarily. Warn
loudly, refuse nothing.

**Recovery, for the record:** re-activating `kimi-k2.7-code` (192,500 eff)
restored the 94k-192k band and `glm-5.3` (782,324 eff) restored everything
above it, giving a continuous ladder of 94,196 → 192,500 → 782,324. The
`kimi-k3` family stays deprecated deliberately: `glm-5.3` covers the identical
782k band at half the prompt price and a third of the completion price.

---

## 5. File-by-file

- `src/poller.py`
  - `main()` — after `fetch_neuralwatt()` and before `upsert()`, compare
    `len(rows)` against the current row count; raise/exit(1) on zero, warn below
    half. Keep the existing `RequestException` handler as the single exit path.
  - `mark_stale` docstring — say that it assumes the caller has already
    established the response was plausible, and point at the check.
- `src/metrics.py` — the catalog-age warning (§4.3) and the two
  context-ceiling warnings (§4.4) in the existing `warnings` list. Factor the
  ceiling computation into one helper — `context_ceilings(conn, cfg) ->
  {(tier, latency_tolerance): (ceiling, eligible_count)}` — reusing
  `routing.select_candidates` rather than re-implementing the filters, so it
  cannot drift from real routing.
- `src/admin.py` — `POST /api/models/{model_id}/{provider}/availability`
  (`:502`) recomputes ceilings after the write; logs at WARNING and returns the
  warning in the response body when the change introduces an inversion, drops a
  ceiling below observed demand, or empties a tier. Never blocks the change.
- `admin/frontend/models.html` — render that inline warning against the toggle
  that caused it. This is the highest-value half of §4.4; a dashboard warning
  the operator has to go looking for is what failed here.
- `tests/test_poller_parsing.py` (or a new `tests/test_poller_freshness.py`) —
  there is currently **no** coverage of either function. Pin:
  - a zero-row payload exits non-zero and marks **nothing** stale;
  - a payload at 40% of current rows proceeds but warns;
  - a full payload marks a row absent from it stale, and only that row;
  - a previously-stale row returns to `active` after one good poll
    (the auto-recovery in §1, currently untested);
  - `main()` on `RequestException` marks nothing.
- `CLAUDE.md` "Run as a service" and `README.md` "As a systemd service" —
  **already corrected in this pass**; no further edit needed.

### Second entry point to keep in mind

`admin.py:734` `POST /api/refresh-catalog` shells out to `python -m poller`, so
it shares `main()` and inherits the floor for free. Worth a manual click after
the change to confirm the admin trigger still reports success correctly when the
poller now exits 1 on a degraded fetch.

---

## 6. Summary

| | |
|---|---|
| Is `stale_after_days: 3` too tight? | **No.** 36 successful polls of margin at the current 2-hourly cadence, and a false stale-out self-heals in one cycle. |
| Does the plan in `proficiency-exposure-bias-and-exploration.md` cover it? | **No** — different subsystem entirely (that one touches `proficiency` and candidate ranking). |
| What is actually wrong | `fetch_neuralwatt` has no floor on row count, so a 200-with-no-data arms a catalog-wide stale-out on a 3-day fuse. And an unpolled catalog fails **silently open**, which is the opposite of what the docs claimed. |
| Second path to zero candidates (§4.4) | Admin availability overrides. Deprecating 7 models on 2026-09-01 collapsed tier 3's context ceiling from 782,324 to **94,196** — an 8.3x drop, below the 268,168 max that tier actually receives — with no warning anywhere. Took the router down mid-task and surfaced ~19h later as an opaque agent failure. |
| Fix | A row-count floor in `main()`; a catalog-age warning and two context-ceiling warnings in `/metrics`; an inline warning at the admin toggle that causes a ceiling collapse; and the first tests this module has ever had. |

**The through-line across all of §3, §4.3 and §4.4:** every failure here is the
router quietly losing the ability to serve requests while continuing to report
itself healthy. None of them are wrong decisions — a frozen catalog, a
deprecated expensive model, and a tight budget are all defensible. What is wrong
is that each one is invisible until an agent fails with an error that names none
of them. The fix in every case is the same shape: compute the thing that is
actually true, and say so where the operator is already looking.