6krrt/plans/classifier-fallback-cascade.md

Classifier failure should degrade gradually, not to a fixed guess

Status: done -- the four-step cascade in classify()

Status: FINAL — decision-complete. Written 2026-09-04.

The gap

classify() (src/dispatcher.py:~555-590) has exactly one endpoint (_classifier_client(), cfg.classifier.base_url) and one failure behaviour: return cfg.classifier.fallback_category / fallback_tier — currently general_chat / tier 2 — flagged source="fallback".

That is a fixed guess, and general_chat is the worst possible one: proficiency_score is the ONLY category-dependent term in ranking, so a request mislabelled general_chat loses the single signal that makes routing category-aware. Every request in the outage window routes as if it were small talk.

The session cache cannot help, because of where it sits. session_cache.get() is called at dispatcher.py:3077, before classifying. classify() resolves its own failure internally and never sees the session key. So a session with 200 prior turns, every one classified coding_refactor, still degrades to general_chat the moment the local model is unavailable.

Why this matters now, and why it hasn't yet

source='fallback' appears zero times in 16,744 recorded decisions (cached 11,089, classifier 5,631, override 24). The path is real but cold — which is precisely why it should be hardened rather than trusted: untested code that only runs during an outage is the worst kind.

And it is about to get warmer. The operator stops Ollama to play games on the same GPU. That is a deliberate, recurring, whole-session outage of the local classifier — exactly the condition this path exists for.

The fix: a cascade, cheapest first

Replace the single static fallback with an ordered cascade. Each step is tried only if the previous failed, and each records a distinct classification_source so the fallback mix is visible in route_decisions rather than collapsed into one bucket.

1. Local classifier — unchanged. source='classifier'.
2. Stale session reuse — source='session_stale'. On classifier failure, reuse this session's most recent classification ignoring the staleness bound. A stale but real classification of the same conversation beats a fixed guess, and this is free: no network, no tokens, works when everything external is down.
3. Durable session history — source='session_history'. If the in-memory cache has nothing (it is module-level and resets on restart, so a router restart during an Ollama outage empties it), read the most recent task_category / task_tier for this session_key from route_decisions. One indexed query on a table that is already written on every decision.
4. Cloud classifier — source='classifier_cloud'. Optional, off unless configured. See below.
5. Static fallback — unchanged, source='fallback'. Last resort only.

Steps 2 and 3 are the valuable ones: free, offline, and they cover the actual scenario (a running session whose local model went away mid-conversation). Step 4 covers a new session started during an outage, which steps 2-3 cannot.

The cloud classifier step

cfg.classifier already accepts any OpenAI-compatible endpoint, so this is a second endpoint plus failover, not new transport.

It is worth having, and it is cheap. Measured previously and recorded in CLAUDE.md: deepseek-v4-flash classifying the same prompts returned 1.02s mean, 5/5 categories correct, 0 hard failures, at $0.093 per 1,000 calls — 0.19% of the plan allowance. Against a local qwen3.5 that averaged 11.58s with one hard failure.

Requirements:

• New optional block (e.g. classifier.cloud_fallback) with its own base_url, model, api_key_env, timeout_seconds. Default disabled — it spends the user's credits, so it must be opt-in.
• Do not reuse the primary classifier block's fields. The whole point is a different endpoint.
• max_retries=0, same as the primary. The primary's timeout already becomes a 3x wall-clock bound with SDK retries on; do not repeat that mistake.
• Its timeout must be short. This runs only after the local attempt already failed, so the request has already spent that budget. A slow cloud fallback turns one bad request into two.
• Never fall back to cloud when the account is out of credits. Plan 5 added _account_level_refusal; a cloud classifier call during a credit outage wastes latency to reach a foregone 4xx. Skip step 4 if a recent account-level refusal is known.

Observability

The distinct classification_source values are the point. Today every degradation is one bucket, so an operator cannot tell "Ollama is down but sessions are being reused correctly" from "everything failed, we are guessing".

• metrics.scoring_coverage warnings: warn when the non-classifier, non-cached share of recent decisions exceeds a threshold. That is the signal that the classifier is down, and nothing currently reports it.
• Surface the source mix on the admin dashboard. decisions.html already displays source; the values just need to be recognised and styled.

Non-goals

• Do not change fallback_category / fallback_tier semantics or defaults. They remain the last resort.
• Do not make the cloud classifier the primary. Local-first is the project's premise, and the local model currently wins on latency (1.07s vs 1.02s is parity, and local costs no credits).
• Do not classify with a dispatch model as a side effect. Step 4 is an explicit configured endpoint, not an opportunistic reuse of a routed call.
• Do not extend the session-cache staleness bound for the normal path. Step 2 ignores staleness ONLY on the failure path; normal operation keeps its existing freshness guarantee.

Success criteria

• A test with the classifier raising and a populated session cache yields the session's prior category with source='session_stale', not general_chat.
• A test with the classifier raising and an EMPTY in-memory cache but existing route_decisions rows for that session_key yields source='session_history'.
• A test with the classifier raising, no session data, and cloud fallback disabled yields source='fallback' — the current behaviour, preserved.
• A test with cloud fallback configured and the local classifier raising yields source='classifier_cloud'.
• Cloud fallback is skipped when a recent account-level refusal is known.
• A warning fires when the degraded share of recent decisions crosses the threshold.
• Full suite green with local_energy.enabled both true and false.
• The user's config/config.local.yaml is byte-identical after the run (local_energy.enabled: true, tariff_usd_per_kwh: 0.159). Corrected 2026-09-05: this previously named config/config.yaml. PR #26 moved deployment values into the gitignored overlay, so config/config.yaml is clean and tracked, and editing it is an ordinary commit. The file that cannot be recovered is the overlay — it is not in git history at all.