6krrt/plans/local-decision-classifier.md

# `classifier.mode: local_decision` — a Jev-style classifier on the local GPU

Status: planned -- plan reviewed 2026-09-28; gate G (tuning + measurement) decides whether the wiring proceeds
Date: 2026-09-27
Prototype: `plans/local-decision-classifier-prototype.py` (a benchmark you can run, NOT the
implementation)
Held-out prompts: `plans/local-decision-classifier-heldout.yaml`
Background: TypeSafe's hosted "Jev" decision model and the OpenJev/SemIf open
copies. We are copying the **technique**, not using either product.

## Goal

Add a fourth `classifier.mode`, `local_decision`. It asks a small local LLM
(`qwen3.5:4b` on Ollama) one multiple-choice question and reads the answer
from the **first output token's logprobs**, normalised over the option letters
that were offered. There is no text generation, JSON or reasoning trace, so the
runaway-reasoning failure that retired `local_llm` can't happen, and it is
calibrated far better than `local_encoder`.

## Evidence (2026-09-27, RTX 6000 24 GB, measured with the prototype)

46 eval tasks and 30 held-out tasks, each at 3 noise levels (`_wrap_agent_noise`):

| | `evals/tasks.yaml` | held-out | p50 / p95 ms |
|---|---|---|---|
| `local_decision` (`qwen3.5:4b`, GPU) | 71% | 100% | 340-385 / 510-610 |
| `local_encoder` + trained head (live, CPU) | 100%\* | 47% | 80-115 / 100-360 |
| `local_encoder` zero-shot (CPU) | 39% | 60% | 80-115 / 100-350 |

\* The head was fitted on a synthetic corpus derived from `evals/tasks.yaml`, so
this is leakage. On held-out prompts the head is **worse** than zero-shot.

- **Calibration:** `local_decision` averages 0.96 confidence when right and 0.71 when wrong. The encoder
  zero-shot sits at 0.2-0.3 on both.
- **Eval-set misses:** 18 were `coding_general`→`reasoning_math` (algorithm tasks), 18 were
  `file_summarization`→`debugging`/`reasoning_math`/`docs_writing`. Both look like
  option-description problems (component 2).
- **VRAM:** `qwen3.5:4b` holds about 7 GB at `num_ctx` 8192 next to `qwen2.5-coder-router:14b`.
  The card sat at 23.4/24 GB. It's tight, so measure again when planning.
- **Weak held-out set:** a single author wrote all 30 prompts and they are short and clean. Treat
  100% as a ceiling, not a forecast.

## Mechanism (proven by the prototype)

Ollama 0.22 native `POST /api/chat`:
`think: false`, `logprobs: true`, `top_logprobs: 20`,
`options: {num_predict: 1, temperature: 0, num_ctx: N}`. The system prompt says
"answer with the letter only". The options are `A.`–`J.` followed by a natural-language description.
Parse `logprobs[0].top_logprobs`:
- strip each token and remove a trailing `.`
- sum the probability mass for each option letter (`"B"` and `" B"` both count)
- confidence = winner's mass ÷ total option mass
- coverage = total option mass

Tokens like `<think>` show up in the top 20 and must be ignored.

## Components

### 1. `src/local_decision.py` (stdlib + `requests`, no new dependency)

- **`classify_choice(text, options, *, base_url, model, num_ctx, timeout)`** returns
  `(label, confidence, coverage)`. It is generic over the option list, so tier
  reuses it.
- **Coverage floor:** below a configurable `coverage_min` (the model didn't answer
  with a letter), raise. Don't guess.
- **Pure parse function:** split the logprob parsing into its own function so tests
  can drive it with recorded responses and no Ollama.
- **Noise:** decide whether to run `local_encoder._isolate_task_text` first. The prototype
  sent the raw noisy text and noise cost about 2 points; measure both.

### 2. Its own option descriptions

Give it a separate `_DECISION_DESCRIPTIONS` map. **Do not edit**
`local_encoder._CATEGORY_DESCRIPTIONS`, because that would shift the encoder and
invalidate its head. Tune it against the two confusions above, and re-measure after
every change on **both** sets so nobody overfits `evals/tasks.yaml` the way the head did.

### 3. Wiring (follow how `local_encoder` was added)

- **`src/config.py`:** add `"local_decision"` to the `mode` Literal (line ~1187). Add a
  `classifier.decision` block (`base_url`, `model`, `num_ctx`, `timeout_s`,
  `confidence_min`, `coverage_min`, `tier_enabled`) and a validator that requires
  it when the mode is selected. Metering (line ~1712): this mode is an HTTP call to
  Ollama, so it meters like `local_llm` via its own `base_url`, not like the in-process encoder.
- **`src/dispatcher.py`:**
  - add `_classify_via_local_decision` and a branch in `_classify_via_configured_mode`
  - add a startup readiness check in `_ensure_classifier_mode_ready` (model is pulled and
    logprobs come back)
  - on success record `source="classifier"`
  - treat below-threshold confidence as a failure that walks the **unchanged** cascade
- **Gating:** it uses the GPU, so it obeys `_local_classifier_skip_reason()`
  (`local_compute.enabled` plus the circuit breaker), the same as `local_llm`.
- **Admin:** every new scalar knob gets a control on the Classifier card (North Star #1).
  `tests/test_admin_knob_coverage.py` enforces this.

### 4. Tier (behind `tier_enabled`, default false)

- **How:** a second `classify_choice` call with three options (cheap/simple, general,
  frontier/high-stakes), fired **in parallel** with the category call so it adds no
  wall-clock time.
- **Why it defaults off:** no gold tier labels exist yet. Until they do, tier keeps
  falling back to `fallback_tier`, exactly as `local_encoder` does now.
- **Output:** a gold tier set of about 30 labelled prompts, used to decide the default.

### 5. Evaluation harness

- **Backend flag:** add a `--backend encoder|decision` flag to `src/eval_classifier.py`
  (latency is reported per call).
- **Held-out set:** promote the held-out prompts to `evals/` and add a `--tasks` default for them.
- **Head flag:** add a flag that runs the encoder with the head disabled, so the leakage
  stays visible.
- **Report:** accuracy for both sets, calibration (confidence correct vs wrong), and p50/p95.

### 6. Docs

- **CLAUDE.md:** the "Which implementation is PRIMARY" section, which also still says
  the live encoder is `device: cuda` while the overlay says `cpu`. Fix that.
- **Also:** `docs/local-models.md` and `docs/admin-portal.md`.

## Acceptance

- **Held-out accuracy:** at least 90% on the held-out set.
- **Eval-set accuracy:** at least 85% on `evals/tasks.yaml` after component 2. **Stop and
  report** if that needs more than description tuning.
- **Latency:** p95 at or below 800 ms on the RTX 6000 with the 14b router model resident.
  The router model must not be evicted: check `/api/ps` before and after a 100-call run.
- **Failure paths:** Ollama down, model missing, coverage below the floor and low
  confidence each walk the cascade, covered by tests with a fake Ollama.
- **Tests:** the full suite is green, including `test_admin_knob_coverage.py`.

## Out of scope

- **Replacing gaming mode** with an automatic Ollama reachability check. It's a separate
  lift, and when it lands this mode inherits it for free.
- **What to do about the trained encoder head.** It's a separate decision, but the
  evidence above is a strong argument for turning it off.
- **Other routes:** hosted Jev (cloud-only, bans benchmarking, sends raw task text off-box),
  OpenJev 27B (no FP8 on Turing, CC BY-NC, evicts the router model), and fine-tuning.
- **Defaults:** making `local_decision` the `config.yaml` default. It's an overlay opt-in
  until it has run on live traffic.

## Guardrails

- **Worktree:** do the work in its own worktree (North Star #3). There are 15 open ones,
  and `feat/local-encoder-accuracy-rebuild` touches classifier code. Commit by explicit path.
- **Live router:** exercise the admin portal on the 8081 sandbox, never on production 8080,
  and don't edit `config.local.yaml` (see the `6krrt-ops` skill).
- **Live database:** open `router.db` read-only.

## Open questions for planning

1. Isolate noise before calling or not (component 1)? Decide from a measurement.
2. Should tier's second call be a separate request, or both questions in one prompt
   read from two token positions? Separate is simpler; measure the latency before
   choosing the clever option.
3. What `num_ctx` gives the best VRAM trade-off against the 8000-char
   `max_input_chars` clamp?