6krrt/code_plans/context-dual-use-and-classify-once-per-session.md

Spec: two design decisions deferred out of the context-pruning/framing fix pass

Origin. The fix pass for context-pruning-and-framing-review.md (reviewed in context-pruning-and-framing-fixes-review.md) explicitly declined two items as design decisions rather than confirmed bugs: the TaskRequest.context dual-use ambiguity (review finding #6, partially mitigated but not resolved), and classifying once per session instead of once per message (an existing item on the project's own "what's NOT built yet" list). Both are forward-looking — no bug is being reported here, no code changes accompany this document.

---

1. TaskRequest.context carries two unrelated meanings

The problem

context was originally, and is still documented as, "assembled context (docs/code) to send with the task" — a /route//dispatch caller pastes reference material alongside a task description. chat_completions now also uses the same field to carry the prior conversational turn (_previous_context), so a short follow-up like "Yes" inherits that turn's complexity instead of being classified as trivial in isolation.

The current fix (moving the framing instruction from the static system prompt into _classifier_user_content's output, appended only when context is non-empty) narrowed the blast radius — the instruction no longer reaches every classify() call, only ones that actually supply context — but it didn't resolve which of the two meanings a given context value has. A /route caller pasting 800 lines of Django code under task="Refactor this" gets the same "a short follow-up continues the prior turn, classify by the CONTEXT's complexity" instruction that exists for the conversational case.

Why this might not need fixing

The instruction's general principle — "short task + large/complex context implies a non-trivial task" — is arguably a reasonable heuristic for the docs-paste case too, even though it was written for conversational follow-ups. It has never been measured against real /route//dispatch traffic with context set. This project's own stated epistemics apply directly here: POST /outcome is "the only ground truth," and several sections of the README describe correcting an assumption only after measuring it, not before. Speculating about which framing is "more correct" without a measurement repeats the mistake the project has already named and moved past.

Options, if it turns out to matter

Option	Sketch	Trade-off
A. Decouple the mechanisms	Keep TaskRequest.context as the public docs/code field, untouched. Give chat_completions's conversational-continuation signal its own internal path — e.g. classify() gains a private prior_turn: Optional[str] parameter distinct from context, so the instruction is only ever built for genuine conversational continuations	Cleanest semantically; requires touching classify()'s signature and both call sites; /route//dispatch behavior is provably unaffected
B. Tag the field	Add a context_kind: Literal["reference", "prior_turn"] = "reference" field to TaskRequest; only chat_completions's internal calls set "prior_turn"; the instruction is only appended for that kind	Smaller diff than A; adds a field to the public request model that external callers never need to know about
C. Leave as-is, measure	No code change. Watch route_decisions (already logs task_category/task_tier/source per request) for /route//dispatch calls that supply context and see whether their tier/category looks skewed relative to before this change	Zero engineering cost; consistent with the project's own "measure before correcting" pattern; only viable if /route//dispatch-with-context traffic is common enough to be observable

Recommendation: C first. This codebase already has the instrumentation (route_decisions, feedback.py folding in outcomes) to tell whether this is a real problem instead of a theoretical one, and the existing pattern in this project — cost-vs-eco, tier-from-price, the whole classifier-model swap — is "measure, then fix what the measurement shows," not "fix what looks fishy." If /route//dispatch-with-context traffic turns out to be rare or nonexistent, this is not worth A's or B's added surface at all.

---

2. Classify once per session, not once per message

The problem, restated from the README

~10s of local overhead on every message is a real tax for an interactive agent... Still unaddressed: classify once per session rather than per message, cache by prompt hash, or skip classification for short prompts.

With the local mistral-nemo classifier this is now ~1.7s/call (§"The classifier is the latency floor"); with a cloud classifier (measured against deepseek-v4-flash) it's ~1.0s. Either way, every single turn in a long agent session pays this again, even though the session's category (coding_general, debugging, etc.) rarely changes turn to turn — what changes is mostly the token count, which chat_completions already measures directly via estimate_prompt_tokens and doesn't need the classifier for.

The mechanism already half-exists

route()'s override branch (dispatcher.py:673) already skips classify() entirely whenever task_category, task_tier, and required_context_tokens are all supplied — this is exactly what the measured-context reroute (fixed in review finding #4) uses today, just within a single request. Session-level caching is the same mechanism applied across requests: classify once, store (task_category, task_tier) keyed by session, and on every later turn in that session call route() with the cached category/tier plus a freshly measured required_context_tokens (which is cheap — pure token counting, no model call) — landing on the override branch and skipping the classifier round-trip entirely.

Design questions to settle before implementing

1. Session identity. session_fingerprint/session_directory (dispatcher.py) already derive a session identity from message content for observation purposes. Whether that's the right key for a classification cache (vs. e.g. a client-supplied session id, if opencode's protocol carries one) needs checking — a cache keyed on the wrong signal either misses constantly (no benefit) or collides across genuinely different sessions (wrong category persists into unrelated work).
2. Invalidation. A session's task can genuinely change category mid-way (debugging turns into a docs-writing turn turns into refactoring). Pure "classify once, cache forever" risks staleness. Candidate triggers to re-classify: a large jump in required_context_tokens between turns (a proxy for "something new started"), a fixed number of turns (e.g. re-classify every 20), or a TTL. This needs the same "measure before deciding" treatment as everything else in this project — a cheap thing to instrument via route_decisions.source (add a "cached" value alongside "classifier"/"override"/"fallback") and watch category drift over real sessions before picking a policy.
3. Interaction with escalation and retries. apply_escalation currently runs on every fresh classification. A cached category/tier bypasses it entirely on cache-hit turns — need to decide whether escalation state should also be cached per-session or re-evaluated each turn (it's cheap, pure Python, so probably always re-evaluate rather than cache).
4. Storage. In-memory dict keyed by session identity is the obvious starting point (matches the process lifetime of the dispatcher; a restart just means the next turn in every active session re-classifies once, which is a safe failure mode) — no new persistence layer needed unless multi-process deployment becomes a requirement.

Recommendation

Worth building — the latency case is strong and the mechanism is a small extension of code that already exists (the override branch) rather than a new one. But settle invalidation policy (§2) with a measurement pass first, the same way context_framing's default-on-and-measure and the classifier model swap were each decided by running both and comparing, not by argument. A reasonable first cut: cache with no re-classification, ship behind a config flag (default off, matching every other new-and-unproven knob in this project — pinch.enabled, min_tool_proficiency), watch route_decisions on real sessions for category drift, then decide whether any invalidation trigger is actually needed or whether "classify once, never again" is good enough in practice.