Local LLM Model Router: NeuralWatt routing service + context-aware framing and pruning #6

alee · 2026-08-24T02:24:21Z

alee commented

2026-08-24 02:24:21 +00:00

Implements the Local LLM Model Router (see design/local-llm-model-router.md,
README.md) as a FastAPI dispatcher against NeuralWatt Cloud, plus
context-aware classification framing and relevance-based context pruning.

Highlights

Routing: local Ollama classifier picks category/tier/context, hard
filters + quality-first ranking dispatch to the cheapest/best-fit open-weight
model on NeuralWatt, weighted by cost, per-category proficiency, and a
per-request energy ceiling. Cost is priced per request from catalog token
prices scaled to the request's shape.
OpenAI-compatible surface (/v1): auto / auto:batch virtual models
and any real model id, streaming proxy with SSE energy/cost telemetry
scraping, capability gates (vision / JSON mode / tools), local vision
fallback.
Context-aware framing (classifier.context_framing): _previous_context
feeds the classifier the prior assistant turn so short follow-ups inherit its
complexity; framing instruction travels only with content that justifies it.
Context pruning (pinch.*, off by default): trims old tool results once
past budget_tokens; user/assistant/system always kept verbatim, non-text
blocks preserved, honest tokens_saved accounting, image payloads stubbed so
trimming actually shrinks what ships upstream.
Observability: route_decisions persistence, GET /metrics, TUI
dashboard, router_cli probe.
Verification & learning: structural + async local-LLM checks,
POST /outcome, feedback.py folding real outcomes into proficiency.

Reviewed

Both rounds of fixes (the 11 confirmed review findings plus the
image-accounting regression found on second pass) are included and verified:
545 offline tests pass, lsp_diagnostics clean.

Implements the Local LLM Model Router (see `design/local-llm-model-router.md`, `README.md`) as a FastAPI dispatcher against **NeuralWatt Cloud**, plus context-aware classification framing and relevance-based context pruning. ## Highlights - **Routing**: local Ollama classifier picks category/tier/context, hard filters + quality-first ranking dispatch to the cheapest/best-fit open-weight model on NeuralWatt, weighted by cost, per-category proficiency, and a per-request energy ceiling. Cost is priced per request from catalog token prices scaled to the request's shape. - **OpenAI-compatible surface** (`/v1`): `auto` / `auto:batch` virtual models and any real model id, streaming proxy with SSE energy/cost telemetry scraping, capability gates (vision / JSON mode / tools), local vision fallback. - **Context-aware framing** (`classifier.context_framing`): `_previous_context` feeds the classifier the prior assistant turn so short follow-ups inherit its complexity; framing instruction travels only with content that justifies it. - **Context pruning** (`pinch.*`, off by default): trims old tool results once past `budget_tokens`; user/assistant/system always kept verbatim, non-text blocks preserved, honest `tokens_saved` accounting, image payloads stubbed so trimming actually shrinks what ships upstream. - **Observability**: `route_decisions` persistence, `GET /metrics`, TUI dashboard, router_cli probe. - **Verification & learning**: structural + async local-LLM checks, `POST /outcome`, `feedback.py` folding real outcomes into proficiency. ## Reviewed Both rounds of fixes (the 11 confirmed review findings plus the image-accounting regression found on second pass) are included and verified: 545 offline tests pass, `lsp_diagnostics` clean.

alee added 3 commits 2026-08-24 02:24:21 +00:00

feat(classifier): context-aware framing and relevance-based context pruning fbcc63630a

Port two ideas from the MIT-licensed alexrudloff/llmrouter:

1. Context-aware classification. chat_completions now feeds the classifier
   the message before the last user turn, framed as llmrouter's
   "Context: <prior>\n---\nMessage: <current>" (classifier.context_framing,
   default on). A short follow-up ("Yes", "Try now?") inherits the prior
   turn's complexity instead of being classified in isolation as trivial.
   _classifier_user_content is a pure, testable framing helper; the system
   prompt gains the inherit-from-context rule.

2. Relevance-based context pruning (context_prune.py). An optional,
   pre-dispatch stage that trims old tool results from the provider-bound
   conversation once it exceeds pinch.budget_tokens (pinch.enabled defaults
   off). User/assistant/system messages are always kept verbatim; only old
   tool results are summarized (head+tail with an elision marker) or dropped
   to a placeholder, with role pairing preserved so the result still parses
   as a conversation. Applies to both the streaming and non-streaming
   upstream bodies; the classifier input is exempt (it already clamps).

Tests: +23 (pruning invariants, framing, previous-context extraction, pinch
config validation). 529 pass.

docs: review + plan for context-pruning-and-framing feature 099958b392

fix: resolve context-pruning-and-framing review findings 5f7716e121

Address the 11 confirmed bugs and 4 cleanups from
context-pruning-and-framing-review.md, plus the image-accounting
regression it found on a second pass.

classifier framing:
- _previous_context walks back to the nearest assistant turn only,
  skipping system/user/tool so raw tool output and the system prompt
  never contaminate the framing signal; reuses context_prune.extract_text
  instead of a duplicated block parser.
- classify() drops its redundant if/not-context branch.
- framing instruction moved out of the static system prompt and into
  _classifier_user_content so it travels only with content that justifies
  it (respects context_framing opt-out).

context pruning:
- extract_text counts image_url bytes so image-bearing tool results are
  sized (and pruned) correctly.
- -- new, follow-up review -- image_url blocks now count toward the size
  trigger but not toward the replacement text, and are stubbed to a short
  marker when the message is trimmed, so an image-heavy tool result
  genuinely shrinks rather than staying full-size while tokens_saved lied
  and base64 leaked into the text block.
- tokens_saved is now the honest whole-list before/after reduction
  (max(orig - final, 0)), never a per-message estimate that could drift.
- max_summarize_chars validated >= 3000 at config load; trim branch guards
  against negative math and message growth.
- recency guard protects the newest tool result even with no user turn.
- non-text blocks survive trimming (type preserved) rather than being
  flattened away.
- dropped/stat terminology corrected to summarized; _turn_of() removed;
  module-level pinch defaults removed in favor of PinchConfig.

Dispatch and config:
- prune_context runs once before the measured-context routing decision
  (reused at dispatch, no double-prune, no in-place mutation); no-op when
  pinch is disabled.
- dead context=prev_context arg removed from the measured reroute.
- TaskRequest.context dual use documented.

15 new regression tests pin the fixes; 545 tests pass.

alee merged commit b9cdd2b494 into main

2026-08-24 02:30:47 +00:00

alee referenced this issue from a commit

2026-08-24 02:30:49 +00:00

Merge pull request 'Local LLM Model Router: NeuralWatt routing service + context-aware framing and pruning' (#6) from neuralwatt-router-service into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alee/6krrt#6