alee/6krrt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(pinch): instrument savings and count tool overhead in the budget #18

Merged
alee merged 4 commits from feat/pinch-instrumentation-and-token-accounting into main 2026-09-02 03:16:53 +00:00
Conversation 0 Commits 4 Files Changed 18 +600 -49
alee commented 2026-09-02 03:14:47 +00:00
Owner
Copy Link

Pinch is the router's only lever on the ~92.6% of request cost that is prompt tokens, and until now there was no way to ask whether it earned its place. This makes it observable and fixes an accounting bug that was hiding most of its work.

Instrumentation

  • route_decisions gains pinch_original_tokens / pinch_final_tokens, in config/schema.sql and the code-side ensure_route_decisions migration, so live databases predating the columns pick them up on next start.
  • metrics.pinch_summary aggregates 30-day savings: share pruned, total and median tokens saved, estimated dollars saved.
  • Wired into /metrics, /admin/api/snapshot, the TUI model, and an admin dashboard card. Returns None when pinch is disabled, matching local_energy_summary's contract; all three consumers guard for it.
  • The pinch log moves from debug to info, and only when pruning actually occurred.

The dollar figure is priced at the same blended prompt rate routing.estimated_cost uses (src/routing.py:363-374), reading cfg.objective.assumed_cache_rate. An earlier draft of the spec said to price it at the cached rate, which is wrong twice over — cost_per_1m_prompt_cached is already discounted, so multiplying by cache_rate applies the discount twice, and it drops the (1 - cache_rate) fraction billed at full price. That would have put a savings number on the dashboard computed on a different basis than the cost model the router ranks on.

Accounting fix

prune_context and _relevance_order_for both take extra_fixed_tokens, and the dispatcher passes the tool-definition overhead. opencode sends ~32k prompt tokens of tool definitions on a trivial request, and pinch was excluding all of it from the budget comparison.

Threading it through _relevance_order_for as well is load-bearing: without it, relevance ordering silently declines to run on exactly the requests that newly need pruning — no error, no log, just uniform pruning instead of least-relevant-first.

Review finding fixed mid-flight

Persistence covers the routed success, routed rejection, and passthrough paths. On passthrough the persist originally ran 43 lines before the prune, so kind='passthrough' rows recorded NULL even when pruning occurred. The prune is now hoisted above the persist.

The persist deliberately stays above _check_pinned_capabilities, which raises 422 — so a rejected pin still writes the decision row that explains it. Anyone tidying that order will silently delete rejection records.

Guarded by test_passthrough_records_pinch_columns_when_pruned, which asserts both columns non-NULL and original > final.

Verification

  • 860 tests pass — 26.8s serially, 7.9s under pytest-xdist -n auto. Both modes, so xdist is not masking ordering bugs.
  • router.db byte-identical before and after the suite (same mtime and size) — no test writes to the live database.
  • config/config.yaml untouched; budget_tokens unchanged.
  • Four independent reviewers returned APPROVE.

Note on scope

Includes 4cb7a92, a correction to the plan spec in plans/, which is an ancestor of this branch but not yet on origin/main.

Also worth knowing before the next merge: feat/proficiency-exposure-bias-and-exploration appends columns to the tail of the same route_decisions block in both config/schema.sql and ensure_route_decisions. Whichever of the two merges second gets a small conflict at the flex_forced line.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U

Pinch is the router's only lever on the ~92.6% of request cost that is prompt tokens, and until now there was no way to ask whether it earned its place. This makes it observable and fixes an accounting bug that was hiding most of its work. ## Instrumentation - `route_decisions` gains `pinch_original_tokens` / `pinch_final_tokens`, in `config/schema.sql` and the code-side `ensure_route_decisions` migration, so live databases predating the columns pick them up on next start. - `metrics.pinch_summary` aggregates 30-day savings: share pruned, total and median tokens saved, estimated dollars saved. - Wired into `/metrics`, `/admin/api/snapshot`, the TUI model, and an admin dashboard card. Returns `None` when pinch is disabled, matching `local_energy_summary`'s contract; all three consumers guard for it. - The pinch log moves from `debug` to `info`, and only when pruning actually occurred. **The dollar figure is priced at the same blended prompt rate `routing.estimated_cost` uses** (`src/routing.py:363-374`), reading `cfg.objective.assumed_cache_rate`. An earlier draft of the spec said to price it at the cached rate, which is wrong twice over — `cost_per_1m_prompt_cached` is already discounted, so multiplying by `cache_rate` applies the discount twice, and it drops the `(1 - cache_rate)` fraction billed at full price. That would have put a savings number on the dashboard computed on a different basis than the cost model the router ranks on. ## Accounting fix `prune_context` and `_relevance_order_for` both take `extra_fixed_tokens`, and the dispatcher passes the tool-definition overhead. opencode sends ~32k prompt tokens of tool definitions on a trivial request, and pinch was excluding all of it from the budget comparison. Threading it through `_relevance_order_for` as well is load-bearing: without it, relevance ordering silently declines to run on exactly the requests that newly need pruning — no error, no log, just uniform pruning instead of least-relevant-first. ## Review finding fixed mid-flight Persistence covers the routed success, routed rejection, and passthrough paths. On passthrough the persist originally ran 43 lines *before* the prune, so `kind='passthrough'` rows recorded NULL even when pruning occurred. The prune is now hoisted above the persist. The persist deliberately stays *above* `_check_pinned_capabilities`, which raises 422 — so a rejected pin still writes the decision row that explains it. Anyone tidying that order will silently delete rejection records. Guarded by `test_passthrough_records_pinch_columns_when_pruned`, which asserts both columns non-NULL and `original > final`. ## Verification - **860 tests pass** — 26.8s serially, 7.9s under `pytest-xdist -n auto`. Both modes, so xdist is not masking ordering bugs. - `router.db` byte-identical before and after the suite (same mtime and size) — no test writes to the live database. - `config/config.yaml` untouched; `budget_tokens` unchanged. - Four independent reviewers returned APPROVE. ## Note on scope Includes `4cb7a92`, a correction to the plan spec in `plans/`, which is an ancestor of this branch but not yet on `origin/main`. Also worth knowing before the next merge: `feat/proficiency-exposure-bias-and-exploration` appends columns to the tail of the same `route_decisions` block in both `config/schema.sql` and `ensure_route_decisions`. Whichever of the two merges second gets a small conflict at the `flex_forced` line. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U
alee added 4 commits 2026-09-02 03:14:47 +00:00
docs(plans): correct pinch dollars-saved formula to the blended prompt rate 4cb7a92a8a 
The spec said to price saved tokens at the cached prompt rate. That is wrong
twice: cost_per_1m_prompt_cached is already the discounted per-1M price, so
multiplying it by cache_rate applies the discount a second time, and it drops
the (1 - cache_rate) fraction billed at full price entirely.

Match routing.estimated_cost (src/routing.py:363-374), including its COALESCE
fallback for a NULL cached price. Otherwise the admin dashboard would report a
savings figure computed on a different basis than the cost model the router
actually ranks on.

Caught by Metis during plan gap analysis.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U
feat(pinch): instrument pinch savings and count tool overhead in the budget 7938a777dc 
Pinch is the router's only lever on the ~92.6% of cost that is prompt
tokens, and until now there was no way to ask whether it earned its place.

Instrumentation:
- route_decisions gains pinch_original_tokens / pinch_final_tokens, in
  config/schema.sql and the code-side ensure_route_decisions migration, so
  live databases predating the columns pick them up on next start.
- metrics.pinch_summary aggregates 30-day savings: share pruned, total and
  median tokens saved, and estimated dollars saved. The dollar figure is
  priced at the SAME blended prompt rate routing.estimated_cost uses
  (src/routing.py:363-374), including the fallback when the cached price is
  NULL, and reads cfg.objective.assumed_cache_rate so the dashboard and the
  router's own cost model cannot disagree.
- Wired into /metrics, /admin/api/snapshot, the TUI model, and an admin
  dashboard card. Returns None when pinch is disabled, matching
  local_energy_summary's contract; all consumers guard for it.
- The pinch log moves from debug to info, and only when pruning occurred.

Accounting fix:
- prune_context and _relevance_order_for both take extra_fixed_tokens, and
  the dispatcher passes the tool-definition overhead. opencode sends ~32k
  prompt tokens of tool definitions on a trivial request, and pinch was
  excluding all of it from the budget comparison. Threading it through
  _relevance_order_for as well matters: without it, relevance ordering would
  silently decline to run on exactly the requests that newly need pruning.

Persistence covers the routed success, routed rejection, and passthrough
paths. On passthrough the prune is hoisted above the persist so pinch stats
are recorded rather than NULL -- the persist deliberately stays above
_check_pinned_capabilities, which raises 422, so a rejected pin still writes
the decision row that explains it.

860 tests pass, serially and under pytest-xdist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U
test(eval_scoring): bound infinite-loop test timeout to 1s 4bab724c84 
score_code's CODE_TIMEOUT_SECONDS is 15, and this test waits the whole
thing out to prove an infinite loop is bounded rather than hung -- 15.08s,
about 39% of the entire suite's runtime, to assert a timeout fires.

Monkeypatching the module attribute to 1 gives the identical
(0.0, 'timeout') result in ~1s. src/eval_proficiency.py is untouched; the
production timeout stays 15s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U
chore(tests): add pytest-xdist for parallel execution d7f051bb2e 
860 tests: ~26.8s serially, ~7.9s under -n auto. Pinned like every other
dependency -- recreating the venv with unpinned ranges has silently jumped
major versions in this repo before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U
alee merged commit 9d50b60975 into main 2026-09-02 03:16:53 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
alee
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: alee/6krrt#18
Delete Branch "feat/pinch-instrumentation-and-token-accounting"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?