6krrt/plans/token-waste-waves.md

Token waste: a five-wave plan

Status: in progress -- Wave 1 shipped, deployed and EVALUATED (PR #88; gate run 2026-09-14 on 1,288 decisions at full coverage); Wave 2 built and gated-passed (PR #89, ships off, awaiting the enable decision); Wave 3 DEMOTED to correctness -- its savings case was refuted by the same gate.

Measured read-only against the live /home/alee/Sources/6krrt/router.db on 2026-09-13 at repo state 26398d7. Companion diagnosis: plans/ten-thousand-foot-review.md.

Intended executor: opencode (Atlas), directly. Every item lands as its own commit. Waves are ordered by dependency, not by size.

---

The one-paragraph thesis

The provider's cache is worth ~92% of every prompt on this workload (measured 2026-09-14 at full coverage: 0.919 on same-model turns, n=1,227). The router's only real action, choosing a model, is also the act that throws that cache away — a switch drops the rate to 0.348 (n=55) and doubles the billed cost per prompt token — and it takes that action without knowing the cache exists. Everything below follows from that.

Narrowed 2026-09-14. This paragraph previously read "token waste here is almost entirely cache destruction, not volume", which was too broad. Cache destruction by model switching is real and is the leak. Cache destruction by editing the payload is not: these providers tolerate a mid-prompt change without invalidating the remainder, so the router's pruning does not cost cache (see Wave 3, demoted on that evidence). The distinction matters because it moves one whole wave out of the savings column.

Read this before executing: what the data supports, and what it does not

Two claims were wrong during analysis and are corrected here so nothing gets built on them. This repo has a history of scoring the rig instead of the model; the same discipline applies to this plan.

CONFIRMED — model switching destroys the cache. Re-measured 2026-09-14 on the post-Wave-1 window at ~100% cached-token coverage, which supersedes the thin 12-22%-coverage figures this section first carried:

turn	n	cache rate	µ$ / prompt token
same model as previous	1,227	0.919	0.0369
switched model	55	0.348	0.0734
first in session	5	0.000	0.0403

The gap widened on the honest sample (0.571 against the thin sample's 0.474) while the cost ratio settled to 1.99x from 2.46x. Switch turns are 4.3% of traffic. Two independent measurements — cache rate and billed cost per prompt token — agree in direction and magnitude. This is the real leak, and it is the only one that survived re-measurement.

REFUTED 2026-09-14 — payload rewriting does NOT cost cache. The mechanism is real: a direct test of prune_context showed the relevance path diverging at message index 19 of 82 with a literally identical relevance order, where the uniform path diverges nowhere. A growing target_save walks a relevance-ordered list, so the newly-compressed candidate lands at an arbitrary position.

But the inference drawn from it was wrong. At full coverage, turns with half the prompt sitting after the divergence point still cache at 0.910 — which a strict prefix cache cannot do. These providers tolerate a mid-prompt edit without invalidating the remainder. Wave 3 is demoted on that evidence; see it for the full tables.

This entry has now been wrong in both directions, which is the useful part. It first read NOT CONFIRMED on a 0.924 aggregate — one number averaged across two cohorts at 12-22% coverage, too coarse to separate them. It was then flipped to CONFIRMED on a cohort split from a single hour (n=46, 0.705 vs 0.940). The full window (n=410 rewritten) put the gap at 0.013 and the mechanism test removed it entirely.

Three readings, two reversals, one underlying question. What distinguished the final answer was not more data but a mechanism test — asking whether cache rate falls as the share of the prompt after the divergence rises, rather than comparing two buckets and trusting the difference. A cohort split can only tell you two groups differ; it cannot tell you why, and it will happily report noise as signal at small n.

The lesson generalises past this entry: an aggregate that mixes two populations will report the majority's number and hide the minority's, and "the data does not show it biting" is a claim about resolution as much as about reality.

---

Standing acceptance gate — applies to every wave

In addition to each wave's own criteria below:

Every config knob a wave introduces ships with an admin control — runtime, persisted, or both as its mechanism warrants — or a recorded decision saying why it must not.

The escape clause is load-bearing, not hedging. objective.credit_attenuation.enabled is deliberately absent from the admin allowlist and from provider edits, because turning it on must be a config edit plus a restart (CLAUDE.md, "Routing notes"). So the rule is not "everything must have a control" — it is that the ABSENCE of one is a decision someone made, rather than an oversight nobody noticed.

Wave 2 is why this is written down: objective.incumbent_cache_pricing and objective.incumbent_challenger_cache_rate shipped with no control and no decision, and a dial whose whole point is tuning from neutral to full penalty without reverting code is worth little if turning it means hand-editing a tracked file and restarting the service.

tests/test_admin_knob_coverage.py enforces this for the sections the portal already reaches, and fails naming the knob.

---

Wave 1 — See the leak

No behavior change. Every later wave's acceptance criteria read what this produces, so nothing else should start first.

Commit 1bd8c6e (2026-09-10) already landed streamed cost capture and cached_prompt_tokens parsing at dispatcher.py:2056, which is why data begins on 09-11. The instrument exists; it is under-covered.

1.1 OpenRouter usage opt-in

Coverage since the instrument landed:

provider	obs	cost	cached tokens	duration
neuralwatt	805	804 (99.9%)	103 (12.8%)	804 (99.9%)
openrouter	734	165 (22.5%)	164 (22.3%)	0 (0%)

OpenRouter's cost coverage (22.5%) and cached coverage (22.3%) match almost exactly. That is one gap, not two: the usage block is simply absent on ~78% of requests. dispatcher.py:4254 sets stream_options: {"include_usage": True}, which is the OpenAI spelling; OpenRouter requires its own usage: {"include": true} in the request body to return accounting. Add it per-provider — the reports_cost_in_usage flag from 1bd8c6e is the right place to hang it.

This single change is the highest-leverage item in the plan: it takes OpenRouter from ~22% to ~100% on cost, cached tokens and latency at once, on the provider carrying ~73% of decisions.

Verify: coverage for all three fields above 95% on OpenRouter rows written after the change.

1.2 Decide what an absent cached_tokens means

NeuralWatt reports cost on 99.9% of rows but prompt_tokens_details on only 12.8%. Determine whether the field is omitted on a full cache miss or is model-dependent. If omitted on a miss, record an explicit 0 rather than NULL, because every cache-rate reading in this plan is otherwise conditioned on a hit having occurred. Nine rows currently store 0, so zeros do reach the DB sometimes — that needs explaining before the 0.924 figure can be trusted.

Verify: a test pinning the parse for a usage block with cached_tokens: 0, one with the key absent, and one with prompt_tokens_details absent entirely, asserting the three map to distinct stored values.

1.3 Prefix-stability probe

The decisive instrument for Wave 3, and about twenty lines. After prune_context returns, hash the pruned payload cumulatively by message and store the hash of the longest stable prefix (or simply a per-turn list of message hashes) on route_decisions. Within a session, compare consecutive turns to get the real first-divergence index and the tokens after it.

That converts Wave 3 from an argument into a measurement, and it will either justify Wave 3 or retire it.

Verify: on a replayed synthetic session, the probe reports divergence at the index the direct test predicts.

1.4 Cache rate in /metrics

A cache-rate series per (provider, model) over a trailing window, plus a warning when a session's rate falls below a configurable floor. Follow the existing detector conventions in metrics.py — novelty-or-rate, not bare presence, per the reactive rejection detector.

This is also the premise-expiry check for assumed_cache_rate: 0.917: the constant was measured 2026-08-23 and nothing has re-measured it since.

---

Wave 2 — Stop the confirmed leak

The router has no incumbent. rank_candidates does not know what ran last turn, so there is no hysteresis and no switching cost. Three sub-items, and the first is most of the work.

2.1 Thread the incumbent into ranking

The session's last selected_model is already on disk in route_decisions and usually in memory. Pass it into rank_candidates as the incumbent.

Keep the existing shape: quality band first, cost as tiebreak. The incumbent changes only the cost key.

2.2 Price the cache loss

In the tiebreak, price the incumbent at the measured cache rate and every challenger at a cold rate. Concretely, estimated_cost already takes cache_rate; pass cfg.objective.assumed_cache_rate for the incumbent and 0.0 for challengers. A challenger then has to beat the incumbent by more than the cache it is about to discard, which on a 100k prompt is most of the prompt.

This is the first time the router's own decision becomes an input to its own cost model, and it is why it belongs before the deeper reframe in Wave 5.

Two guards so this cannot become stickiness-at-any-cost:

• the quality band is computed before the cost key, exactly as today, so a genuine quality gap still wins outright and the incumbent gets no quality advantage;
• a hard-filter failure (context ceiling, capability gate, circuit breaker, profile allowlist) still removes the incumbent unconditionally.

2.3 Exploration becomes session-scoped

Epsilon-greedy prices a suboptimal pull as the arm's cost difference. Here exploring means switching, so the real cost is that difference plus a cold prompt — and exploration.py cannot see it. Move the coin flip to session start rather than per turn: one exploratory session costs one cold prompt instead of one per turn, and it produces a cleaner outcome signal because the whole session is attributable to the explored model.

exploration.py takes an injected RNG and holds no mutable state, so this is a call-site change, not a rewrite.

Wave 2 acceptance, read off Wave 1's instruments:

• switch rate per session falls;
• the same-model / switched cache-rate gap (0.952 vs 0.478) narrows, or the remaining switches are deliberate — driven by a quality band or a hard filter, not by a cost re-rank;
• median billed µ$ per prompt token moves toward the same-model figure.

---

Wave 3 — Make prefix stability guaranteed rather than incidental

DEMOTED 2026-09-14, back to correctness rather than savings. This section was promoted on 2026-09-13 on the probe's first hour of data, which showed rewritten turns at 0.705 cache rate against 0.940 for turns that merely grew. That promotion carried an explicit caveat — "n=46 from roughly one hour, do not treat 0.705 as settled, re-measure on a fuller sample." The re-measurement is in, and the caveat is the part that held.

What the full window actually measured

1,288 decisions over ~5 hours, at ~100% cached-token coverage, joined on cached_tokens_source = 'reported':

prefix state	pruning	n	avg tokens after divergence	cache rate
grew only	pruned	854	702	0.904
rewritten	pruned	387	32,901	0.891
grew only	under budget	14	6,322	0.765
rewritten	under budget	23	9,075	0.548

Among pruned turns the gap is 0.013, not the 0.235 the first hour showed.

The mechanism test, which is what settles it

If a mid-prompt edit invalidated everything after it, cache rate would fall as the share of the prompt sitting after the divergence point rises. It does not:

tokens after divergence	n	share of prompt	cache rate
<5k	24	7%	0.813
5-20k	89	28%	0.888
20-50k	227	50%	0.910
>50k	66	50%	0.835

No monotonic relationship. Turns with half the prompt after the divergence point still cache at 0.910 — arithmetically impossible under a strict prefix cache, which would cap that bucket near 0.50.

Conclusion: these providers do not use a strict prefix cache. A mid-prompt edit does not invalidate the remainder. That assumption was load-bearing under this wave's savings case and under the reading of the prune_context simulation, and it is wrong.

The simulation itself was never wrong about what it measured — the relevance path really does rewrite the payload at an arbitrary position, and the uniform path really does not. What was wrong was the inference that a rewritten payload costs cache. It measurably does not.

Gate 3 is untouched by this. Switching models is not a mid-prompt edit; it is a different cache namespace entirely, and it still measures 0.919 against 0.348. Nothing here weakens Wave 2.

What this changes for the sub-items

• 3.1 (monotone region) is now optional and probably not worth it. It buys no measurable cache. It would buy determinism in a working path, at the cost of editing that path. Do not do it for savings; there are none.
• 3.2 (tripwire test) as originally specced asserts prefix stability, which the relevance path violates by design. Without 3.1 it would ship red. Either it follows 3.1 or it is dropped — it is not independently landable.
• 3.3 (config comment reconciliation) is still worth doing, and is now a clean decision. With cache out of the picture the only difference between the paths is token volume against context quality: the direct test measured the relevance path shipping 109,606 tokens against uniform's 94,716 on the same input, because it stops as soon as the deficit is covered. So relevance costs ~15k more tokens per turn and buys keeping the most relevant content verbatim. That is a real tradeoff, just not a cache one — decide it on its own terms.
• 3.4 (non-pruning rewrite source) drops to low priority. The under-budget rewritten cohort (n=23, 0.548) is small and its low rate is better explained by session-opening turns than by a distinct leak.

An opportunity this created

If a mid-prompt edit costs no cache, pruning is cheaper than assumed and could be more aggressive without a cache penalty. Pruned turns currently run 132k -> 89k (a 33% cut); pinch.budget_tokens could go lower for real prompt- side savings. The binding constraint is answer quality, not cache. That is a new, evidence-created item and it belongs in Wave 4 or 5, not here.

A second rewrite source exists, outside pruning

Cross-referencing the same rows against whether pruning actually fired:

prefix state	pruning fired	under budget
grew only	193	6
rewritten	39	7

Pruning explains 39 of 46 rewrites. Seven turns rewrote the prefix with pruning never engaged at all — under budget, so prune_context returned the messages untouched. On those turns the payload is the client's own message list, which means something upstream of the router edited its own history (opencode compaction, a changed tool-definition array, or a mutated system prompt are the candidates).

Two consequences, and both matter for how this wave is judged:

• 3.1 cannot fix all of it. Fixing the relevance path addresses at most 39 of 46. Crediting Wave 3 with the whole gap would overstate it.
• The residual is worth identifying before it is assumed benign. Among under-budget turns the rewrite rate is 7 of 13 — proportionally higher than the pruned cohort's 39 of 232, though on a sample far too small to lean on. If the client rewrites its own history routinely, that is a larger cache leak than pruning and the router cannot fix it by changing prune_context.

3.1 Constrain relevance compression to a monotone region

Keep the relevance ranking and change what it decides. Instead of compressing a scattered relevance-ordered subset, let relevance pick where the compressed boundary sits on first crossing, then only ever extend that boundary forward. Once a message is compressed it is never un-compressed.

That preserves the feature's intent — relevance still decides what is worth keeping verbatim — while restoring the property the uniform path has for free: appends cannot change a byte before the boundary.

3.2 A prefix-stability tripwire test

This repo already writes exactly this kind of test (test_tui_schema_drift.py, test_tui_warnings.py). Assert that across a simulated multi-turn session, prune_context's output for turn N+1 is byte-identical to turn N's up to the appended messages. Fail naming the divergent message index.

The uniform path passes this today; the relevance path does not. That asymmetry is the whole point of the test.

3.3 Reconcile the config with reality

pinch.relevance.enabled: true, while the comment directly above it still reads "Off by default… ship it, watch route_decisions / pinch stats on real traffic, then decide the default." It was switched on and the comment never followed. Same shape as session_cache below it. Decide the default on Wave 1 data and rewrite both comments to say what is actually true.

The Wave 1 data now exists, so this is decidable rather than deferred. Note the decision is not automatically "turn it off": the earlier direct test showed the relevance path also ships more tokens than uniform (109,606 against 94,716 on the same input, because it stops as soon as the deficit is covered rather than compressing every candidate). If 3.1 makes it prefix-stable, relevance becomes strictly better than uniform. If 3.1 proves harder than expected, disabling it is the cheap fallback that wins on both axes today.

3.4 Identify the non-pruning rewrite source

Scoped by the measurement above, not speculative. Seven of 46 rewrites occurred with pruning disengaged, so something upstream of prune_context is editing conversation history between turns.

This is an investigation, not a fix: determine whether the client is compacting its own context, whether the tool-definition array changes between turns, or whether the system prompt is mutated. The probe already stores what is needed to find the turns; the question is what differs across them.

Sequence it after 3.1 so the pruning-caused rewrites are removed from the population first, leaving a clean residual to study. Attempting it now means diagnosing two overlapping causes at once.

Acceptance for Wave 3

The probe that promoted this wave is also its acceptance instrument, which is the point of having built it first:

1. The rewritten share among pruned turns falls toward zero (39 of 232 today).
2. The cache-rate gap between the two cohorts closes — 0.705 against 0.940 today. Re-measure both on a fuller sample before and after, per the re-measure discipline in Wave 2's acceptance; do not hardcode today's figures as the target.
3. The tripwire test in 3.2 stays green on both compression paths.
4. Any residual rewriting is attributed to a named cause by 3.4, not left as unexplained variance.

---

Wave 4 — Retries, and the half nobody guards

4.1 Denominate the iteration budget in prompt re-bills

iteration.py counts attempts. A retry on a 100k-token conversation is a 100k prompt re-bill to redo a ~400-token answer, and malformed escalates to the next-ranked candidate, which makes it a cold one. Tier 3 allows two.

Budget in re-bills instead: gate a retry on prompt size, and prefer same-model-with-more-tokens (which keeps the cache) over escalation (which does not) wherever the failure permits it. The existing failure taxonomy already supports this — truncated retries the same model by design; it is malformed that escalates.

4.2 Accept that structural verification is inert here

98.96% of structural verdicts are unverifiable (28,958 of 29,263), because agent turns end in tool calls and has_tool_calls short-circuits both checkers. That is correct behavior and the documented fix to a real false-failure incident. The conclusion not yet drawn is that the free checker now checks nothing on the only workload this router serves: 305 substantive verdicts out of 29,263.

Meanwhile a completion token costs ~201x a prompt token, so the expensive half of the ledger has no guard and the cheap half carries all the machinery.

Two actions, both small:

• Give feedback.py a timer. It is the only loop without one — deploy/ ships timers for the poller, seed sweep, backup and offsite sync, and a baseline-report timer is installed. proficiency was written in one batch at 2026-09-10T00:44:55 and the 95 unapplied outcomes are exactly the models carrying today's traffic. Client outcomes are the only guard on wasted completions, and they are applied by hand.
• Stop reporting the structural checker as a safety net in the docs, and either narrow it to the paths where it still fires or retire it.

---

Wave 5 — The reframe, and the expiry checks

Decisions, not tasks. Make them before adding machinery to the subsystems they touch.

5.1 The session is the routing unit

Waves 2 and 3 patch a per-request frame. The unit is wrong: the workload is a session of hundreds to thousands of turns sharing a monotonically growing prefix, and estimated_cost is a function of prompt_tokens, so the ranking's key input changes every turn even when nothing else does. CLAUDE.md documents the consequence as a feature — the winner at 50k differs from the winner at 120k — which inside a session is a cache dump.

If the session is the unit and a turn a delta, then: routing decides once and re-decides only on a threshold that includes the cache loss; classification becomes "has the task changed?" rather than a full taxonomy inference (96.6% of turns already reuse a cached label); exploration is naturally session-scoped; and routing.min_tool_proficiency becomes expressible, because "can this model be trusted with tools" is a session-level property, which is how the docs already describe it.

5.2 Recalibrate estimated_cost against billed rows

Depends on 1.1. Per-request estimate against bill spans 0.63x to 13.55x, a 21x spread in the error, which reorders candidates. 31,119 NeuralWatt rows carry a real billed figure joinable by request_id. A per-model correction factor over a trailing window, refreshed by the poller, fixes the ordering without a sweep.

The cause is already in CLAUDE.md two sections apart and never reconciled: attribution ratio spans 750x between models and is "most of the real cost difference in the catalog", while estimated_cost prices from catalog token prices, which carry no attribution term.

5.3 Premise-expiry checks

Four settings carry an explicit revisit condition in their own comment, all four conditions are met, and none re-fired: quality_tolerance: 0.1 ("narrow it as samples accumulate" — now 46 outcome-backed rows per category averaging 51.8 samples); cost-as-tiebreak ("all real traffic to date totals $0.07" — now $118.79); assumed_cache_rate: 0.917 (measured 2026-08-23, never re-measured); pinch.relevance ("off by default… then decide").

Every such comment should have its condition as a check that fails loudly — a test, a /metrics warning, a line in the poller. 1.4 is the first instance; generalize it.

---

Order of execution

wave	items	gate to proceed
1	1.1, 1.2, 1.3, 1.4	coverage >95% on OpenRouter cost/cached/duration
2	2.1, 2.2, 2.3	switch cache-rate gap narrows, or switches are deliberate
3	demoted — 3.3 only (a token-volume vs context-quality decision); 3.1/3.2 optional, 3.4 low	no savings gate; payload rewriting was measured not to cost cache
4	4.1, 4.2	proficiency refreshing on a timer, unapplied backlog ~0
5	decisions	—

1.1 first, alone, and measure for a day before anything else lands. It is small, it is reversible, and until it is in place every acceptance criterion below it is reading a 22%-covered sample.