69ac8f7745
deepseek-v4-flash was losing every routing decision to glm-5.2-fast on real
agent traffic, and it turned out to be excluded twice over by the same
substitution made in two different places. It has a 1M advertised window,
scores 1.00 on all three coding categories, and lists at $0.14/$0.28 per 1M
against glm's $1.45/$4.50.
The cost axis was measuring the wrong workload. `cost` came from the median
billed USD over seed_energy.py's reference sweep, which sends a 400-token
prompt with a 400-token completion. Real traffic through this router is a
150,000-token prompt with a ~400-token completion and 84% cache hits, and the
ranking does not survive the change of shape:
reference sweep (400/400) glm-5.2-fast 3.2x cheaper
realistic (70k prompt) deepseek-v4-flash 5.0x cheaper
Measured live, three samples each, not inferred. The attribution ratio is what
moves: glm sits at 0.006 on a toy prompt and 0.50 on a 70k one, because it
batches beautifully at small sizes and badly at real ones, while deepseek
barely shifts (0.21 -> 0.25). A fixed-shape benchmark cannot rank models for a
workload of another shape, and no amount of re-sweeping fixes that -- it
measures one wrong thing more precisely.
routing.estimated_cost now prices each request from catalog token prices
scaled to that request's actual shape, via objective.assumed_cache_rate (0.84,
measured from real traffic) and objective.assumed_completion_tokens. List
price is not what gets billed -- NeuralWatt charges per kWh -- but billing is
capped at 3x list, so it tracks the real ordering and bounds it, and on the
one case checked live it agrees with the measurement in direction and
magnitude (7.8x predicted vs 5.0x measured). It is also free, needs no sweep,
and refreshes whenever the poller runs. A row the catalog has no price for
keeps whatever measured cost it arrived with; a missing list price is not
free.
Three signals said deepseek -- catalog token price, NeuralWatt's own published
per-request energy, and a live 70k measurement. Only the 400-token benchmark
disagreed, and it was the one being scored on.
Tiering made the identical mistake independently. Tier is a capability FLOOR:
routing.py drops any row with tier < required_tier. Resolving tier on
completion price alone put deepseek in tier 1 for no reason but being cheap,
which excluded it OUTRIGHT from every tier-2 request -- so the cost fix alone
would have changed nothing. tiering.resolve_tier now gates tier 1 on
tier1_context_max (512000) as well as cost: tier 1 means small AND cheap, not
merely cheap.
The gate reads the ADVERTISED context_window, whose catalog values are the
clean market classes (131056 / 199984 / 262128 / 1048560), rather than
effective_context_window, which varies within a class. 512000 sits in the
empty band between the 256K and 1M classes with a 2x margin either side, so it
is not fitted to any one model. It only ever demotes -- a huge window never
promotes an expensive model into tier 1 -- and a missing window does not block
tier 1, since absent evidence should not decide anything.
Distribution 4/6/9 -> 1/9/9; only the three deepseek rows moved. No
model_tiers override was added, deliberately: the point is that the heuristic
now gets this right, and pinning it in config would mask whether it does.
deepseek now wins coding_general at every context size (16.5x cheaper than
kimi-k3 at 200k) and is still correctly absent from tool_use_agentic, where
its measured 0.33 drops it out of the quality band. That is the eval data
earning it the slot rather than a thumb on the scale. Verified live against
the running service.
Tests 249 -> 256.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ
5.8 KiB
5.8 KiB