proficiency_score is the only category-dependent term in the composite, so
with the table empty the classifier's category output was computed, paid for
at ~10s a request, and then discarded. Across 27 decisions (9 categories x 3
tiers) routing produced 2 distinct models under list-price scoring and 3
under measured cost/eco. It now produces 7, with four different models
winning tier 1 depending on category.
Adds:
- proficiency.py / proficiency_store.py -- pure blending plus the single
write path, so blended_score and source cannot drift from their inputs.
Scores accumulate into a running mean rather than replacing, so re-running
the harness tightens estimates instead of discarding history.
- leaderboards.yaml / leaderboard.py -- curated per-family priors and their
importer, for cold start: a newly listed NeuralWatt family has no self-eval
history and would otherwise be indistinguishable from a model measured and
found average. Ships EMPTY on purpose; inventing benchmark numbers would put
fabricated data into routing, the same failure as the provider's
static_fallback carbon constant this project already excludes.
`leaderboard.py --check` names every family missing a prior.
- evals/tasks.yaml / eval_proficiency.py -- 23 tasks over all 9 categories,
scored objectively wherever the category admits it: code executed against
checks, exact answers compared, tool calls inspected structurally. Only the
four prose categories use a judge, and a judge never grades its own family.
- base_model_id on models, so -flex rows inherit their family's scores rather
than being re-measured: same weights, different queue.
The blending rule needed a fallback the design doc did not specify. Read
literally, a model with no leaderboard prior and 9 real samples scores
nothing. Self-eval now carries it, labelled self_eval_thin so thin evidence
stays distinguishable from evidence that cleared the threshold.
Findings: coding does NOT discriminate this catalog -- all 13 rows score 1.00
on all three coding categories even after the tasks were hardened with
touching intervals, present-but-falsy defaults, late-binding closures and a
binary search that infinite-loops. What discriminates is tool use, arithmetic
traps and prose. deepseek-v4-flash scores 1.00 on coding but 0.33 on
tool_use_agentic: given a prompt containing both times it needed, it calls two
tools instead of subtracting. The router now avoids it there while still
choosing it for coding.
Three harness defects were found and fixed along the way, each of which
scored the rig rather than the model: a token budget shared between a
reasoning trace and the answer (empty completions scored 0.00), a single
leading space making valid code an IndentationError, and judge malfunctions
recorded as model failures. tests/test_task_set.py now validates every task
against a reference solution so a broken check cannot masquerade as
difficulty -- it caught one on its first run.
Tests 134 -> 153.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa
ef06e7d0d1