ef06e7d0d1
proficiency_score is the only category-dependent term in the composite, so with the table empty the classifier's category output was computed, paid for at ~10s a request, and then discarded. Across 27 decisions (9 categories x 3 tiers) routing produced 2 distinct models under list-price scoring and 3 under measured cost/eco. It now produces 7, with four different models winning tier 1 depending on category. Adds: - proficiency.py / proficiency_store.py -- pure blending plus the single write path, so blended_score and source cannot drift from their inputs. Scores accumulate into a running mean rather than replacing, so re-running the harness tightens estimates instead of discarding history. - leaderboards.yaml / leaderboard.py -- curated per-family priors and their importer, for cold start: a newly listed NeuralWatt family has no self-eval history and would otherwise be indistinguishable from a model measured and found average. Ships EMPTY on purpose; inventing benchmark numbers would put fabricated data into routing, the same failure as the provider's static_fallback carbon constant this project already excludes. `leaderboard.py --check` names every family missing a prior. - evals/tasks.yaml / eval_proficiency.py -- 23 tasks over all 9 categories, scored objectively wherever the category admits it: code executed against checks, exact answers compared, tool calls inspected structurally. Only the four prose categories use a judge, and a judge never grades its own family. - base_model_id on models, so -flex rows inherit their family's scores rather than being re-measured: same weights, different queue. The blending rule needed a fallback the design doc did not specify. Read literally, a model with no leaderboard prior and 9 real samples scores nothing. Self-eval now carries it, labelled self_eval_thin so thin evidence stays distinguishable from evidence that cleared the threshold. Findings: coding does NOT discriminate this catalog -- all 13 rows score 1.00 on all three coding categories even after the tasks were hardened with touching intervals, present-but-falsy defaults, late-binding closures and a binary search that infinite-loops. What discriminates is tool use, arithmetic traps and prose. deepseek-v4-flash scores 1.00 on coding but 0.33 on tool_use_agentic: given a prompt containing both times it needed, it calls two tools instead of subtracting. The router now avoids it there while still choosing it for coding. Three harness defects were found and fixed along the way, each of which scored the rig rather than the model: a token budget shared between a reasoning trace and the answer (empty completions scored 0.00), a single leading space making valid code an IndentationError, and judge malfunctions recorded as model failures. tests/test_task_set.py now validates every task against a reference solution so a broken check cannot masquerade as difficulty -- it caught one on its first run. Tests 134 -> 153. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa
1.9 KiB
1.9 KiB