ef06e7d0d1
proficiency_score is the only category-dependent term in the composite, so with the table empty the classifier's category output was computed, paid for at ~10s a request, and then discarded. Across 27 decisions (9 categories x 3 tiers) routing produced 2 distinct models under list-price scoring and 3 under measured cost/eco. It now produces 7, with four different models winning tier 1 depending on category. Adds: - proficiency.py / proficiency_store.py -- pure blending plus the single write path, so blended_score and source cannot drift from their inputs. Scores accumulate into a running mean rather than replacing, so re-running the harness tightens estimates instead of discarding history. - leaderboards.yaml / leaderboard.py -- curated per-family priors and their importer, for cold start: a newly listed NeuralWatt family has no self-eval history and would otherwise be indistinguishable from a model measured and found average. Ships EMPTY on purpose; inventing benchmark numbers would put fabricated data into routing, the same failure as the provider's static_fallback carbon constant this project already excludes. `leaderboard.py --check` names every family missing a prior. - evals/tasks.yaml / eval_proficiency.py -- 23 tasks over all 9 categories, scored objectively wherever the category admits it: code executed against checks, exact answers compared, tool calls inspected structurally. Only the four prose categories use a judge, and a judge never grades its own family. - base_model_id on models, so -flex rows inherit their family's scores rather than being re-measured: same weights, different queue. The blending rule needed a fallback the design doc did not specify. Read literally, a model with no leaderboard prior and 9 real samples scores nothing. Self-eval now carries it, labelled self_eval_thin so thin evidence stays distinguishable from evidence that cleared the threshold. Findings: coding does NOT discriminate this catalog -- all 13 rows score 1.00 on all three coding categories even after the tasks were hardened with touching intervals, present-but-falsy defaults, late-binding closures and a binary search that infinite-loops. What discriminates is tool use, arithmetic traps and prose. deepseek-v4-flash scores 1.00 on coding but 0.33 on tool_use_agentic: given a prompt containing both times it needed, it calls two tools instead of subtracting. The router now avoids it there while still choosing it for coding. Three harness defects were found and fixed along the way, each of which scored the rig rather than the model: a token budget shared between a reasoning trace and the answer (empty completions scored 0.00), a single leading space making valid code an IndentationError, and judge malfunctions recorded as model failures. tests/test_task_set.py now validates every task against a reference solution so a broken check cannot masquerade as difficulty -- it caught one on its first run. Tests 134 -> 153. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa
42 lines
1.9 KiB
YAML
42 lines
1.9 KiB
YAML
# Curated leaderboard priors, keyed by model FAMILY (models.base_model_id).
|
|
#
|
|
# Why this file is hand-maintained rather than scraped: there is no unified
|
|
# leaderboard API, model naming differs everywhere (this catalog's `kimi-k3`
|
|
# against a leaderboard's "Kimi K3 (0731)"), and a scraper that silently
|
|
# starts returning nothing looks identical to a model that scores nothing.
|
|
#
|
|
# What it is FOR: cold start. NeuralWatt adds models, and a newly listed one
|
|
# has zero self-eval history — without a prior it sits at the neutral 0.5 and
|
|
# is indistinguishable from a model that was measured and found average. A
|
|
# prior gives it a defensible starting position until the eval harness has
|
|
# run enough samples to take over (proficiency.self_eval_min_samples).
|
|
#
|
|
# SHIPPED EMPTY ON PURPOSE. Inventing plausible-looking numbers here would put
|
|
# fabricated data directly into routing decisions — the same failure as the
|
|
# provider's static_fallback carbon constant, which this project already
|
|
# excludes for exactly that reason. Fill in only figures you actually looked
|
|
# up, and say where they came from in `source`.
|
|
#
|
|
# Serving variants inherit from their family automatically: an entry for
|
|
# `kimi-k3` covers kimi-k3-fast and kimi-k3-flex. Do not list variants.
|
|
#
|
|
# `python leaderboard.py` imports this and names any active family missing an
|
|
# entry. Categories must come from proficiency.categories in config.yaml, and
|
|
# scores are 0-1. Partial entries are fine — list only the categories a
|
|
# benchmark actually measures, and leave the rest to self-eval.
|
|
|
|
families: {}
|
|
# Shape, for when you have real numbers to add:
|
|
#
|
|
# kimi-k3:
|
|
# source: "aider polyglot 2026-08-14; livebench 2026-08-01"
|
|
# scores:
|
|
# coding_general: 0.82
|
|
# coding_refactor: 0.79
|
|
# reasoning_math: 0.88
|
|
#
|
|
# gemma-4-31b:
|
|
# source: "lmarena 2026-08-10"
|
|
# scores:
|
|
# general_chat: 0.61
|