42 lines
1.9 KiB
YAML
42 lines
1.9 KiB
YAML
# Curated leaderboard priors, keyed by model FAMILY (models.base_model_id).
|
|
#
|
|
# Why this file is hand-maintained rather than scraped: there is no unified
|
|
# leaderboard API, model naming differs everywhere (this catalog's `kimi-k3`
|
|
# against a leaderboard's "Kimi K3 (0731)"), and a scraper that silently
|
|
# starts returning nothing looks identical to a model that scores nothing.
|
|
#
|
|
# What it is FOR: cold start. NeuralWatt adds models, and a newly listed one
|
|
# has zero self-eval history — without a prior it sits at the neutral 0.5 and
|
|
# is indistinguishable from a model that was measured and found average. A
|
|
# prior gives it a defensible starting position until the eval harness has
|
|
# run enough samples to take over (proficiency.self_eval_min_samples).
|
|
#
|
|
# SHIPPED EMPTY ON PURPOSE. Inventing plausible-looking numbers here would put
|
|
# fabricated data directly into routing decisions — the same failure as the
|
|
# provider's static_fallback carbon constant, which this project already
|
|
# excludes for exactly that reason. Fill in only figures you actually looked
|
|
# up, and say where they came from in `source`.
|
|
#
|
|
# Serving variants inherit from their family automatically: an entry for
|
|
# `kimi-k3` covers kimi-k3-fast and kimi-k3-flex. Do not list variants.
|
|
#
|
|
# `python leaderboard.py` imports this and names any active family missing an
|
|
# entry. Categories must come from proficiency.categories in config.yaml, and
|
|
# scores are 0-1. Partial entries are fine — list only the categories a
|
|
# benchmark actually measures, and leave the rest to self-eval.
|
|
|
|
families: {}
|
|
# Shape, for when you have real numbers to add:
|
|
#
|
|
# kimi-k3:
|
|
# source: "aider polyglot 2026-08-14; livebench 2026-08-01"
|
|
# scores:
|
|
# coding_general: 0.82
|
|
# coding_refactor: 0.79
|
|
# reasoning_math: 0.88
|
|
#
|
|
# gemma-4-31b:
|
|
# source: "lmarena 2026-08-10"
|
|
# scores:
|
|
# general_chat: 0.61
|