7965394b70
This project's comments carry the reasoning, which makes a stale one worse
than none: it is a confident account of a decision that was reversed. CLAUDE.md
already records the reversals correctly, so these were the copies that
disagreed with it, and a reader has no way to tell which one is current.
schema.sql said "That product is what scoring ranks on" about
avg_power_watts * duration_seconds. Scoring reads the ATTRIBUTED figures, and
ranking on the pre-attribution product was tried and rejected -- it discards a
750x between-model signal to suppress a 1.8x within-model one. seed_energy.py's
docstring carried the same claim, and also still described the sweep as feeding
cost; it feeds eco and the energy ceiling now.
load_candidates' docstring described `cost` as the sweep's median.
rank_candidates overwrites it with routing.estimated_cost, priced from catalog
prices scaled to the request -- the measured value survives only as a fallback
for a row the catalog has no price for. The /health warning about missing sweep
data said the same thing and is corrected the same way.
routing.py and scoring.py both still stated the retired weighted composite as
the scoring model. scoring.py additionally now says which of its functions the
router actually calls: eco_score and composite_score are arithmetic with tests
and no callers.
config.yaml's "no flex discount knob" note justified itself with cost scoring
that no longer works that way; the real reason is that latency_tolerance is a
hard filter, and flex and standard carry the same catalog price anyway.
Test counts in README.md and CLAUDE.md were 264/14 files; 320 across 18 now.
CLAUDE.md's "until several sweeps have accumulated" section gets the correction
it needs most: none have. The sweep has been dying on its first billed call
since 6e729ad, so that accumulation starts from the next run, not from months
of history.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ
82 lines
3.3 KiB
Python
82 lines
3.3 KiB
Python
"""Pure scoring functions for the local LLM model router.
|
|
|
|
These functions are deliberately free of I/O: no DB reads, no file reads,
|
|
no config import at runtime. Thresholds and weights are passed as
|
|
arguments so the module stays testable and reusable.
|
|
|
|
What is still live here: ``proficiency_score``, which is the objective
|
|
``routing.rank_candidates`` orders on, and ``cost_score``, which it reports for
|
|
visibility without ranking on it.
|
|
|
|
The weighted composite below is NOT the scoring model any more. It was
|
|
``w_cost*cost + w_eco*eco + w_prof*proficiency`` until measurement retired it:
|
|
moving the cost weight from 0.4 to zero changed the winner in only 2 of 6
|
|
categories, so 40% of every decision was steering nothing, while eco was
|
|
optimizing a goal nobody had stated. Ranking is now quality first, cost as the
|
|
tiebreak inside ``quality_tolerance``, and a hard energy ceiling. ``eco_score``
|
|
and ``composite_score`` survive as arithmetic with tests; nothing in the
|
|
router calls them.
|
|
|
|
None-handling rule (consistent across cost/eco): candidates without data
|
|
are excluded from the min/max range computation and receive a neutral 0.5.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from collections.abc import Sequence
|
|
|
|
|
|
def normalize_inverted(values: Sequence[float | None]) -> list[float]:
|
|
"""Inverted min-max over a candidate set: lower input = higher score.
|
|
|
|
score = (max - value) / (max - min). A cheapest/cleanest candidate scores
|
|
1.0; the rest scale relative to max — this compression is intended.
|
|
|
|
None values are excluded from the min/max computation and get a neutral
|
|
0.5, so a model is never penalized merely for having no data yet. If no
|
|
candidate has data, all get 0.5. If min == max (all equal, or a single
|
|
candidate), present-data candidates get 1.0.
|
|
"""
|
|
known = [v for v in values if v is not None]
|
|
if not known:
|
|
return [0.5 for _ in values]
|
|
lo, hi = min(known), max(known)
|
|
if lo == hi:
|
|
return [1.0 if v is not None else 0.5 for v in values]
|
|
span = hi - lo
|
|
return [0.5 if v is None else (hi - v) / span for v in values]
|
|
|
|
|
|
# Cost and eco normalize identically; they differ only in what is fed to
|
|
# them. Both are kept as named axes because they rank models DIFFERENTLY:
|
|
# cost tracks energy (NeuralWatt bills per kWh) while carbon is energy times
|
|
# the serving region's grid intensity, and that intensity spans 37 gCO2/kWh
|
|
# (FI) to 505 (US-MIDA-PJM) across the catalog. glm-5.2-fast is the second
|
|
# cheapest model and only the sixth cleanest; kimi-k3-flex draws 3.7x less
|
|
# energy than kimi-k2.7-code while emitting 3.6x more carbon. Collapsing
|
|
# these into one axis would silently pick a side.
|
|
cost_score = normalize_inverted
|
|
eco_score = normalize_inverted
|
|
|
|
|
|
def proficiency_score(blended: float | None) -> float:
|
|
"""Thin proficiency lookup: pass through the blended value, else 0.5.
|
|
|
|
Row presence is irrelevant; only the value matters (Metis finding #2).
|
|
"""
|
|
return blended if blended is not None else 0.5
|
|
|
|
|
|
def composite_score(
|
|
cost_s: float,
|
|
eco_s: float,
|
|
prof_s: float,
|
|
weights: tuple[float, float, float] = (0.4, 0.2, 0.4),
|
|
) -> float:
|
|
"""Weighted composite of the three sub-scores.
|
|
|
|
Default weights (0.4, 0.2, 0.4) mirror config.yaml `weights`.
|
|
"""
|
|
w_cost, w_eco, w_prof = weights
|
|
return w_cost * cost_s + w_eco * eco_s + w_prof * prof_s
|