6krrt/tiering.py

"""Pure tier resolver for the local LLM model router.

This module is deliberately free of I/O: no DB reads, no file reads, no
config import at runtime. The model row and thresholds are passed as
arguments so the resolver stays testable and reusable. The DB upsert
(``apply_tiering``) is a separate concern and lives in ``tier.py``.

Tier semantics (design doc §4 / config.yaml ``tiers`` labels):
    1 = cheap / simple
    2 = mid / general
    3 = frontier / high-stakes

Why not ``supports_reasoning``: that column mirrors the catalog's
``capabilities.reasoning``, which only means "this endpoint accepts a
reasoning parameter". It is true for all but two rows in the NeuralWatt
catalog, so tiering on it collapsed 17 of 19 models into tier 3 and left
tier 1 empty. The discriminating signals are ``reasoning_default_enabled``
(whether the model actually thinks unless told otherwise) and
``reasoning_mode`` (whether this is a ``-fast`` row with thinking disabled
or capped).

Tier is a capability FLOOR: ``routing.py`` excludes any row whose
``tier < required_tier``, so tier 1 does not mean "cheap", it means "only
suitable for simple work". That distinction is what rule 3 below turns on.

Resolution precedence (EXACT order):
    1. If ``override_map`` has this model's ``model_id`` -> that value wins.
    2. Reasoning is "effectively on" when ``reasoning_default_enabled`` is
       True AND ``reasoning_mode`` is not ``'reduced'``. A ``-fast`` row is
       the same weights served without chain-of-thought, so it does not earn
       tier 3 on its sibling's behalf.
    3. Reasoning effectively OFF, ``cost_per_1m_completion`` is not None and
       not ``pricing_tbd`` and < ``cheap_completion_max``, AND
       ``context_window`` is below ``tier1_context_max`` -> 1.
    4. Reasoning effectively ON -> 3.
    5. Otherwise (missing/NULL cost, ``pricing_tbd``, cost >= threshold, or a
       large context window) -> 2.

Rules 3 and 4 are mutually exclusive, so their relative order is immaterial;
cost is stated first because the previous version's reasoning-before-cost
ordering is what pushed $0.28/1M models into tier 3.

Why context window gates tier 1: price is a market signal, not a capability
measurement, and this project already learned that once in the cost domain —
list price ranks these models backwards. Tiering on price alone made the same
substitution and it capped ``deepseek-v4-flash`` at tier 1 on nothing but its
$0.28/1M completion price, which excluded it OUTRIGHT from every tier-2
request. It has a 1M advertised window and scores 1.00 on all three coding
categories in the eval set; "simple tasks only" is not a defensible reading of
that row. A model that can hold a million tokens of context is not a small
model however little it charges, so tier 1 now requires the model to be small
AND cheap rather than merely cheap.

``context_window`` (advertised) is used rather than
``effective_context_window`` because the advertised figure is the clean market
class — the catalog's values are 131056, 199984, 262128 and 1048560, i.e.
128K/200K/256K/1M — while the effective figures are provider-derived and vary
within a class (a 256K row reports 180212 or 192500 depending on family).
``tier1_context_max`` therefore sits in the wide empty band between the 256K
and 1M classes rather than being fitted to any one model.

NULL-cost semantics: tier 1 REQUIRES a non-NULL completion cost strictly
below the threshold. A missing/None ``context_window`` does NOT block tier 1 —
absent capability evidence should not promote a model.

The override map remains the escape hatch for a row the heuristic still gets
wrong.
"""

from __future__ import annotations


def reasoning_effectively_on(model_row: dict) -> bool:
    """Whether this row actually reasons by default.

    Falls back to ``supports_reasoning`` when ``reasoning_default_enabled``
    is absent, matching the poller's own fallback for models that expose no
    reasoning block (the kimi-k2.7-code family).
    """
    default_enabled = model_row.get("reasoning_default_enabled")
    if default_enabled is None:
        default_enabled = model_row.get("supports_reasoning", False)
    return bool(default_enabled) and model_row.get("reasoning_mode") != "reduced"


def resolve_tier(
    model_row: dict,
    cheap_completion_max: float,
    override_map: dict[str, int],
    tier1_context_max: float = float("inf"),
) -> int:
    """Resolve a model row to a tier in {1, 2, 3}.

    ``model_row`` is a row-like dict with at least these keys (from the
    ``models`` table / ``ModelRow``): ``model_id`` (str),
    ``reasoning_default_enabled`` (bool), ``reasoning_mode`` (str),
    ``cost_per_1m_completion`` (float or None), ``pricing_tbd`` (bool) and
    ``context_window`` (int or None, the ADVERTISED window).
    ``supports_reasoning`` is consulted only as a fallback.

    ``tier1_context_max`` defaults to infinity so an omitted argument keeps
    the pre-existing cost-only behaviour rather than silently re-tiering a
    caller that has not opted in.

    ``override_map`` is keyed by ``model_id`` only and therefore applies to
    ALL providers serving that model (documented limitation vs the
    (model_id, provider) primary key).
    """
    model_id = model_row["model_id"]

    if model_id in override_map:
        return override_map[model_id]

    if not reasoning_effectively_on(model_row):
        cost = model_row["cost_per_1m_completion"]
        # A missing window is not evidence of a small model, so it does not
        # block tier 1 — only a window we can see and that is large does.
        window = model_row.get("context_window")
        large_context = window is not None and window >= tier1_context_max
        if (
            not model_row["pricing_tbd"]
            and cost is not None
            and cost < cheap_completion_max
            and not large_context
        ):
            return 1
        return 2

    return 3