feat: per-provider quota measurement and credit-aware routing attenuation #40

alee · 2026-09-06T19:01:29Z

alee commented

2026-09-06 19:01:29 +00:00

What this PR does

Per-provider quota measurement (fixes NeuralWatt-only global balance/burn)

Adds provider_balance_observations account-balance table and polls any provider with balance_url inside the existing 2h catalog poller.
OpenRouter configured with balance_url: https://openrouter.ai/api/v1/credits; its real credit balance now surfaces in /metrics.
Rewrites metrics.quota_balance_and_burn to return per-provider balance/burn/runway under quota.by_provider plus a total_balance_usd sum.
Removes the old flat quota keys (balance_usd, burn_rate_usd_per_hour, runway_*) from /metrics — a clean break, all in-repo consumers updated.
quota_burn meters kWh only against telemetry-capable providers (avoiding empty IN ()).

Opt-in credit-aware routing attenuation

New objective.credit_attenuation config block (enabled: false by default).
credit_attenuation_multiplier and a new rank_candidates effective_cost tiebreak: healthy providers stay at 1.0; low-balance providers get a multiplier up to max_multiplier.
Attenuation applies ONLY to providers with balance_url set (prepaid-account path). Telemetry-allowance-billed providers (NeuralWatt) are pinned to multiplier 1.0, so normal ~-0.004 USD overage readings never bias routing toward OpenRouter.
est_cost_usd and logged cost remain honest catalog estimates; effective_cost is routing-only.
Dispatcher resolver caches via module globals (refresh_seconds) and emits a structured log line when multipliers != 1.0.

Reviewer note: F2 mutation-testing catch-and-fix

The initial end-to-end attenuation test priced NeuralWatt cheaper than OpenRouter, so the test "flipped" to the healthy provider even without attenuation (tautological). A final-wave reviewer temporarily removed the provider_cost_multipliers passthrough from route() and the whole suite stayed green — the wiring had zero regression coverage. This was fixed by repricing the fixture so OpenRouter is raw-cheaper and only a working passthrough produces the flip; the mutation check now fails exactly the two flip tests (assert 'openrouter' == 'neuralwatt'). Full suite remains green at 1463 tests.

Verification

python -m pytest -q → 1463 passed, 0 failures.
Live throwaway-port QA: /metrics shows independent per-provider balances; attenuation probe flips selection from OpenRouter → NeuralWatt when OpenRouter credit is low.
No config/config.local.yaml changes; port 8080/production untouched.

Post-merge action

After merge, restart the production dispatcher: systemctl --user restart llm-router.service.

## What this PR does ### Per-provider quota measurement (fixes NeuralWatt-only global balance/burn) - Adds `provider_balance_observations` account-balance table and polls any provider with `balance_url` inside the existing 2h catalog poller. - OpenRouter configured with `balance_url: https://openrouter.ai/api/v1/credits`; its real credit balance now surfaces in `/metrics`. - Rewrites `metrics.quota_balance_and_burn` to return per-provider balance/burn/runway under `quota.by_provider` plus a `total_balance_usd` sum. - Removes the old flat quota keys (`balance_usd`, `burn_rate_usd_per_hour`, `runway_*`) from `/metrics` — a clean break, all in-repo consumers updated. - `quota_burn` meters kWh only against telemetry-capable providers (avoiding empty `IN ()`). ### Opt-in credit-aware routing attenuation - New `objective.credit_attenuation` config block (`enabled: false` by default). - `credit_attenuation_multiplier` and a new `rank_candidates` `effective_cost` tiebreak: healthy providers stay at 1.0; low-balance providers get a multiplier up to `max_multiplier`. - Attenuation applies ONLY to providers with `balance_url` set (prepaid-account path). Telemetry-`allowance`-billed providers (NeuralWatt) are pinned to multiplier 1.0, so normal ~-0.004 USD overage readings never bias routing toward OpenRouter. - `est_cost_usd` and logged `cost` remain honest catalog estimates; `effective_cost` is routing-only. - Dispatcher resolver caches via module globals (`refresh_seconds`) and emits a structured log line when multipliers != 1.0. ### Reviewer note: F2 mutation-testing catch-and-fix The initial end-to-end attenuation test priced NeuralWatt cheaper than OpenRouter, so the test "flipped" to the healthy provider even without attenuation (tautological). A final-wave reviewer temporarily removed the `provider_cost_multipliers` passthrough from `route()` and the whole suite stayed green — the wiring had zero regression coverage. This was fixed by repricing the fixture so OpenRouter is raw-cheaper and only a working passthrough produces the flip; the mutation check now fails exactly the two flip tests (`assert 'openrouter' == 'neuralwatt'`). Full suite remains green at 1463 tests. ## Verification - `python -m pytest -q` → 1463 passed, 0 failures. - Live throwaway-port QA: `/metrics` shows independent per-provider balances; attenuation probe flips selection from OpenRouter → NeuralWatt when OpenRouter credit is low. - No `config/config.local.yaml` changes; port 8080/production untouched. ## Post-merge action After merge, restart the production dispatcher: `systemctl --user restart llm-router.service`.

alee added 13 commits 2026-09-06 19:01:29 +00:00

feat(routing): credit_attenuation_multiplier pure function 54cfd2e8b7

feat(config): DispatchProvider.balance_url with fail-closed parser/telemetry validators 2c4cee0fe9

feat(schema): provider_balance_observations table + poller record_balance 2f3f9607bb

feat(config): objective.credit_attenuation block, off by default 2663e867be

feat(routing): provider cost multipliers as an effective-cost rank tiebreak ad65fad8e3

fix(metrics): per-provider balance, burn, and runway instead of a global series b9f70c1745

feat(poller): poll provider account balances inside the catalog loop aa59300575

fix(metrics): quota report carries by_provider and total_balance_usd; flat keys removed 98b77a2028

feat(dispatcher): credit-aware routing attenuation behind opt-in config a5e1b9d1a5

feat(tui): per-provider quota panel rows and binding-provider lead 01a52204ba

feat(admin): per-provider balance and runway rows in quota chip and modal c1988392f6