6krrt/plans/multi-provider-support.md

Generalizing to multiple providers

Status: parked -- parked on provider selection; the coupling surface in it is still valid

Status: PARKED 2026-09-04 — DRAFT, not decision-complete, and now blocked on provider selection rather than on design. Written 2026-09-04 against main at 340453e; coupling surface measured and three open questions closed by code inspection on the same commit.

Parked by the user: Z.ai is no longer a settled first target. The GLM overlap that made it attractive is a real argument, but fit is unconfirmed and the candidate search is still open. Nothing below should be read as a commitment to a specific provider.

Remaining gaps before this can go to the opencode/Prometheus pipeline:

1. Which provider goes first (was treated as answered; it is not).
2. The proficiency-sharing decision — a judgement call, not a lookup.
3. The eco-without-telemetry decision.
4. Success criteria / test plan / phasing, still empty.

What does NOT need to wait. The measured coupling surface below is provider-agnostic: the 20 hardcoded neuralwatt references are literals that are wrong regardless of which provider lands second, and three of them are latent defects today. Re-verified on 2026-09-04 against feat/config-local-overlay — still 20 references, same six-file distribution. Those can be parameterised independently of this plan and should not be held hostage to the provider question.

Why this, why now

Everything downstream of the catalog — scoring.py, tiering.py, routing.py, proficiency.py, verification.py, feedback.py, the admin portal, the TUI — already operates on normalized DB rows with no idea which provider produced them. dispatch_providers: dict[str, DispatchProvider] in config and the (model_id, provider) composite key were deliberately kept when OpenRouter was dropped (c3484f0) for exactly this. The coupling that's actually left is narrow:

• poller.py — fetch_neuralwatt() is a bespoke function: hardcoded URL, NeuralWatt's specific catalog JSON shape, provider="neuralwatt" written as a literal.
• dispatcher.py — three things tangled together that need separating:
  1. Usage/telemetry parsing (NeuralWatt reports energy via SSE comment lines — a protocol quirk, not an OpenAI standard).
  2. Cost semantics (the $8/kWh-billed-capped-at-3x-list model is NeuralWatt-specific billing behavior). What routing.estimated_cost actually scores on — catalog price × request shape — is already provider-agnostic; the kWh math is validation for one provider's billing quirk, not the load-bearing input.
  3. Account-refusal handling (degrade to local assumes one cloud account — see dispatcher.py ~2537). With N providers this mostly disappears: circuit-break the failing provider's rows and let ranking fail over to the next-best candidate on a different provider.

On re-adding OpenRouter specifically: it was dropped for a real reason, not a bad one — the project pivoted to scoring on measured billing and carbon, and OpenRouter (an aggregator) doesn't expose per-request energy/carbon telemetry the way NeuralWatt does. That's not a reason to avoid it now — it's the first real test case for a has_energy_telemetry capability flag, since eco/energy needs to degrade gracefully per-provider rather than assuming every row has NeuralWatt's shape. Worth remembering before re-proposing it as if it were untried.

Constraint: cheap, no-commitment testing

Budget is small and this is for testing the plumbing, not production spend — prepaid credits in ~$10 increments, no subscriptions. Candidates, pricing and minimums need re-verification before committing to one (this kind of thing moves):

provider	fit	notes
Z.ai	best first target	Pay-as-you-go API, OpenAI-compatible (https://api.z.ai/api/paas/v4 or /api/openai/v1), no stated minimum top-up. Three free-tier models (GLM-4.7-Flash, GLM-4.5-Flash, GLM-4.6V-Flash) for zero-cost plumbing tests. Actually the publisher of the GLM family this catalog already serves via NeuralWatt (glm-5.2-fast, glm-5.2-flex) — a same-weights cross-provider comparison, which is a stronger generalization test than an unrelated model set, and a way to sanity-check NeuralWatt's static_fallback carbon figure for GLM. That is not hypothetical: CLAUDE.md records the GLM rows reporting grid_id: FI at 475 gCO2/kWh as carbon_source: static_fallback — a substituted constant, not a measurement — and they are currently EXCLUDED from eco scoring for that reason. A second provider serving the same weights is a free experiment on one of this project's standing open questions. No response-level energy/carbon telemetry — same capability-flag case as OpenRouter, not a differentiator there. Don't confuse this with the separate "GLM Coding Plan" subscription ($18-168/mo) — that's a different product and not what fits the budget constraint here.
OpenRouter	good second	prepaid, no minimum, OpenAI-compatible, many free models for zero-cost plumbing tests, broad catalog overlap (kimi/deepseek/qwen/gemma too, not just GLM), and this repo already has git history (c3484f0 and its parent) to mine for catalog-normalization shape. No per-request energy/carbon. It's an aggregator of aggregators, so if it ever does report grid/energy data, treat it as less trustworthy than a direct provider's own figure.
DeepInfra	plausible	prepaid, no subscription, OpenAI-compatible, cheap open-weight catalog. No energy telemetry.
Together AI	plausible	prepaid credits, OpenAI-compatible, per-token billing (straightforward vs. NeuralWatt's kWh math).
Fireworks AI	plausible	same shape as Together.
Groq	maybe later	OpenAI-compatible, free tier + pay-as-you-go, but a much smaller catalog — more useful for a latency-tolerance test than a routing-breadth test.

Recommendation, downgraded 2026-09-04. The shape still holds: one provider first, all the way through poller → dispatch → a handful of real routed requests, before touching a second. That is what confirms the abstraction generalizes instead of just looking like it does on paper.

Which provider is reopened. Z.ai was the pick on the strength of the GLM overlap — a same-weights cross-provider comparison, and a free experiment on the static_fallback carbon question. That argument is still good and should be re-used to score whatever candidate comes next; it was never a claim that Z.ai's API, pricing or terms actually fit. Treat the table above as a shortlist to re-verify, not a ranking to execute.

Worth noting for the search: the criterion that made Z.ai attractive — catalog overlap with what NeuralWatt already serves — is separable from Z.ai itself. OpenRouter carries kimi, deepseek, qwen and gemma alongside GLM, so it satisfies the same-weights test more broadly, and this repo already has git history (c3484f0 and its parent) with a working catalog normalizer to mine. It was dropped for a reason that no longer disqualifies it, now that has_energy_telemetry is the planned answer to missing energy data.

Architecture sketch

A Provider protocol/interface — not fleshed out yet, but the shape:

• fetch_catalog() -> list[ModelRow]
• parse_usage(response) -> UsageInfo (cost, tokens, energy if present)
• detect_refusal(error) -> bool
• capability flags: has_energy_telemetry, has_regional_carbon

fetch_neuralwatt and the SSE-comment parser move behind it as the first concrete implementation. dispatch_providers entries get a type: discriminator so config knows which implementation to instantiate.

Measured coupling surface

Claims about how narrow the coupling is should be counted, not asserted. grep -rn neuralwatt src/*.py on main at 340453e returns 20 hardcoded references across 6 files:

file	refs	what they are
poller.py	8	the bespoke fetch: URL, fetch_neuralwatt(), literal provider="neuralwatt", the sanity-floor count query, log prefixes
eval_proficiency.py	5	provider: str = "neuralwatt" defaults, the judge's dispatch_providers["neuralwatt"] lookup, and an identity["provider"] != "neuralwatt" skip
dispatcher.py	4	passthrough default, and a capability lookup (see below)
metrics.py	1	catalog-staleness query
leaderboard.py	1	set_leaderboard(..., "neuralwatt", ...)
seed_energy.py	1	dispatch_providers["neuralwatt"]

That is the actual work list. Two entries deserve calling out because they contradict the "needs ~zero change" section below.

metrics.py:175 — catalog staleness would ignore a second provider

SELECT MAX(last_updated) AS last_updated FROM models WHERE provider='neuralwatt'

The staleness warning is computed over NeuralWatt rows only. Add a provider whose poller silently stops and the warning stays green while its catalog freezes — the exact silent-and-open failure CLAUDE.md's "Run as a service" section describes, now with a second way in. metrics.py is listed under "needs ~zero change"; it does not.

leaderboard.py:125 — priors are written to a hardcoded provider

set_leaderboard(conn, cfg, model_id, "neuralwatt", category, score) pins the provider literal, so a curated prior for a shared model (GLM, say) attaches to the NeuralWatt row and never to the Z.ai one. leaderboards.yaml ships empty today so nothing is broken yet, but this decides the answer to the proficiency-sharing question above by accident rather than on purpose.

dispatcher.py:2879 — _model_exists silently strips a real model id

Corrected 2026-09-04. An earlier revision of this plan attributed this line to _check_pinned_capabilities. That was wrong: _check_pinned_capabilities (dispatcher.py:2887) already takes provider as a parameter and passes it to its query. It is not a coupling site. Line 2879 belongs to _model_exists, and the defect there is worse.

SELECT 1 FROM models WHERE model_id = ? AND provider = 'neuralwatt'

_model_exists decides whether a provider/model string a client sent is an opencode-style alias to strip (llm-router/...) or a real id that merely contains a slash. Scoped to NeuralWatt, a real second-provider id returns False and gets treated as an alias.

That matters more than it looks, because vendor/model is the native id format for the two strongest candidates: OpenRouter ids are always deepseek/deepseek-chat-shaped, and several other aggregators follow suit. So the failure is not exotic — it is the default case for a whole class of provider.

And it fails silently in the wrong direction. A provider-scoped 422 would at least be loud. This one strips the prefix and proceeds with the remainder, so a client pinning deepseek/deepseek-chat gets whatever deepseek-chat resolves to — a different row, on a different provider, at a different price — with nothing in the response indicating a substitution occurred. Fail-closed was the design intent everywhere else in this capability path; here it fails open.

Requirement: _model_exists must resolve across all configured providers, and alias-stripping must be decided by something other than "no NeuralWatt row has this id".

Risk not in the draft: per-provider fetch isolation

poller.mark_stale runs only inside poller.main(), and main() returns early on RequestException — before upsert and before mark_stale. With two providers in one run, a failure fetching provider B can skip mark_stale for provider A entirely, or a partial/empty data array from B can mark rows stale that were never B's. docs/incidents.md records the empty-data path as the one that can empty the candidate set.

Requirement: each provider's fetch, upsert and staleness marking must be isolated. One provider's outage must not affect another's rows in either direction, and the sanity floor (poller.py:420) must be per-provider.

What should need ~zero change

Scoring, tiering, routing, proficiency, verification, feedback, admin, TUI. If any of these turn out to need provider-aware branching, that's a sign the interface boundary is in the wrong place — worth treating as a red flag during implementation, not a shrug.

Two of them already fail that test, per the inventory above: metrics.py hardcodes the provider in its staleness query and leaderboard.py hardcodes it when writing priors. Neither is a deep coupling — both are literals, not branching — but the list should be read as "should need ~zero change after those two literals are parameterised", not as a claim that they are already clean. The red flag to watch for during implementation is provider-aware logic appearing in these modules; a hardcoded string is a different and much cheaper problem.

Open questions

• Eco/carbon for a provider with no telemetry. Exclude those rows from eco ranking entirely, or treat eco as unweighted/missing for them without disqualifying them? Affects whether a non-NeuralWatt row can ever win on the eco axis, or only ever competes on cost/quality.
• Same model, two providers — per-model or per-(model, provider)? PARTLY RESOLVED, and the remaining half is a design decision rather than a verification. The schema already supports divergence: proficiency's primary key is (model_id, provider, category). But propagate_to_variants copies scores across variants keyed on base_model_id, on the stated principle that proficiency is "a property of the weights, not the queue". Same-weights-different-serving-stack is exactly the case that principle does not decide: quantization and serving differences could justify separate scores, while the existing logic would share one. Decide this explicitly before implementation — it is the one question here that code inspection cannot answer.
• RESOLVED — circuit breaker is genuinely provider-generic. Verified in code, not prose: is_down, record_failure and record_success all take (model_id, provider) and _store is keyed on that tuple (src/circuit_breaker.py:29,37,57). Per-provider breaking works today with no change.
• Does the account-refusal-degrades-to-local path actually get simpler or just get an if added? Stated above as an assumption; worth confirming against the real code before it's a success criterion.
• RESOLVED — "no serving class" is already a real state. poller.parse_serving_class returns the schema defaults ('standard', 'default', 'full') for a base id with no suffix, so a Z.ai or OpenRouter row lands standard and is interactive-eligible rather than null. No change needed.

Non-goals (this pass)

• Don't wire up more than one new provider before the first one round-trips end to end.
• Don't try to make eco/carbon methodology uniform across providers that don't expose the same telemetry — graceful absence beats a fabricated number (same principle as the empty leaderboards.yaml).
• Don't rebuild the cost model — catalog-price × shape already generalizes; confirm that rather than redesigning it.

Not yet defined

Success criteria, test plan, and phasing/milestones — fill in once the open questions above are resolved and this graduates to FINAL.