6krrt/.omo/notepads/multi-provider-openrouter/learnings.md

multi-provider-openrouter learnings

Task 1: Extend DispatchProvider config model for OpenRouter

• DispatchProvider in src/config.py now carries has_energy_telemetry: bool = False and enabled: bool = True.
• Config is strict (extra="forbid") — adding new keys to dispatch_providers entries without matching model fields would fail at load.
• config/config.yaml keeps neuralwatt enabled with energy telemetry on, and adds an openrouter provider entry without telemetry.
• .env.example now lists both NEURALWATT_API_KEY= and OPENROUTER_API_KEY=.
• Full suite passes (1251 tests). Verification script exits 0.

Task 5: Multi-provider main loop in poller

• Refactored poller.main() to iterate over cfg.dispatch_providers.items() instead of always polling dispatch_settings.default_provider.
• main() now skips disabled providers, dispatches neuralwatt rows via fetch_neuralwatt() and openrouter rows via fetch_openrouter(), and logs/skips unknown provider keys.
• Per-provider fetch failures (including CatalogTooSmall and zero-row catalogs) are caught and logged; other enabled providers continue unaffected.
• mark_stale() gained an optional provider= kwarg that scopes the staleness update with AND provider = ?; calling without provider retains the historical global behavior, preserving any external callers/tests.
• After all providers are processed, main() calls tier.apply_tiering(conn, cfg) so freshly polled rows are immediately routable without a separate tier run.
• Existing freshness tests in tests/test_poller_freshness.py were updated to isolate the neuralwatt provider under test and to expect provider-skip semantics rather than process abort on fetch error/zero rows.
• Added tests/test_multi_provider_poller.py with tests for: two providers both upserting, failure isolation (one fails, other succeeds), disabled provider skip, zero-row provider skip log, per-provider mark_stale scoping, unknown provider key skip, and backward-compatible global mark_stale.
• Verification: python -m py_compile src/poller.py tests/test_multi_provider_poller.py passes; targeted pytest suite passes (64 tests).

Task 6: Fix passthrough pin dispatch to resolve owning provider from catalog

• Added _resolve_pinned_provider(model_id) to src/dispatcher.py that queries models for active rows across configured providers, preferring cfg.dispatch_settings.default_provider when the same model_id exists on multiple providers.
• Updated alias stripping to handle opencode's llm-router/<vendor>/<model> prefix before falling back to rsplit("/", 1)[-1]; this preserves full OpenRouter ids like openai/gpt-6-astra instead of mangling them to gpt-6-astra.
• Passthrough branch now uses _resolve_pinned_provider(requested) or cfg.dispatch_settings.default_provider and passes the resolved provider to _check_pinned_capabilities(...).
• Added three tests in tests/test_chat_completions.py covering OpenRouter-only pin dispatch, llm-router/openai/gpt-6-astra prefix handling, and Neuralwatt-only pins still using the default provider.
• Full suite remains green; changed path verified by tests/test_chat_completions.py.

Task 8: Admin provider CRUD tests

• Expanded tests/test_admin_providers.py from 3 to 8 tests using isolated temporary config directories.

• DELETE /admin/api/providers/neuralwatt (the current dispatch_settings.default_provider) returns 422 and leaves both config.yaml and config.local.yaml untouched.

• DELETE /admin/api/providers/openrouter (base-configured, non-default provider) returns 403 with a detail naming config/config.yaml and read-only provenance.

• DELETE /admin/api/providers/nonexistent returns 404 and does not modify config.yaml.

• POST /admin/api/providers/testprov writes only to config.local.yaml, leaves config.yaml unchanged, and creates a config.local.yaml.bak.<ts> backup.

• GET /admin/api/providers returns the merged view, including both base providers (neuralwatt, openrouter) and an overlay provider (overlayprov).

• Verification: PYTHONPATH=src python -m pytest tests/test_admin_providers.py -x --tb=short passes (8 tests, ~2 s). No real provider APIs are called because the tests target the admin router's in-memory config handling only.

• Added fetch_openrouter(provider) and parse_openrouter_model(raw_model) to src/poller.py.

• Virtual routers (openrouter/auto, openrouter/auto-beta, openrouter/free, openrouter/fusion, openrouter/pareto-code, openrouter/bodybuilder) are dropped before becoming rows.

• Variant suffixes are preserved; :batch maps to latency_class='flex', other variants and base rows to 'standard'.

• Catalog id is used as model_id; canonical_slug is used as base_model_id.

• Per-token USD strings from pricing.prompt and pricing.completion are converted to cost per 1M tokens.

• pricing.web_search and pricing.cache_read differentials are intentionally omitted from catalog cost estimates.

• Added tests/test_openrouter_poller.py with 15 mocked tests covering row construction, variant handling, virtual-router exclusion, malformed prices, and request-exception propagation.

• Full suite passes (1266 tests).

Task 9: Verify poller and multi-provider backend test coverage

• Confirmed tests/test_openrouter_poller.py already covers: fetch_openrouter() row construction, virtual-router exclusion, :batch -> latency_class='flex', :free/base -> 'standard', malformed pricing handling, and request exception propagation.
• Confirmed tests/test_multi_provider_poller.py already covers: two mocked providers both upsert, failure isolation, disabled provider skip, zero-row provider skip, per-provider mark_stale() scoping, unknown provider key skip, and backward-compatible global mark_stale().
• Added test_in_process_tiering_runs_after_upsert to explicitly verify that main() calls in-process tiering so OpenRouter rows receive a non-null tier immediately after the poll.
• Targeted verification passes: PYTHONPATH=src python -m pytest tests/test_openrouter_poller.py tests/test_multi_provider_poller.py tests/test_poller_freshness.py -x --tb=short -> 30 passed.
• poller.py coverage on these files: 199 stmts, 12 miss, 94%.
• Evidence recorded in .omo/evidence/task-9-multi-provider-openrouter.json.

Task 7: Per-provider telemetry and refusal isolation

• extract_telemetry({}) now safely returns a Telemetry() instance with all None fields; added an explicit non-dict guard so a malformed payload does not crash downstream logging.
• _sniff_telemetry_line(": OPENROUTER PROCESSING") returns None because only comments whose first word is exactly "energy" or "cost" are treated as telemetry.
• Replaced module-level _last_account_refusal: float with _provider_refusal_since: dict[str, float].
• Renamed _record_account_refusal() to _record_provider_refusal(provider) and keyed the timestamp by provider.
• Updated the classifier cascade refusal check (around line 654) to consult _provider_refusal_since.get(cfg.dispatch_settings.default_provider) using cfg.classifier.cooldown_seconds.
• Updated non-streaming and streaming dispatch refusal triggers to call _record_provider_refusal(provider).
• SSE : energy / : cost sniffing is now gated by cfg.dispatch_providers[provider].has_energy_telemetry; when false, telemetry comments are dropped rather than proxied.
• Added a module _self_check() that runs at import time to lock extract_telemetry({}) and _sniff_telemetry_line behavior.
• Targeted tests tests/test_classifier_cascade.py and tests/test_local_dispatch_fallback.py fail as expected because they still reference the removed _last_account_refusal / _record_account_refusal symbols; task 10 will migrate those fixtures.
• Verification: python -m py_compile src/dispatcher.py passes; PYTHONPATH=src python -c "from dispatcher import extract_telemetry; t = extract_telemetry({}); assert t.energy_kwh is None" passes.
• Evidence recorded in .omo/evidence/task-7-multi-provider-openrouter.json.

Task 10: migrate classifier cascade tests to per-provider refusal symbols

• Updated tests/test_classifier_cascade.py _clean_state fixture: replaced the removed _last_account_refusal monkeypatch with dispatcher._provider_refusal_since.clear() (reset both before and after each test).
• Renamed test_account_refusal_skips_the_cloud_step to test_provider_refusal_skips_the_cloud_step and replaced _record_account_refusal() with _record_provider_refusal("neuralwatt").
• Verified tests/test_local_dispatch_fallback.py contains no references to _last_account_refusal / _record_account_refusal; the existing tests continue to pass because _provider_refusal_since is scoped per provider and the helper unit tests exercise _account_level_refusal() semantics unchanged.
• Added tests/test_multi_provider_dispatch.py with two focused tests:
  • test_openrouter_streaming_ignores_keepalive_comments_and_logs_null_telemetry: a streamed OpenRouter response containing : OPENROUTER PROCESSING keeps the comment in the proxied body, does not crash, and writes an energy_observations row with energy_kwh and cost_usd NULL.
  • test_provider_refusal_is_isolated_between_providers: recording a refusal on neuralwatt skips the cloud classifier when default_provider=neuralwatt, but does not skip it when default_provider=openrouter.
• Targeted run: PYTHONPATH=src python -m pytest tests/test_classifier_cascade.py tests/test_local_dispatch_fallback.py tests/test_multi_provider_dispatch.py -x --tb=short → 39 passed.
• Full suite: PYTHONPATH=src python -m pytest --tb=short → 1288 passed, zero failures.
• Evidence recorded in .omo/evidence/task-10-multi-provider-openrouter.json.

Task 11: Final verification — Playwright smoke and full test suite

• Threw away uvicorn on port 8081 (never touched port 8080 / systemd). Verified /admin/providers and /admin/api/providers return 200 before driving the UI.
• Playwright (Chromium, headless) smoke executed at 1400px and 800px.
  • Passed: page renders, Providers navbar link visible, base providers neuralwatt and openrouter displayed, create provider smoketest appears in the list, XSS payload <script>alert('xss')</script> is escaped in the DOM, and 30s idle produced zero console errors.
  • Failed / defect found: newly-created overlay providers render with source badge unknown and no edit/delete buttons because GET /admin/api/providers omits in_base/in_overlay provenance fields. The UI (providers.html) keys editability off in_overlay, so the toggle and delete flow cannot be exercised via the real page. Root cause: src/admin.py:admin_providers_list() returns only name, base_url, api_key_env, has_energy_telemetry, enabled; it does not include the provenance flags returned for profiles.
• Full pytest suite passes: 1288 passed, 0 failed. Command: PYTHONPATH=src python -m pytest --tb=short.
• Overlay cleanup: restored config/config.local.yaml to its pre-test state (local_energy block only); backup files created by _persist_to remain in config/ but are gitignored; git status is clean.
• Evidence, screenshots (1400/800), and console log captured in .omo/evidence/task-11-multi-provider-openrouter.json.