6krrt/AGENTS.md

AGENTS.md — working guide for AI agents on this repo

The three-tier docs:

• README.md — what's built, module by module (user-facing).
• CLAUDE.md — working state + immediate next steps + design rationale (the one to trust on what is currently true).
• design/local-llm-model-router.md — architecture and rationale, including parts still unbuilt.

This file is the quick-reference for an agent starting work: where things live, what's safe to touch, and the conventions that aren't obvious from the code.

Stack snapshot

Dimension	Value
Language	Python 3.10+ (3.10 floor tested; 3.14 also verified)
Framework	FastAPI + uvicorn
Database	SQLite (router.db)
Config	config.yaml + Pydantic (config.py), extra="forbid"
HTTP client	requests (pinned in requirements.txt) — do NOT add httpx2/aiohttp without a requirements bump
TUI	textual==8.2.8 — imported only by tui*.py modules, never by the dispatch path
Testing	pytest, 562 tests, all offline (no provider or local-model calls)
Dependencies	Pinned. Bump deliberately, never use >=

Module map and file boundaries

Dispatcher / service (I/O: DB + network)

File	Role	Agent notes
dispatcher.py	FastAPI service: routes, calls providers, logs, streams	~2500 LOC; only edit the specific function you need. Must not import metrics (circular). Imports events for SSE fan-out.
metrics.py	Read-only aggregations for /health and /metrics	Takes (conn, cfg) args — must not import dispatcher to avoid circular import.
events.py	In-memory decision-event broker	Pure stdlib (queue, collections.deque). Thread-safe. No Textual import. The SSE endpoint lives in dispatcher.py and calls events.subscribe()/publish_decision().
config.py	Pydantic models + YAML loader	All config models inherit StrictModel (extra="forbid"). Unknown keys fail at load.
capabilities.py	Request-side capability detection (tools, images, json_mode, reasoning)	Reads from the OpenAI-format request body, not from the classifier.
context_prune.py	Relevance-based context pruning (pinch)	Ships disabled (pinch.enabled: false). Imports PinchConfig from config.py (no circular import: config.py doesn't import context_prune).
logs.py	Structured logging (logfmt, journald, ContextVar trace ids)	logs.bind() exists for StreamingResponse generators that lose the ContextVar.

Pure scoring/routing modules (no I/O)

File	Role
scoring.py	normalize_inverted + weighted composite (quality-first, cost as tiebreak)
routing.py	Hard filters (select_candidates) + ranking (rank_candidates)
tiering.py	Pure tier resolver: 1=cheap+small, 2=mid, 3=frontier
proficiency.py	Score blending: leaderboard + self-eval → weighted composite
iteration.py	Retry budget per tier, matching retry to failure kind

TUI modules (import textual; never imported by the dispatch path)

File	Role
tui.py	DashboardApp — main Textual app with CSS, bindings, compose, render
tui_model.py	Pure data layer: build_model, build_category_breakdown, decision_row — no Textual import, testable without a terminal
tui_sse.py	Background-thread SSE consumer (DecisionStream): reconnects on failure, marshals decisions to UI thread via call_from_thread
tui_screens.py	DecisionDetailScreen — ModalScreen showing full decision JSON (Enter or e key)

Other entrypoints

File	Role
poller.py	Fetches Neuralwatt catalog, upserts models table
seed_energy.py	Reference workload sweep → energy_observations
eval_proficiency.py	Self-eval harness → proficiency table
feedback.py	Folds verification failures into proficiency
tier.py	DB tiering pass
router_cli.py	One-shot /route probe
proficiency_store.py	DB write path for proficiency

Conventions

Code style

• The codebase uses Optional[X] (not X | None) and dict return types throughout — match the existing style in the file you're editing.
• No # type: ignore, no as any, no @ts-ignore equivalent.
• except Exception is acceptable at top-level boundaries with # noqa: BLE001 comment (see dispatcher._refresh, persist_route_decision).
• Config is strict: every knob belongs in config.yaml, not only in a Pydantic default. A default the file never mentions is invisible to a tuner.
• Named constants use Final in new code (events.py); existing code is inconsistent — don't refactor just for this.

Import discipline

• metrics.py must not import dispatcher (circular import).
• events.py imports nothing from the project (pure stdlib).
• tui_model.py imports requests but not textual — it's the testable data layer.
• tui_sse.py imports requests but not textual — it's a thread worker.
• tui.py and tui_screens.py import textual — that's fine, they're TUI.
• The service dispatch path (dispatcher.py → provider call) never touches textual.

Database

• SQLite, PRAGMA foreign_keys = ON.
• _db() in dispatcher.py returns a sqlite3.Connection with row_factory = sqlite3.Row.
• Schema in schema.sql uses CREATE TABLE IF NOT EXISTS — safe to re-run.
• Code-side table creation (ensure_route_decisions, proficiency_store.ensure_columns) mirrors the schema for live DBs that predate a feature.

Testing

• All tests are offline. No test calls a provider or local model.
• Pure modules take rows and config as arguments, so they're testable without DB/network.
• Test files that exercise FastAPI use starlette.testclient.TestClient with a temp SQLite DB (tests/test_metrics_endpoint.py).
• TUI tests use App.run_test() with a stubbed fetcher and a _no_real_network safety-net fixture that guards both requests.get and the SSE consumer.
• SSE tests must not use TestClient.stream() on the infinite endpoint — it hangs. Test the generator directly (dispatcher._decision_event_stream()) or stub the generator to a bounded one for header/route checks.

How to run things

# Setup
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
sqlite3 router.db < schema.sql
cp .env.example .env   # fill in NEURALWATT_API_KEY

# Start the service (binds 127.0.0.1:8080)
python -m uvicorn dispatcher:app --reload
# Or via systemd:
systemctl --user start llm-router.service

# Run the TUI (service must be running)
python tui.py

# Run the full test suite
python -m pytest

# Quick routing probe (no spend)
python router_cli.py "Refactor this Django view"

# Populate the catalog
python poller.py && python tier.py

What's NOT built yet (open items)

1. Leaderboard priors unfilled — leaderboards.yaml ships empty.
2. Three models unsettled on eco stability (attribution noise).
3. Retry does not reach streaming — POST /outcome is the answer for streamed traffic.
4. Local energy not on the ledger — local classifier/verifier electricity is unmeasured.
5. Session-directory attribution picks the wrong directory — heuristic resolves to a dependency's source dir instead of the project being edited.

See CLAUDE.md → "What's NOT built yet — pick up here" for the full list.

After a code change

• Run python -m pytest — 562 tests, ~30s.
• If you changed dispatcher.py, restart the systemd service: systemctl --user restart llm-router.service (it doesn't auto-reload code).
• If you changed the TUI, run python tui.py to verify it starts.
• Check lsp_diagnostics on changed files.
• Match the existing commit-message style: fix:, feat:, docs:, test:.