6krrt/AGENTS.md

# AGENTS.md — working guide for AI agents on this repo

The three-tier docs:
- **`README.md`** — what's built, module by module (user-facing).
- **`CLAUDE.md`** — working state + immediate next steps + design rationale
  (the one to trust on what is currently true).
- **`design/local-llm-model-router.md`** — architecture and rationale,
  including parts still unbuilt.

This file is the quick-reference for an agent starting work: where things
live, what's safe to touch, and the conventions that aren't obvious from the
code.

## Stack snapshot

| Dimension | Value |
|---|---|
| Language | Python 3.10+ (3.10 floor tested; 3.14 also verified) |
| Framework | FastAPI + uvicorn |
| Database | SQLite (`router.db`) |
| Config | `config.yaml` + Pydantic (`config.py`), `extra="forbid"` |
| HTTP client | `requests` (pinned in `requirements.txt`) — do NOT add httpx2/aiohttp without a requirements bump |
| TUI | `textual==8.2.8` — imported only by `tui*.py` modules, never by the dispatch path |
| Testing | `pytest`, 562 tests, all offline (no provider or local-model calls) |
| Dependencies | Pinned. Bump deliberately, never use `>=` |

## Module map and file boundaries

### Dispatcher / service (I/O: DB + network)

| File | Role | Agent notes |
|---|---|---|
| `dispatcher.py` | FastAPI service: routes, calls providers, logs, streams | ~2500 LOC; only edit the specific function you need. **Must not import `metrics`** (circular). Imports `events` for SSE fan-out. |
| `metrics.py` | Read-only aggregations for `/health` and `/metrics` | Takes `(conn, cfg)` args — **must not import `dispatcher`** to avoid circular import. |
| `events.py` | In-memory decision-event broker | Pure stdlib (`queue`, `collections.deque`). Thread-safe. No Textual import. The SSE endpoint lives in `dispatcher.py` and calls `events.subscribe()`/`publish_decision()`. |
| `config.py` | Pydantic models + YAML loader | All config models inherit `StrictModel` (`extra="forbid"`). Unknown keys fail at load. |
| `capabilities.py` | Request-side capability detection (`tools`, `images`, `json_mode`, `reasoning`) | Reads from the OpenAI-format request body, not from the classifier. |
| `context_prune.py` | Relevance-based context pruning (`pinch`) | Ships **disabled** (`pinch.enabled: false`). Imports `PinchConfig` from `config.py` (no circular import: `config.py` doesn't import `context_prune`). |
| `logs.py` | Structured logging (logfmt, journald, ContextVar trace ids) | `logs.bind()` exists for StreamingResponse generators that lose the ContextVar. |

### Pure scoring/routing modules (no I/O)

| File | Role |
|---|---|
| `scoring.py` | `normalize_inverted` + weighted composite (quality-first, cost as tiebreak) |
| `routing.py` | Hard filters (`select_candidates`) + ranking (`rank_candidates`) |
| `tiering.py` | Pure tier resolver: 1=cheap+small, 2=mid, 3=frontier |
| `proficiency.py` | Score blending: leaderboard + self-eval → weighted composite |
| `iteration.py` | Retry budget per tier, matching retry to failure kind |

### TUI modules (import `textual`; never imported by the dispatch path)

| File | Role |
|---|---|
| `tui.py` | `DashboardApp` — main Textual app with CSS, bindings, compose, render |
| `tui_model.py` | Pure data layer: `build_model`, `build_category_breakdown`, `decision_row` — no Textual import, testable without a terminal |
| `tui_sse.py` | Background-thread SSE consumer (`DecisionStream`): reconnects on failure, marshals decisions to UI thread via `call_from_thread` |
| `tui_screens.py` | `DecisionDetailScreen` — `ModalScreen` showing full decision JSON (Enter or `e` key) |

### Other entrypoints

| File | Role |
|---|---|
| `poller.py` | Fetches Neuralwatt catalog, upserts `models` table |
| `seed_energy.py` | Reference workload sweep → `energy_observations` |
| `eval_proficiency.py` | Self-eval harness → `proficiency` table |
| `feedback.py` | Folds verification failures into `proficiency` |
| `tier.py` | DB tiering pass |
| `router_cli.py` | One-shot `/route` probe |
| `proficiency_store.py` | DB write path for `proficiency` |

## Conventions

### Code style
- The codebase uses `Optional[X]` (not `X | None`) and `dict` return types
  throughout — match the existing style in the file you're editing.
- No `# type: ignore`, no `as any`, no `@ts-ignore` equivalent.
- `except Exception` is acceptable at top-level boundaries with
  `# noqa: BLE001` comment (see `dispatcher._refresh`, `persist_route_decision`).
- Config is strict: every knob belongs in `config.yaml`, not only in a
  Pydantic default. A default the file never mentions is invisible to a tuner.
- Named constants use `Final` in new code (`events.py`); existing code is
  inconsistent — don't refactor just for this.

### Import discipline
- `metrics.py` **must not import `dispatcher`** (circular import).
- `events.py` imports nothing from the project (pure stdlib).
- `tui_model.py` imports `requests` but **not `textual`** — it's the testable data layer.
- `tui_sse.py` imports `requests` but **not `textual`** — it's a thread worker.
- `tui.py` and `tui_screens.py` import `textual` — that's fine, they're TUI.
- The service dispatch path (`dispatcher.py` → provider call) **never touches
  `textual`**.

### Database
- SQLite, `PRAGMA foreign_keys = ON`.
- `_db()` in `dispatcher.py` returns a `sqlite3.Connection` with
  `row_factory = sqlite3.Row`.
- Schema in `schema.sql` uses `CREATE TABLE IF NOT EXISTS` — safe to re-run.
- Code-side table creation (`ensure_route_decisions`, `proficiency_store.ensure_columns`)
  mirrors the schema for live DBs that predate a feature.

### Testing
- All tests are offline. No test calls a provider or local model.
- Pure modules take rows and config as arguments, so they're testable without DB/network.
- Test files that exercise FastAPI use `starlette.testclient.TestClient` with a
  temp SQLite DB (`tests/test_metrics_endpoint.py`).
- TUI tests use `App.run_test()` with a stubbed fetcher and a `_no_real_network`
  safety-net fixture that guards both `requests.get` and the SSE consumer.
- **SSE tests must not use `TestClient.stream()` on the infinite endpoint** — it
  hangs. Test the generator directly (`dispatcher._decision_event_stream()`)
  or stub the generator to a bounded one for header/route checks.

## How to run things

```bash
# Setup
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
sqlite3 router.db < schema.sql
cp .env.example .env   # fill in NEURALWATT_API_KEY

# Start the service (binds 127.0.0.1:8080)
python -m uvicorn dispatcher:app --reload
# Or via systemd:
systemctl --user start llm-router.service

# Run the TUI (service must be running)
python tui.py

# Run the full test suite
python -m pytest

# Quick routing probe (no spend)
python router_cli.py "Refactor this Django view"

# Populate the catalog
python poller.py && python tier.py
```

## What's NOT built yet (open items)

1. **Leaderboard priors unfilled** — `leaderboards.yaml` ships empty.
2. **Three models unsettled** on eco stability (attribution noise).
3. **Retry does not reach streaming** — `POST /outcome` is the answer for streamed traffic.
4. **Local energy not on the ledger** — local classifier/verifier electricity is unmeasured.
5. **Session-directory attribution picks the wrong directory** — heuristic resolves to a dependency's source dir instead of the project being edited.

See `CLAUDE.md` → "What's NOT built yet — pick up here" for the full list.

## After a code change

- Run `python -m pytest` — 562 tests, ~30s.
- If you changed `dispatcher.py`, restart the systemd service:
  `systemctl --user restart llm-router.service` (it doesn't auto-reload code).
- If you changed the TUI, run `python tui.py` to verify it starts.
- Check `lsp_diagnostics` on changed files.
- Match the existing commit-message style: `fix:`, `feat:`, `docs:`, `test:`.