feat(config): gitignored config.local.yaml overlay, incident #5, backup tooling #26

alee · 2026-09-05T00:05:07Z

alee commented

2026-09-05 00:05:07 +00:00

Three related pieces of work from one session.

1. `config/config.local.yaml` overlay (Plan 8)

Deployment-specific values lived as uncommitted modifications to a tracked
file. That arrangement failed six times in one session -- three agent
tree-cleanings, two git commit -am sweeps needing history rewrites, and one
ordinary git switch. The last one is the point: once a value is correctly
absent from git, every checkout wipes it. The arrangement was the bug.

load_config now deep-merges an optional gitignored config/config.local.yaml
over the base file before validation. Merge rules: absent overlay means
identical behaviour, mappings deep-merge key by key, lists replace wholesale,
and the merged result is validated once so extra="forbid" still catches a
typo in the overlay.

Allowlisted admin scalar writes now target the overlay, so config/config.yaml
stops being dirty in normal operation.

2. Incident #5

git clean -fdx destroyed the database, API key and venv. Full write-up in
docs/incidents.md, including why the ignored files in this repo are the
deployment.

3. Backup tooling

llm-router-backup.timer -- hourly sqlite3 .backup snapshots, keeps 24,
integrity-checked before rotating.
llm-router-offsite.timer -- 6-hourly push to a private repo. .env is
excluded by construction.
deploy/workstation-backup.sh -- tarball with an embedded RESTORE.md.

Verification

1167 tests passing.
config/config.yaml byte-identical after a portal edit, with the overlay
carrying the change -- pinned by test_config_POST_preserves_comments_and_changes_value.
Overlay survives git restore of the base file -- test_overlay_survives_git_restore_of_base.
Deep merge verified live: setting local_energy.tariff_usd_per_kwh alone
leaves meter, sample_interval_seconds and grid_intensity_g_per_kwh intact.

Known gap, deliberately not fixed here

Profile CRUD still writes to config/config.yaml rather than the overlay
(src/admin.py:670 documents it). It predates the overlay decision. Tracked
as a follow-up rather than widened into this PR.

Three related pieces of work from one session. ## 1. `config/config.local.yaml` overlay (Plan 8) Deployment-specific values lived as **uncommitted modifications to a tracked file**. That arrangement failed six times in one session -- three agent tree-cleanings, two `git commit -am` sweeps needing history rewrites, and one ordinary `git switch`. The last one is the point: once a value is correctly absent from git, every checkout wipes it. The arrangement was the bug. `load_config` now deep-merges an optional gitignored `config/config.local.yaml` over the base file before validation. Merge rules: absent overlay means identical behaviour, mappings deep-merge key by key, lists replace wholesale, and the *merged* result is validated once so `extra="forbid"` still catches a typo in the overlay. Allowlisted admin scalar writes now target the overlay, so `config/config.yaml` stops being dirty in normal operation. ## 2. Incident #5 `git clean -fdx` destroyed the database, API key and venv. Full write-up in `docs/incidents.md`, including why the ignored files in this repo *are* the deployment. ## 3. Backup tooling - `llm-router-backup.timer` -- hourly `sqlite3 .backup` snapshots, keeps 24, integrity-checked before rotating. - `llm-router-offsite.timer` -- 6-hourly push to a private repo. `.env` is excluded by construction. - `deploy/workstation-backup.sh` -- tarball with an embedded `RESTORE.md`. ## Verification - **1167 tests passing.** - `config/config.yaml` byte-identical after a portal edit, with the overlay carrying the change -- pinned by `test_config_POST_preserves_comments_and_changes_value`. - Overlay survives `git restore` of the base file -- `test_overlay_survives_git_restore_of_base`. - Deep merge verified live: setting `local_energy.tariff_usd_per_kwh` alone leaves `meter`, `sample_interval_seconds` and `grid_intensity_g_per_kwh` intact. ## Known gap, deliberately not fixed here Profile CRUD still writes to `config/config.yaml` rather than the overlay (`src/admin.py:670` documents it). It predates the overlay decision. Tracked as a follow-up rather than widened into this PR.

alee added 8 commits 2026-09-05 00:05:08 +00:00

docs(plans): TUI overhaul -- catch the dashboard up to the schema c2b2fee452

The decision table declares "id, kind, category, tier, ctx, selected, est $,
flex" and has never had a timestamp -- observed_at is already carried by
decision_row and simply never rendered, so that one is a display fix.

Five columns were added to route_decisions across this session's merges and
none reached the TUI: profile (PR #25), exploration (proficiency branch),
pinch_original_tokens/pinch_final_tokens (PR #18), plus request_id and
session_key. The admin portal got a Profile column; the TUI did not, from the
same data.

Deliberately does NOT add six columns to an eight-column table on a terminal.
profile earns a column because it changes which models were considered at all;
exploration becomes a flag beside flex, because an epsilon-greedy pick is not a
ranking result and must not read as the router's judgement; the rest go to the
existing detail popup as per-decision forensics.

The quota panel section is explicitly SEQUENCED behind
plans/quota-balance-and-burn-rate.md and must not land before it -- a progress
bar against a plan figure that is routinely exceeded (146% observed, nothing
failed) implies a ceiling that does not exist. If that plan has not landed,
skip the panel rather than reimplementing balance/burn in the TUI.

The load-bearing success criterion is a test that diffs
PRAGMA table_info(route_decisions) against what the TUI model exposes and fails
when a column is added without a decision about surfacing it. The rest of this
plan is a one-time catch-up; that test is what stops the drift recurring.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U

docs(plans): outline multi-provider support 076f5289b0

Draft outline for generalizing routing beyond NeuralWatt: the real coupling
is narrow (poller.py's fetch_neuralwatt, and three tangled things in
dispatcher.py -- SSE-comment usage parsing, the kWh cost model, and
single-account refusal handling), while scoring/tiering/routing/proficiency
already operate on provider-agnostic DB rows. Recommends Z.ai as the first
test provider (same GLM weights already served via NeuralWatt, making it a
same-weights cross-provider comparison rather than a disjoint catalog),
with OpenRouter, DeepInfra, Together, and Fireworks as further candidates
for cheap prepaid-credit testing. Not decision-complete -- several open
questions flagged before this graduates to FINAL.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFq4QnaQx1CPvoEXoNijk2

docs(plans): beef up multi-provider draft with measured coupling surface ad39db2915

Counts the coupling instead of asserting it: grep -rn neuralwatt src/*.py
returns 20 hardcoded refs across 6 files, tabulated as the actual work list.

Three findings the draft did not have:

- metrics.py:175 computes catalog staleness over provider='neuralwatt' only,
  so a second provider whose poller stops would keep the warning green while
  its catalog froze -- the silent-and-open failure mode again, by a new door.
- leaderboard.py:125 writes priors to a hardcoded provider, so a curated prior
  for a shared model attaches to the NeuralWatt row only. Harmless today
  (leaderboards.yaml ships empty) but it decides the proficiency-sharing
  question by accident.
- dispatcher.py:2879 resolves a pinned model id against NeuralWatt rows only.
  Since capability flags fail closed, a pin on a second provider would 422 and
  read as a capability problem rather than provider scoping.

Also adds a risk the draft missed: poller.mark_stale runs only inside main(),
which returns early on RequestException, so with two providers one fetch
failure can skip staleness marking for the other -- or mark rows stale that
were never that provider's. Requires per-provider fetch isolation and a
per-provider sanity floor.

Closes three open questions by inspection: circuit_breaker IS provider-generic
(keyed on the (model_id, provider) tuple), 'no serving class' IS a real state
(parse_serving_class returns schema defaults), and proficiency's PK already
supports per-provider divergence -- though the propagate_to_variants sharing
question survives as a genuine design decision.

Corrects the 'needs ~zero change' list, which metrics.py and leaderboard.py
already contradict.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U

feat(config): gitignored config.local.yaml overlay 17d7b72c37

WIP checkpoint committed by Claude while Atlas was still in its final
verification wave. Committed early deliberately: 11 files were sitting
uncommitted with two of them untracked, and untracked files in this tree have
been destroyed twice today by agent git cleanup -- the user's
multi-provider-support plan and a config.local.yaml holding their tariff.
Protecting the work cost nothing; losing it would have cost a whole plan.

Deployment-specific values previously lived as an UNCOMMITTED modification to
the tracked config/config.yaml. That arrangement failed seven times in one
session: three agent checkout/stash/restore calls, two `git commit -am` sweeps
that each needed a history rewrite, one ordinary branch switch, and one
cleanup during this very plan. Once a value is correctly absent from git,
every checkout, switch, pull and rebase wipes it -- that is the intended fix
behaving as designed, which is what makes the arrangement itself the bug.

load_config now deep-merges an optional config/config.local.yaml over the base
before validation, so every consumer inherits it (poller, feedback,
eval_proficiency, seed_local_dispatch_energy, admin, dispatcher). Mappings
deep-merge; lists replace wholesale; the MERGED result is validated once so
extra="forbid" still catches an overlay typo.

The admin portal now writes to the overlay and never to config/config.yaml.
An operator changing a knob in a loopback-only portal is making a local
operational decision, not a project decision -- someone changing a project
default edits config.yaml and commits it through git. This also removes the
shadowing trap by construction rather than guarding against it.

config/config.yaml stops being dirty in normal operation, which removes the
condition behind every clobbering above.

NOTE: final verification wave had not reported when this was committed. The
suite was green at 1155 before it started; Atlas may amend or add commits on
top.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U

docs(incidents): #5 -- git clean -fdx destroyed the database, key and venv d570ebd671

First incident in this project with real data loss, and the first where
recovery depended on luck rather than design.

An agent ran `git clean -fdx`. The -x flag removes IGNORED files as well as
untracked ones, and everything this deployment needs to run is ignored by
design: router.db (truncated to 0 bytes -- 22,776 energy observations, 17,321
route decisions, 148 proficiency rows), .env (the NeuralWatt API key), .venv
(the virtualenv systemd's ExecStart runs from), and config/config.local.yaml
(the operator's tariff). Router returned 500 with "no such table:
energy_observations" while systemd reported active.

`git clean -fd` is a reasonable thing for an agent to run. Adding -x turns it
from "discard my scratch files" into "delete the deployment", and nothing in
the repo warned about that.

Recovery was luck: the plan running at the time had made a QA copy in /tmp 25
seconds before the wipe, and that copy happened to include .env. Restore was
effectively lossless. A different plan and the entire measurement history would
be gone.

Records the recovery runbook, and the gap it exposes: router.db has no backup
policy despite holding data that either costs money to rebuild (proficiency,
via eval runs) or cannot be rebuilt at all (historical observations).

Also notes what limited the damage -- committing the in-flight plan's 11 files
minutes earlier, and parking the tariff outside the repo -- both of which were
reactions to that same file being clobbered seven times the same day.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U

feat(deploy): hourly database snapshots and a workstation backup tarball 9dfe00acd9

Direct response to docs/incidents.md #5, where `git clean -fdx` truncated
router.db to 0 bytes and deleted .env, .venv and config/config.local.yaml.
Recovery was luck -- a QA copy happened to exist in /tmp from 25 seconds
earlier. There was no backup policy at all.

llm-router-backup.sh + .service + .timer: hourly, keeps 24. Uses sqlite3
.backup rather than cp, because copying a live database with an open writer can
capture a torn page set that passes a size check and fails integrity_check. It
verifies the new snapshot with integrity_check BEFORE rotating, so a failing run
never leaves fewer copies than it started with.

workstation-backup.sh: tarballs what git does not have -- router.db, .env,
config/config.local.yaml, .omo/ (plans and evidence ledger), tuned systemd
units, opencode plugins, and this project's Claude memory -- plus a RESTORE.md
with the clone -> venv -> restore sequence. Excludes .venv and node_modules
(rebuildable from pinned requirements) and Ollama models (~30GB, re-pullable,
recipes in docs/local-models.md).

Backups write OUTSIDE the repository by design. A backup kept inside it, even
gitignored, would have been destroyed by the same command that caused the
incident.

Test-restored before committing: 22,821 observations with integrity=ok, API key
present, tariff intact, 7 systemd units, 2 opencode plugins, 11 memory files.
That test caught a second loss nobody had noticed -- .omo/plans had also been
destroyed by the same clean, taking 35 plan artifacts and 127 evidence files,
since .omo/ is gitignored too. Recovered from the same QA copy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U

feat(deploy): 6-hourly off-site backup to a private git repo 4a32b0180f

Closes the last gap from docs/incidents.md #5. The hourly local snapshots
survive `git clean -fdx` because they live outside the repo, but they do not
survive the disk. This pushes the irreplaceable-and-small state to a private
Gitea repo: .omo plans and evidence ledger, tuned systemd units, opencode
plugins, this project's Claude memory, config/config.local.yaml, and a DAILY
gzipped router.db.

The database is committed daily rather than hourly on purpose: it is binary and
~5MB gzipped, so git cannot delta it. Hourly would grow the repo ~120MB/day
instead of ~5MB. SYNC_DB=0 turns it off entirely.

.env is deliberately NOT synced. There is no usable secret key on this host to
encrypt it to (public keys only), so it would sit in git history in plaintext,
and history is forever even in a private repo. An API key is replaceable by
regenerating it from the provider; 22,821 energy observations are not. It stays
in the local backups only, and the user confirmed that trade.

Verified by fresh clone: 35 plan artifacts, 127 evidence files, 9 systemd
units, 2 opencode plugins, 11 memory files, the tariff, and a 5.1MB db
snapshot. Checked the actual 67-character key VALUE appears in zero off-site
files -- an earlier check grepped for the variable NAME and for 'sk-', which
matches 'task-', and produced 50 false positives. Grep for the secret, not for
its label.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U

docs(plans): park multi-provider; correct a misattributed coupling site 0570123029

Provider selection is reopened -- Z.ai was recommended on the strength of
the GLM catalog overlap, which was an argument for the *criterion*, not a
confirmation that Z.ai's API or terms fit. Downgrade the recommendation to
a shortlist and keep the criterion for whatever candidate comes next.

Also corrects a real error. The draft attributed dispatcher.py:2879 to
_check_pinned_capabilities and called it a spurious-422 risk. That function
already takes provider as a parameter and is not a coupling site. 2879 is
_model_exists, and it fails open rather than closed: it decides whether a
`provider/model` string is an opencode alias to strip or a real id, so a
second-provider id resolves to False, gets stripped, and dispatches to
whatever the remainder matches -- silently, on a different provider, at a
different price. `vendor/model` is the native id format for OpenRouter and
most aggregators, so this is the default case, not an edge one.

Records that the 20 hardcoded neuralwatt literals are provider-agnostic
work that need not wait on the provider decision. Re-verified today: still
20 refs across the same six files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U

alee added 1 commit 2026-09-05 00:08:40 +00:00

docs(plans): profile CRUD to the overlay; provider literal cleanup be2c1ee93d

Two small, contained specs that can run as one wave.

admin-profile-writes-to-overlay: profile CRUD still writes config.yaml
while allowlisted scalars write the overlay. That split is chronological
accident -- profile CRUD shipped in PR #25, the overlay decision came
after. Redirect create/update to the overlay, scope delete to
overlay-defined profiles, and classify base-defined profiles as read-only
alongside built-ins, which sidesteps the fact that a deep merge cannot
express "remove" without tombstones.

Two defects found while specifying it, both present today. An
overlay-defined profile is LIVE in routing but invisible to the portal --
confirmed live: load_config sees it, _persisted_profiles() does not,
because the latter reads the base file only. And delete reads the merged
store for its default-profile guard but writes the base file, so an
overlay-defined profile is found and then 404s on the delete.

provider-literal-cleanup: the 20 hardcoded neuralwatt literals are wrong
whichever provider lands second, so they need not wait on the parked
multi-provider plan. Three are latent defects: _model_exists fails OPEN
and silently strips a real vendor/model id (the native format for
OpenRouter and most aggregators), catalog staleness is blind to a second
provider's frozen catalog, and leaderboard priors are pinned to one
provider -- which would decide the still-open proficiency-sharing question
by accident, so the spec explicitly refuses to resolve it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U

alee merged commit fc63eab554 into main

2026-09-05 00:28:38 +00:00

alee referenced this issue from a commit

2026-09-05 00:28:40 +00:00

Merge pull request 'feat(config): gitignored config.local.yaml overlay, incident #5, backup tooling' (#26) from feat/config-local-overlay into main

alee referenced this issue from a commit

2026-09-05 04:30:31 +00:00

docs(plans): correct a stale protected-file criterion in the quota plan

alee referenced this issue from a commit

2026-09-05 04:49:29 +00:00

docs(plans): refresh the ceiling-warnings plan against merged work

alee referenced this issue from a commit