9dfe00acd9
Direct response to docs/incidents.md #5, where `git clean -fdx` truncated router.db to 0 bytes and deleted .env, .venv and config/config.local.yaml. Recovery was luck -- a QA copy happened to exist in /tmp from 25 seconds earlier. There was no backup policy at all. llm-router-backup.sh + .service + .timer: hourly, keeps 24. Uses sqlite3 .backup rather than cp, because copying a live database with an open writer can capture a torn page set that passes a size check and fails integrity_check. It verifies the new snapshot with integrity_check BEFORE rotating, so a failing run never leaves fewer copies than it started with. workstation-backup.sh: tarballs what git does not have -- router.db, .env, config/config.local.yaml, .omo/ (plans and evidence ledger), tuned systemd units, opencode plugins, and this project's Claude memory -- plus a RESTORE.md with the clone -> venv -> restore sequence. Excludes .venv and node_modules (rebuildable from pinned requirements) and Ollama models (~30GB, re-pullable, recipes in docs/local-models.md). Backups write OUTSIDE the repository by design. A backup kept inside it, even gitignored, would have been destroyed by the same command that caused the incident. Test-restored before committing: 22,821 observations with integrity=ok, API key present, tariff intact, 7 systemd units, 2 opencode plugins, 11 memory files. That test caught a second loss nobody had noticed -- .omo/plans had also been destroyed by the same clean, taking 35 plan artifacts and 127 evidence files, since .omo/ is gitignored too. Recovered from the same QA copy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U
56 lines
2.4 KiB
Bash
Executable File
56 lines
2.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Snapshot the router's irreplaceable state.
|
|
#
|
|
# WHY THIS EXISTS: on 2026-09-04 an agent ran `git clean -fdx` in the repo.
|
|
# The -x flag removes IGNORED files, and everything this deployment needs is
|
|
# ignored by design -- router.db went to 0 bytes, taking 22,776 energy
|
|
# observations, 17,321 routing decisions and 148 proficiency scores, plus .env
|
|
# and the whole virtualenv. Recovery was luck: a QA copy happened to exist in
|
|
# /tmp from 25 seconds earlier. See docs/incidents.md #5.
|
|
#
|
|
# Backups therefore live OUTSIDE the repository. A backup inside it -- even
|
|
# gitignored -- would have been destroyed by the same command.
|
|
#
|
|
# What is NOT recoverable without this:
|
|
# - energy_observations : historical measurements, cannot be regenerated
|
|
# - route_decisions : same
|
|
# - proficiency : rebuildable only by re-running evals, which costs
|
|
# real provider credits
|
|
set -euo pipefail
|
|
|
|
REPO="${REPO:-$HOME/Sources/6krrt}"
|
|
DEST="${LLM_ROUTER_BACKUP_DIR:-$HOME/.local/share/6krrt-backups}"
|
|
KEEP="${LLM_ROUTER_BACKUP_KEEP:-24}"
|
|
|
|
mkdir -p "$DEST"
|
|
stamp=$(date +%Y%m%d-%H%M%S)
|
|
|
|
# .backup is the ONLY safe way to copy a live SQLite file. `cp` on a database
|
|
# with an open writer can produce a torn copy that passes a size check and
|
|
# fails integrity_check.
|
|
sqlite3 "$REPO/router.db" ".backup '$DEST/router-$stamp.db'"
|
|
gzip -f "$DEST/router-$stamp.db"
|
|
|
|
# Operator data that also lives only in ignored files.
|
|
[ -f "$REPO/.env" ] && { cp -f "$REPO/.env" "$DEST/env-$stamp.bak"; chmod 600 "$DEST/env-$stamp.bak"; }
|
|
[ -f "$REPO/config/config.local.yaml" ] && cp -f "$REPO/config/config.local.yaml" "$DEST/config.local-$stamp.yaml"
|
|
|
|
# Verify before rotating: a backup that has never been read is a guess.
|
|
tmp=$(mktemp)
|
|
zcat "$DEST/router-$stamp.db.gz" > "$tmp"
|
|
if [ "$(sqlite3 "$tmp" 'SELECT integrity_check FROM pragma_integrity_check LIMIT 1;')" != "ok" ]; then
|
|
rm -f "$tmp" "$DEST/router-$stamp.db.gz"
|
|
echo "backup FAILED integrity_check; discarded, older backups retained" >&2
|
|
exit 1
|
|
fi
|
|
rm -f "$tmp"
|
|
|
|
# Rotate only after a good backup exists, so a failing run never leaves you
|
|
# with fewer copies than you started with.
|
|
for pat in "router-*.db.gz" "env-*.bak" "config.local-*.yaml"; do
|
|
# shellcheck disable=SC2012
|
|
ls -1t "$DEST"/$pat 2>/dev/null | tail -n +$((KEEP + 1)) | xargs -r rm -f
|
|
done
|
|
|
|
echo "backup ok: $DEST/router-$stamp.db.gz ($(du -h "$DEST/router-$stamp.db.gz" | cut -f1)), keeping $KEEP"
|