9dfe00acd9
Direct response to docs/incidents.md #5, where `git clean -fdx` truncated router.db to 0 bytes and deleted .env, .venv and config/config.local.yaml. Recovery was luck -- a QA copy happened to exist in /tmp from 25 seconds earlier. There was no backup policy at all. llm-router-backup.sh + .service + .timer: hourly, keeps 24. Uses sqlite3 .backup rather than cp, because copying a live database with an open writer can capture a torn page set that passes a size check and fails integrity_check. It verifies the new snapshot with integrity_check BEFORE rotating, so a failing run never leaves fewer copies than it started with. workstation-backup.sh: tarballs what git does not have -- router.db, .env, config/config.local.yaml, .omo/ (plans and evidence ledger), tuned systemd units, opencode plugins, and this project's Claude memory -- plus a RESTORE.md with the clone -> venv -> restore sequence. Excludes .venv and node_modules (rebuildable from pinned requirements) and Ollama models (~30GB, re-pullable, recipes in docs/local-models.md). Backups write OUTSIDE the repository by design. A backup kept inside it, even gitignored, would have been destroyed by the same command that caused the incident. Test-restored before committing: 22,821 observations with integrity=ok, API key present, tariff intact, 7 systemd units, 2 opencode plugins, 11 memory files. That test caught a second loss nobody had noticed -- .omo/plans had also been destroyed by the same clean, taking 35 plan artifacts and 127 evidence files, since .omo/ is gitignored too. Recovered from the same QA copy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U
87 lines
3.6 KiB
Bash
Executable File
87 lines
3.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Tarball everything about THIS workstation that git does not have.
|
|
#
|
|
# The repo itself is pushed to Gitea and needs no backup. What has no other
|
|
# copy is the ignored/untracked state around it -- and docs/incidents.md #5
|
|
# records what happens when that is lost: `git clean -fdx` took router.db,
|
|
# .env, .venv and config/config.local.yaml in one command, and recovery was
|
|
# luck.
|
|
#
|
|
# INCLUDED (irreplaceable or expensive to recreate):
|
|
# router.db measurement history; cannot be regenerated
|
|
# .env provider API key
|
|
# config/config.local.yaml operator tariff and local overrides
|
|
# .omo/ plans, evidence ledger, boulder state
|
|
# ~/.config/systemd/user tuned units (the graceful-shutdown fix etc.)
|
|
# ~/.config/opencode agent plugins incl. session-registry.js
|
|
# ~/.claude/.../memory cross-session memory for this project
|
|
#
|
|
# EXCLUDED on purpose:
|
|
# .venv rebuildable: python -m venv .venv && pip install -r requirements.txt
|
|
# node_modules rebuildable
|
|
# ollama models ~30GB and re-pullable; Modelfile recipes are in docs/local-models.md
|
|
# .git the remote has it
|
|
set -euo pipefail
|
|
|
|
REPO="${REPO:-$HOME/Sources/6krrt}"
|
|
DEST="${LLM_ROUTER_BACKUP_DIR:-$HOME/.local/share/6krrt-backups}"
|
|
KEEP="${WORKSTATION_BACKUP_KEEP:-7}"
|
|
|
|
mkdir -p "$DEST"
|
|
stamp=$(date +%Y%m%d-%H%M%S)
|
|
out="$DEST/workstation-$stamp.tar.gz"
|
|
staging=$(mktemp -d)
|
|
trap 'rm -rf "$staging"' EXIT
|
|
|
|
# Snapshot the DB properly rather than tarring a live file: `cp`/`tar` on an
|
|
# open SQLite database can capture a torn page set that still looks valid.
|
|
sqlite3 "$REPO/router.db" ".backup '$staging/router.db'"
|
|
|
|
mkdir -p "$staging/repo/config"
|
|
[ -f "$REPO/.env" ] && cp -f "$REPO/.env" "$staging/repo/.env"
|
|
[ -f "$REPO/config/config.local.yaml" ] && cp -f "$REPO/config/config.local.yaml" "$staging/repo/config/"
|
|
[ -d "$REPO/.omo" ] && cp -a "$REPO/.omo" "$staging/repo/.omo"
|
|
|
|
mkdir -p "$staging/home"
|
|
[ -d "$HOME/.config/systemd/user" ] && cp -a "$HOME/.config/systemd/user" "$staging/home/systemd-user"
|
|
[ -d "$HOME/.config/opencode" ] && cp -a "$HOME/.config/opencode" "$staging/home/opencode"
|
|
mem="$HOME/.claude/projects/-home-alee-Sources-6krrt/memory"
|
|
[ -d "$mem" ] && cp -a "$mem" "$staging/home/claude-memory"
|
|
|
|
cat > "$staging/RESTORE.md" <<'INNER'
|
|
# Restoring this workstation
|
|
|
|
The repo comes from Gitea; this tarball has only what git does not.
|
|
|
|
git clone ssh://git@git.adlee.work:2222/alee/6krrt.git
|
|
cd 6krrt
|
|
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
|
|
|
|
Then from this archive:
|
|
|
|
cp router.db <repo>/router.db
|
|
cp repo/.env <repo>/.env && chmod 600 <repo>/.env
|
|
cp repo/config/config.local.yaml <repo>/config/
|
|
cp -a repo/.omo <repo>/.omo
|
|
cp -a home/systemd-user/* ~/.config/systemd/user/ && systemctl --user daemon-reload
|
|
cp -a home/opencode ~/.config/opencode
|
|
|
|
Ollama models are not here. Re-pull and re-tag per docs/local-models.md:
|
|
ollama pull qwen2.5-coder:14b && ollama pull qwen3-vl:4b
|
|
(then the Modelfile num_ctx tags documented there)
|
|
|
|
Verify:
|
|
sqlite3 <repo>/router.db "select integrity_check from pragma_integrity_check limit 1;"
|
|
systemctl --user restart llm-router.service && curl -s localhost:8080/health
|
|
INNER
|
|
|
|
tar -czf "$out" -C "$staging" .
|
|
|
|
# Verify the archive reads back before rotating anything away.
|
|
tar -tzf "$out" >/dev/null || { rm -f "$out"; echo "archive FAILED to verify; discarded" >&2; exit 1; }
|
|
|
|
# shellcheck disable=SC2012
|
|
ls -1t "$DEST"/workstation-*.tar.gz 2>/dev/null | tail -n +$((KEEP + 1)) | xargs -r rm -f
|
|
|
|
echo "workstation backup ok: $out ($(du -h "$out" | cut -f1)), keeping $KEEP"
|