vigilar/docs/superpowers/specs/2026-04-05-project-documentation-design.md

Vigilar Project Documentation — Design Spec

Date: 2026-04-05 Status: Approved, ready for implementation plan Scope: Create user-facing and architectural documentation for Vigilar, plus a polished top-level README.md.

---

1. Goal

Vigilar currently has no top-level README.md, no user guide, and no architectural reference. Contributors and would-be home users have to read source code to understand what the project is or how to run it. This effort closes that gap with a single coordinated documentation pass.

The docs must:

• Give a homeowner with a mini PC a clear linear path from bare hardware to working cameras on their phone.
• Give a self-hoster a reference for config, CLI, secrets, backups, upgrades, and troubleshooting.
• Give a contributor enough architectural context to navigate the codebase without reading every file.
• Match the project's ethos: plain, no bloat, no cloud, no extra tooling.

2. Non-goals

• No doc site / MkDocs build (files are organized as if MkDocs-ready, but no tooling is added).
• No rewrites of existing code or config.
• No changes to docs/camera-hardware-guide.md (already exists, untouched).
• No changes to anything under docs/superpowers/ other than this spec.
• No CI doc-linting, no link-checker automation beyond a one-time manual verification pass.
• No recording-backup-to-NAS feature work. Docs describe only what exists today; future backup improvements are noted as planned, not documented as present.

3. Audience & doc layout (Approach 3 — Hybrid)

Three organizing principles, one tree:

• User-facing guides are monolithic linear narratives (humans read top to bottom once).
• Architecture docs are split reference material (contributors jump to the subsystem they're touching).
• README is the front door that ties everything together.

Final tree:

README.md                              # NEW — top-level front door
docs/
├── home-user-guide.md                 # NEW — monolithic, linear
├── operator-guide.md                  # NEW — monolithic, reference
├── architecture/
│   ├── overview.md                    # NEW — process model, bus, data flow
│   ├── conventions.md                 # NEW — coding conventions distilled
│   └── subsystems/
│       ├── camera.md                  # NEW
│       ├── detection.md               # NEW
│       ├── events.md                  # NEW
│       ├── alerts.md                  # NEW
│       ├── sensors.md                 # NEW
│       ├── ups.md                     # NEW
│       ├── storage.md                 # NEW
│       ├── highlights.md              # NEW
│       ├── presence.md                # NEW
│       ├── pets.md                    # NEW
│       ├── health.md                  # NEW
│       └── web.md                     # NEW
└── camera-hardware-guide.md           # EXISTING — untouched

4. README.md (top-level)

Sells the project, orients newcomers, links into the doc tree.

Required sections, in order:

1. Title + tagline — "Vigilar — DIY offline-first home security"
2. One-line pitch (~20 words)
3. Hero image placeholder — ![screenshot](docs/images/grid.png) with a comment noting it's a placeholder. Do not create the image.
4. Why Vigilar — bullet list of differentiators: offline-first, HLS grid + MJPEG single, OpenCV MOG2 motion with 5-sec pre-motion ring buffer, event timeline + highlight reels + timelapses, visitor/pet/wildlife tracking, PWA with VAPID push (no Firebase), AES-256 encrypted recordings, NUT UPS monitoring, runs on cheap mini PC.
5. Quick paths table — 4 rows mapping intent → doc: home user guide, operator guide, architecture overview, camera hardware guide.
6. 60-second overview — small ASCII diagram (cameras → mini PC → phone/browser) + 4 tech-stack bullets (Python 3.11, Flask+Bootstrap 5, SQLite WAL, MQTT, FFmpeg).
7. Status — "Alpha. Works on the author's hardware. Expect rough edges and breaking changes."
8. Installation TL;DR — 3-line fenced block (git clone, sudo ./scripts/install.sh, sudo systemctl start vigilar), followed by links to the full home-user and operator guides.
9. Documentation — nested bullet list mirroring the docs/ tree, each entry with a one-line description.
10. License — GPL-3.0. If no LICENSE file exists, the README states the intent and notes a LICENSE file will follow; we do not create the license file as part of this effort unless asked.
11. Contributing — short stub: "Issues and PRs welcome. See CLAUDE.md for code conventions."

5. docs/home-user-guide.md

Monolithic, linear, ~1500–2500 words. Target reader: homeowner with a mini PC, some Linux comfort, wants cameras on phone.

Required sections, in order:

1. What you'll end up with — outcome in 2 sentences plus a small ASCII diagram.
2. What you need — hardware checklist: mini PC (x86_64, 4GB+ RAM, 128GB+ SSD), USB stick for OS install, one or more RTSP cameras, phone, optional NAS. Link to camera-hardware-guide.md for camera picks.
3. Step 1 — Install Debian/Ubuntu Server on the mini PC — brief, points at upstream installer docs, tells the user to enable SSH. No hand-holding on the OS install itself.
4. Step 2 — Get Vigilar onto the box — git clone, sudo ./scripts/install.sh, plus 3 bullets summarizing what install.sh does (read scripts/install.sh at write-time to ground these bullets).
5. Step 3 — First boot — sudo systemctl enable --now vigilar, then open http://<mini-pc-ip>:49735 in a browser on the same LAN. Mention the port is configurable under [web] in vigilar.toml.
6. Step 4 — Set your PIN — UI walkthrough, 2–3 sentences, screenshot placeholder.
7. Step 5 — Add your first camera — UI walkthrough: RTSP URL, credentials, test stream, save. Point at camera-hardware-guide.md for URL formats.
8. Step 6 — Phone push notifications (PWA) — open web UI on phone, "Add to Home Screen", allow notifications. Under-the-hood note: VAPID keys already generated by install.sh.
9. Step 7 — Optional: NAS backup of config + database — mount NAS share at /mnt/nas/vigilar-backups, set VIGILAR_BACKUP_DIR, and set up a nightly run of scripts/backup.sh. The guide provides a copy-pasteable systemd service + timer snippet inline (no new units are shipped in the repo as part of this effort). Explicitly state that this backs up DB + /etc/vigilar (config + secrets) only, and that recordings stay local — point at a "planned" note for recording backup.
10. Troubleshooting — camera won't connect, no push notifications, service won't start (journalctl -u vigilar), motion detection too sensitive, how to reset PIN.
11. Where to go next — links to Operator Guide and Architecture Overview.

Grounding rule: every shell command must correspond to a real file in scripts/ or a real vigilar CLI subcommand. Verify before writing.

6. docs/operator-guide.md

Monolithic, reference-oriented, ~2500–4000 words. Target reader: self-hoster tuning, upgrading, securing.

Required sections, in order:

1. Audience & scope — for admins, not first-time home users. Points at home-user-guide.md for initial setup.
2. Layout on disk — table of /opt/vigilar, /etc/vigilar/{vigilar.toml, certs/, secrets/}, /var/vigilar/{data/vigilar.db, recordings/, hls/, backups/}.
3. Installation — what scripts/install.sh does, systemd/vigilar.service summary, systemd/vigilar-mosquitto.conf summary, system dependencies (ffmpeg, mosquitto, sqlite3, Python 3.11+).
4. Configuration reference (vigilar.toml) — one subsection per [section] in the default TOML. Each key: default, what it controls, when to change. Sections to cover (from current config/vigilar.toml): system, mqtt, web, zigbee2mqtt, ups, storage, remote, alerts.local, alerts.web_push, alerts.email, plus any additional sections discovered by re-reading the TOML at write-time.
5. CLI reference (vigilar ...) — enumerated at write-time by reading vigilar/cli/. One subsection per top-level command. Do not guess commands.
6. Secrets & security — /etc/vigilar/secrets/ layout and permissions; vigilar config set-pin; vigilar config set-password; TLS via scripts/gen_cert.sh → [web] tls_cert/tls_key; VAPID via scripts/gen_vapid_keys.sh; storage encryption key (storage.key) — explicit warning: do not lose it, recordings are unrecoverable without it; recommended firewall stance (LAN-only by default).
7. UPS / NUT integration — scripts/setup_nut.sh, [ups] options, shutdown behavior, low-battery thresholds.
8. Backups — what scripts/backup.sh captures (DB + /etc/vigilar) and what it does not (recordings); VIGILAR_BACKUP_DIR and VIGILAR_BACKUP_RETENTION_DAYS; copy-pasteable systemd service + timer snippet (inline in the doc; no new unit files added to the repo); restore procedure.
9. Upgrades — git pull + pip install -e . + systemctl restart vigilar; rollback by restoring a backup tarball. If DB migrations exist, note how they're applied; if they don't, say so.
10. Logs & health — journalctl -u vigilar, log_level in [system], health endpoints (enumerated at write-time by reading vigilar/web/blueprints/system.py and vigilar/health/).
11. Remote access — [remote] section, tunnel-based remote HLS, bandwidth-shaped downscaled streams, reiterated not-a-cloud.
12. Troubleshooting — service crash loops, MQTT broker won't start, camera worker thrashing, disk full / free_space_floor_gb triggered, HLS stalling.

Grounding rule: every TOML key, every CLI command, every file path, every endpoint must be verified against the current code before writing. Any that can't be verified must be omitted, not guessed.

7. docs/architecture/overview.md

~1000–1500 words. Target reader: contributor new to the codebase.

Required sections, in order:

1. Design principles — offline-first, subsystem isolation via multiprocessing, loose coupling via local MQTT bus, SQLite WAL as single durable store, SQLAlchemy Core (not ORM), adaptive FPS (2 idle / 30 on motion) with ring buffer.
2. Process topology — ASCII or Mermaid diagram showing parent supervisor + N subsystem processes + mosquitto + Flask web.
3. The MQTT bus — broker location, topic naming convention vigilar/<subsystem>/<entity>/<event>, retention/QoS notes (verify at write-time), rationale for MQTT over an in-process queue.
4. Data flow: the motion → alert path — numbered sequence from RTSP capture through motion detection, recording, event creation, highlight scoring, push notification, and UI update. Each step names the actual file/function where it happens (verify at write-time).
5. Storage layout — SQLite table summary (enumerate at write-time by reading vigilar/storage/), recordings (.vge, AES-256-GCM, key path), HLS segments, backups.
6. Configuration & secrets — TOML → Pydantic v2 validation, secrets as file paths (never inline), PIN & password hashing with constant-time compare.
7. The web tier — Flask + Blueprints, Jinja2 + Bootstrap 5 dark, HLS grid + MJPEG single view rationale, PWA + VAPID.
8. What's NOT in the critical path — remote access (optional), email alerts (optional), cloud (never).

8. docs/architecture/conventions.md

~400 words. Distilled from CLAUDE.md but written for human contributors, not the AI. Covers: StrEnum for string constants (vigilar/constants.py), SQLAlchemy Core only (no mapped ORM classes), type hints on public functions, no docstrings unless logic is non-obvious, Ruff line-length 100, multiprocessing-per-subsystem rule, MQTT topic naming, Pydantic-validated TOML config, secrets-as-file-paths.

9. docs/architecture/subsystems/*.md (12 files)

One file per subdirectory under vigilar/: camera, detection, events, alerts, sensors, ups, storage, highlights, presence, pets, health, web.

Uniform template (≈150–400 words each):

# <Subsystem name>

## Purpose
One paragraph — what this subsystem is responsible for.

## Key files
- `vigilar/<sub>/foo.py` — role
- ...

## MQTT topics
**Subscribes:** `vigilar/...`
**Publishes:** `vigilar/...`

## Database tables
`table_name` — what it holds. Or "none."

## Depends on
- sister subsystem X (via topic Y)

## Consumed by
- sister subsystem Z (via topic W)

## Notes
Gotchas or perf notes, only if any.

Grounding rule (hard): every topic name, every table name, every file role must come from reading the actual code. If a topic cannot be found, the doc must say "no MQTT publishers found at time of writing" — not invent one. This rule is the most important verification step in the plan.

10. Verification checklist (before completion)

1. Link check — every relative link in every new file resolves to a real path.
2. Command check — every shell command in the user guides exists as a real script under scripts/ or a real vigilar CLI subcommand.
3. Grounding check — every topic name, table name, file path, and endpoint is verified against code, or omitted. Nothing guessed.
4. TOML coverage check — every [section] in config/vigilar.toml is covered in the operator guide's configuration reference.
5. Subsystem coverage check — every subdirectory in vigilar/ (matching the 12-file list) has a corresponding subsystem doc.
6. Read-through pass — tone and terminology consistent across all files.
7. README link check — all doc tree links in README.md resolve.

11. Out of scope (explicit)

• LICENSE file creation (the README declares GPL-3.0; creating the file is a separate request).
• Screenshot/image creation (placeholders only).
• MkDocs configuration.
• Any code changes.
• Any changes to docs/camera-hardware-guide.md.
• Any doc-linting CI.
• Recording-backup-to-NAS feature or docs beyond the "planned" note.
• Migration documentation beyond noting whether migrations exist.

12. Success criteria

• A new homeowner can go from a bare mini PC to working cameras on their phone using only README.md + docs/home-user-guide.md.
• A self-hoster can answer any "how do I configure / back up / upgrade / troubleshoot" question from docs/operator-guide.md alone.
• A new contributor can identify which subsystem owns a given behavior within 5 minutes using docs/architecture/overview.md + the subsystem files.
• Every claim in every doc is either verified against current code or explicitly flagged.