feat: no-progress detection watchdog — opencode stall detection, admin controls, desktop alerts #102

Merged
alee merged 25 commits from feat/no-progress-lift-a into main 2026-09-26 19:17:36 +00:00

25 Commits

Author SHA1 Message Date
adlee-was-taken
8c76072eb2 fix(watchdog): compare the last tick in milliseconds, like opencode's call times
last_tick is epoch seconds while opencode's call start times are epoch
milliseconds, so every call ever made looked "since the last tick". Each
tick re-judged the whole session history: a dry run against live opencode
triggered three alerts (two critical) for sessions idle since the night
before, which could never resolve because their history never changes.

The tests hid it by stubbing call times in seconds. They now use
milliseconds, and the idle-session test carries a 60-call loop that would
flag if judged, plus the tables a full tick reads, so a swallowed crash can
no longer pass as "no alert". It fails without this fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N9biTbFC63yDfYfUsZmhgd
2026-09-26 15:15:18 -04:00
adlee-was-taken
4b3e7d5908 fix: use sys.executable for backtest subprocess instead of worktree-path-based VENV_PYTHON 2026-09-26 14:58:01 -04:00
adlee-was-taken
f511fe7454 fix: remove KNOWN_DEVIATION_ID, narrow except blocks (R4) 2026-09-26 14:57:08 -04:00
adlee-was-taken
a44e92fd24 fix: noqa S110/S112 on new file except blocks (F2) 2026-09-26 14:54:49 -04:00
adlee-was-taken
6d8f1a99fb fix: commit noqa cleanup, remove uv.lock (R4-4) 2026-09-26 14:47:07 -04:00
adlee-was-taken
be7a42a144 fix: lint gate clean on new and existing files (R4-3) 2026-09-26 14:46:50 -04:00
adlee-was-taken
857b217587 fix: canonicalize args for dup/slow, pass RO keywords to backtest 2026-09-26 14:46:29 -04:00
adlee-was-taken
c7c5b79e96 fix: ruff compliance — ms timestamp, noqa cleanup, SIM117 (F2+F3) 2026-09-26 14:17:23 -04:00
adlee-was-taken
e61c73a372 feat: bash target normalization — strip # comments, cd/env prefixes (R3) 2026-09-26 14:04:06 -04:00
adlee-was-taken
9beeea0837 fix: noqa BLE001 + sqlite3.Row to ChannelConfig in test-alert (F2+F3) 2026-09-26 13:46:26 -04:00
adlee-was-taken
cd66bd0a4b fix: descendant-landed scope and gating (R1 + R2) 2026-09-26 13:31:26 -04:00
adlee-was-taken
1a96840003 fix: ruff style compliance on new files 2026-09-26 13:15:25 -04:00
adlee-was-taken
e796bc76cf feat: no-progress-lift-a — full implementation (watchdog rework, admin, tests) 2026-09-26 13:09:27 -04:00
adlee-was-taken
80a0099092 test: unit tests for watchdog orchestrator
tests/test_watchdog.py: 32 tests covering:
- resolve_roots: 3-depth chains, multi-root, empty, orphans, sort
- _read_rc_servers: nested+flat formats, invalid JSON, dedup
- _make_key: format, empty
- _fire_alert: trigger seed/update, resolve, reopen resolved, ensure_idempotent
- detect_config_from_pydantic: field mapping, None handling
- calls_of: tool extraction, landed heuristics, error handling
- tick state machine: no_opencode outcome, resolve unflagged alert
2026-09-26 12:16:19 -04:00
adlee-was-taken
76e14cf3e2 feat: watchdog orchestrator — opencode loop detection pipeline
src/watchdog.py: main orchestrator (~620 LOC) that:
- Reads rc-servers.json (dual-format support)
- Probes opencode servers, fetches sessions & tool messages
- Resolves parent→root chains and merges calls per root
- Runs progress_detect evaluation with file-line heuristic
- Calls local LLM for second opinion (max 3, yes/no timeout)
- Attributed routed model/provider/cost from route_decisions
- Manages alert state machine (trigger/resolve/upsert) in SQLite
- Writes watchdog_ticks, watchdog_verdicts, watchdog_alerts
- Runs as fresh process per tick; fcntl lock for parallel safety
- Exits 0 on clean/no-opencode, alert count on triggered

src/progress_detect.py: include 'coverage' in evaluation reason dict
so watchdog can persist it in the verdicts table.
2026-09-26 12:08:25 -04:00
adlee-was-taken
99849d16a6 test: unit tests for desktop notifier 2026-09-26 11:51:30 -04:00
adlee-was-taken
1f0a78c1d8 test: unit tests for progress detector and fixture labels 2026-09-26 11:45:32 -04:00
adlee-was-taken
cfb1c92b1b feat: add watchdog tables 2026-09-26 11:45:18 -04:00
adlee-was-taken
44117e8e02 feat: add desktop notifier with PagerDuty-events-v2 shape 2026-09-26 11:42:41 -04:00
adlee-was-taken
e22874904e test: config validation tests for new watchdog config sections 2026-09-26 11:42:05 -04:00
adlee-was-taken
33786e4e9d feat: add progress_detect.py — shared no-progress detector module (pure stdlib)
Port of plans/no-progress-detection-prototype.py.
Fixes: call-count-based window (not seconds), matches prototype semantics.
Landing check is time-based across ALL calls within window's time span.
14/15 calibration labels verified against fixture.
2026-09-26 11:39:29 -04:00
adlee-was-taken
4a8a6252fa feat: add progress_detect.py — shared no-progress detector module (pure stdlib)
Port the signal functions from plans/no-progress-detection-prototype.py into a
clean, importable module with zero external dependencies.

- DetectConfig: frozen dataclass holding all tunable thresholds
- target_of: coarse dedup key for tool calls (filePath, bash files+numbers,
  grep/glob patterns, fallback)
- window_stats: dup share, top-target count, and session-wide slow count
  over a sliding time window
- coverage: max over files of (lines read in window / file length), with
  file_lines injected as a Callable (not a disk read)
- tree_landed: bool — any window call with landed=True
- evaluate: verdict function returning (is_flagged, reason_dict) or
  (False, None) when call count < min_calls
2026-09-26 11:33:36 -04:00
adlee-was-taken
cd4f070829 feat: Add watchdog, notifications, and detector config sections
Add DetectorConfig, ChannelConfig, NotificationsConfig, and
WatchdogConfig Pydantic models to src/config.py, each inheriting
StrictModel (extra='forbid'). Append corresponding defaults to
config/config.yaml.

- DetectorConfig: heuristic thresholds for anomaly detection (window,
  dup_min, top_min, top_min_ro, cum_min, cover_min, min_calls)
- ChannelConfig: typed notification channel with Literal['desktop']
  restriction — unknown types fail at load
- NotificationsConfig: list of ChannelConfig with default desktop
  channel at warning severity
- WatchdogConfig: runtime watchdog with detector, model (defaults to
  verification.model when None), read_only_agents, and dashboard URL

Wire into RouterConfig as watchdog and notifications fields, both
with sensible defaults.
2026-09-26 11:31:29 -04:00
adlee-was-taken
a1e7db1f41 feat: export calibration fixture for progress detection
Export 15 labelled sessions (8 must-flag, 7 must-not-flag) from
opencode SQLite DB through scripts/export_progress_fixture.py into
tests/fixtures/progress/fixture.json.

- Reads opencode ~/.local/share/opencode/opencode.db
- Extracts tool calls per prototype's calls_of logic
- Applies must_flag/must_not_flag labels per the plan brief
- Content scrubbing: sha1[:8] for all args except detector targets
  (filePath, path, offset, limit, command, pattern, output_mode, include, tmux_command)
- file_lines: {path: line_count|null} map for coverage detection
- landed: boolean per prototype (edit/write with non-empty diff, or
  git commit with exit 0)
- Atlas window rule tracked per session label
- Pre-commit sensitive data scan
- Compact JSON format for size
2026-09-26 11:28:45 -04:00
adlee-was-taken
5fa869c3df fix: remove non-function export that broke opencode plugin loader
opencode 1.18 treats every export of a plugin module as a plugin
function.  router-link.js exported a Map (), so the
loader rejected the entire module: chat.headers and tool.execute.after
never ran.  The 29 node tests passed because they imported the module
directly, bypassing the loader.

- Move parentCache from a named export to a property on RouterLink
- Add loader-contract test: every module export must be a function
- Update test imports to use RouterLink.parentCache

Evidence: opencode.log at every start since 2026-09-26T08:02Z shows
"Plugin export is not a function" for router-link.js.
2026-09-26 11:26:40 -04:00