feat: no-progress detection watchdog — opencode stall detection, admin controls, desktop alerts #102

alee · 2026-09-26T19:08:02Z

alee commented

2026-09-26 19:08:02 +00:00

no-progress-lift-a: No-progress detection watchdog

What this adds

A local watchdog that catches opencode agent sessions spinning without progress, alerts the operator via desktop notify-send, surfaces stall data in the admin portal, and lets you block a stalled model with one click.

Components

src/progress_detect.py — Pure-stdlib detector: dup/top/slow/coverage signals
src/notifier.py — Desktop notification channel (notify-send)
src/watchdog.py — Main orchestrator with --once mode, lock file, DB writes
src/watchdog_store.py — DB table management (ticks, verdicts, alerts, channels)
admin/frontend/controls.html — Watchdog card, channels, test alert button
admin/frontend/index.html — Loops panel with open alerts
admin/frontend/models.html — Per-model stall rollup, Block/Unblock
config/schema.sql — 4 new tables (watchdog_ticks, watchdog_verdicts, watchdog_alerts, watchdog_channel_settings)
deploy/llm-router-watchdog.{service,timer} — systemd units (NOT installed)

Tests (24 commits, 2370 passing)

test_progress_detect.py — detector signals + fixture labels (15 sessions)
test_progress_backtest.py — backtest script + subprocess execution
test_config_watchdog.py — config validation
test_notifier.py — notify-send stubs, rate limiting
test_watchdog.py — orchestrator with stubbed opencode HTTP
test_admin_knob_coverage.py — knob coverage updated

Manual QA verified on 8081

Backtest: 8/15 flagged (correct)
Admin portal: all pages load, watchdog card shows data
Test alert: {"sent":1} — real notify-send fires
Block/Unblock roundtrip: POST blocks, DELETE unblocks
Timer: NOT installed or enabled (by design)

Operator steps (post-merge)

systemctl --user daemon-reload
systemctl --user start llm-router-watchdog.timer
systemctl --user enable llm-router-watchdog.timer
Verify: systemctl --user status llm-router-watchdog.timer

## no-progress-lift-a: No-progress detection watchdog ### What this adds A local watchdog that catches opencode agent sessions spinning without progress, alerts the operator via desktop notify-send, surfaces stall data in the admin portal, and lets you block a stalled model with one click. ### Components - **`src/progress_detect.py`** — Pure-stdlib detector: dup/top/slow/coverage signals - **`src/notifier.py`** — Desktop notification channel (notify-send) - **`src/watchdog.py`** — Main orchestrator with `--once` mode, lock file, DB writes - **`src/watchdog_store.py`** — DB table management (ticks, verdicts, alerts, channels) - **`admin/frontend/controls.html`** — Watchdog card, channels, test alert button - **`admin/frontend/index.html`** — Loops panel with open alerts - **`admin/frontend/models.html`** — Per-model stall rollup, Block/Unblock - **`config/schema.sql`** — 4 new tables (watchdog_ticks, watchdog_verdicts, watchdog_alerts, watchdog_channel_settings) - **`deploy/llm-router-watchdog.{service,timer}`** — systemd units (NOT installed) ### Tests (24 commits, 2370 passing) - `test_progress_detect.py` — detector signals + fixture labels (15 sessions) - `test_progress_backtest.py` — backtest script + subprocess execution - `test_config_watchdog.py` — config validation - `test_notifier.py` — notify-send stubs, rate limiting - `test_watchdog.py` — orchestrator with stubbed opencode HTTP - `test_admin_knob_coverage.py` — knob coverage updated ### Manual QA verified on 8081 - Backtest: 8/15 flagged (correct) - Admin portal: all pages load, watchdog card shows data - Test alert: `{"sent":1}` — real notify-send fires - Block/Unblock roundtrip: `POST` blocks, `DELETE` unblocks - Timer: NOT installed or enabled (by design) ### Operator steps (post-merge) 1. `systemctl --user daemon-reload` 2. `systemctl --user start llm-router-watchdog.timer` 3. `systemctl --user enable llm-router-watchdog.timer` 4. Verify: `systemctl --user status llm-router-watchdog.timer`

alee added 24 commits 2026-09-26 19:08:03 +00:00

fix: remove non-function export that broke opencode plugin loader 5fa869c3df

opencode 1.18 treats every export of a plugin module as a plugin
function.  router-link.js exported a Map (), so the
loader rejected the entire module: chat.headers and tool.execute.after
never ran.  The 29 node tests passed because they imported the module
directly, bypassing the loader.

- Move parentCache from a named export to a property on RouterLink
- Add loader-contract test: every module export must be a function
- Update test imports to use RouterLink.parentCache

Evidence: opencode.log at every start since 2026-09-26T08:02Z shows
"Plugin export is not a function" for router-link.js.

feat: export calibration fixture for progress detection a1e7db1f41

Export 15 labelled sessions (8 must-flag, 7 must-not-flag) from
opencode SQLite DB through scripts/export_progress_fixture.py into
tests/fixtures/progress/fixture.json.

- Reads opencode ~/.local/share/opencode/opencode.db
- Extracts tool calls per prototype's calls_of logic
- Applies must_flag/must_not_flag labels per the plan brief
- Content scrubbing: sha1[:8] for all args except detector targets
  (filePath, path, offset, limit, command, pattern, output_mode, include, tmux_command)
- file_lines: {path: line_count|null} map for coverage detection
- landed: boolean per prototype (edit/write with non-empty diff, or
  git commit with exit 0)
- Atlas window rule tracked per session label
- Pre-commit sensitive data scan
- Compact JSON format for size

feat: Add watchdog, notifications, and detector config sections cd4f070829

Add DetectorConfig, ChannelConfig, NotificationsConfig, and
WatchdogConfig Pydantic models to src/config.py, each inheriting
StrictModel (extra='forbid'). Append corresponding defaults to
config/config.yaml.

- DetectorConfig: heuristic thresholds for anomaly detection (window,
  dup_min, top_min, top_min_ro, cum_min, cover_min, min_calls)
- ChannelConfig: typed notification channel with Literal['desktop']
  restriction — unknown types fail at load
- NotificationsConfig: list of ChannelConfig with default desktop
  channel at warning severity
- WatchdogConfig: runtime watchdog with detector, model (defaults to
  verification.model when None), read_only_agents, and dashboard URL

Wire into RouterConfig as watchdog and notifications fields, both
with sensible defaults.

feat: add progress_detect.py — shared no-progress detector module (pure stdlib) 4a8a6252fa

Port the signal functions from plans/no-progress-detection-prototype.py into a
clean, importable module with zero external dependencies.

- DetectConfig: frozen dataclass holding all tunable thresholds
- target_of: coarse dedup key for tool calls (filePath, bash files+numbers,
  grep/glob patterns, fallback)
- window_stats: dup share, top-target count, and session-wide slow count
  over a sliding time window
- coverage: max over files of (lines read in window / file length), with
  file_lines injected as a Callable (not a disk read)
- tree_landed: bool — any window call with landed=True
- evaluate: verdict function returning (is_flagged, reason_dict) or
  (False, None) when call count < min_calls

feat: add progress_detect.py — shared no-progress detector module (pure stdlib) 33786e4e9d

Port of plans/no-progress-detection-prototype.py.
Fixes: call-count-based window (not seconds), matches prototype semantics.
Landing check is time-based across ALL calls within window's time span.
14/15 calibration labels verified against fixture.

test: config validation tests for new watchdog config sections e22874904e

feat: add desktop notifier with PagerDuty-events-v2 shape 44117e8e02

feat: add watchdog tables cfb1c92b1b

test: unit tests for progress detector and fixture labels 1f0a78c1d8

test: unit tests for desktop notifier 99849d16a6

feat: watchdog orchestrator — opencode loop detection pipeline 76e14cf3e2

src/watchdog.py: main orchestrator (~620 LOC) that:
- Reads rc-servers.json (dual-format support)
- Probes opencode servers, fetches sessions & tool messages
- Resolves parent→root chains and merges calls per root
- Runs progress_detect evaluation with file-line heuristic
- Calls local LLM for second opinion (max 3, yes/no timeout)
- Attributed routed model/provider/cost from route_decisions
- Manages alert state machine (trigger/resolve/upsert) in SQLite
- Writes watchdog_ticks, watchdog_verdicts, watchdog_alerts
- Runs as fresh process per tick; fcntl lock for parallel safety
- Exits 0 on clean/no-opencode, alert count on triggered

src/progress_detect.py: include 'coverage' in evaluation reason dict
so watchdog can persist it in the verdicts table.

test: unit tests for watchdog orchestrator 80a0099092

tests/test_watchdog.py: 32 tests covering:
- resolve_roots: 3-depth chains, multi-root, empty, orphans, sort
- _read_rc_servers: nested+flat formats, invalid JSON, dedup
- _make_key: format, empty
- _fire_alert: trigger seed/update, resolve, reopen resolved, ensure_idempotent
- detect_config_from_pydantic: field mapping, None handling
- calls_of: tool extraction, landed heuristics, error handling
- tick state machine: no_opencode outcome, resolve unflagged alert

feat: no-progress-lift-a — full implementation (watchdog rework, admin, tests) e796bc76cf

fix: ruff style compliance on new files 1a96840003

fix: descendant-landed scope and gating (R1 + R2) cd66bd0a4b

fix: noqa BLE001 + sqlite3.Row to ChannelConfig in test-alert (F2+F3) 9beeea0837

feat: bash target normalization — strip # comments, cd/env prefixes (R3) e61c73a372

fix: ruff compliance — ms timestamp, noqa cleanup, SIM117 (F2+F3) c7c5b79e96

fix: canonicalize args for dup/slow, pass RO keywords to backtest 857b217587

fix: lint gate clean on new and existing files (R4-3) be7a42a144

fix: commit noqa cleanup, remove uv.lock (R4-4) 6d8f1a99fb

fix: noqa S110/S112 on new file except blocks (F2) a44e92fd24

fix: remove KNOWN_DEVIATION_ID, narrow except blocks (R4) f511fe7454

fix: use sys.executable for backtest subprocess instead of worktree-path-based VENV_PYTHON 4b3e7d5908

alee added 1 commit 2026-09-26 19:15:21 +00:00

fix(watchdog): compare the last tick in milliseconds, like opencode's call times 8c76072eb2

last_tick is epoch seconds while opencode's call start times are epoch
milliseconds, so every call ever made looked "since the last tick". Each
tick re-judged the whole session history: a dry run against live opencode
triggered three alerts (two critical) for sessions idle since the night
before, which could never resolve because their history never changes.

The tests hid it by stubbing call times in seconds. They now use
milliseconds, and the idle-session test carries a 60-call loop that would
flag if judged, plus the tables a full tick reads, so a swallowed crash can
no longer pass as "no alert". It fails without this fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N9biTbFC63yDfYfUsZmhgd

alee merged commit a24de1bd34 into main

2026-09-26 19:17:36 +00:00

alee referenced this issue from a commit

2026-09-26 19:17:38 +00:00

Merge pull request 'feat: no-progress detection watchdog — opencode stall detection, admin controls, desktop alerts' (#102) from feat/no-progress-lift-a into main

alee referenced this pull request

2026-09-27 01:48:07 +00:00

docs: Lift A docs pass, plus watchdog exits 0 after a tick #103

alee referenced this issue from a commit

2026-09-27 02:33:43 +00:00

plans: no-progress detection spec, Lift A brief, prototype, deploy separation, cockpit

Sign in to join this conversation.