d5194f43ba
Replaces the Task 31 stub README with complete documentation: quickstart, first-time setup (invite flagging, seeding, smoke), usage examples for all three watch modes, CLI flag reference, env var table, scenario descriptions, error handling summary, test account filtering explanation, and architecture overview. Adds CHECKLIST.md with post-deploy verification, bring-up, scenario, watch mode, failure handling, and staging gate items. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
65 lines
2.8 KiB
Markdown
65 lines
2.8 KiB
Markdown
# Soak Harness Validation Checklist
|
|
|
|
Run after significant changes or before calling the harness implementation complete.
|
|
|
|
## Post-deploy schema verification
|
|
|
|
Run after the server-side changes deploy to each environment.
|
|
|
|
- [ ] Server restarted (docker compose up -d or CI/CD deploy)
|
|
- [ ] Server logs show `User store schema initialized` after restart
|
|
- [ ] `\d users_v2` shows `is_test_account` column with default `false`
|
|
- [ ] `\d invite_codes` shows `marks_as_test` column with default `false`
|
|
- [ ] `\d leaderboard_overall` shows `is_test_account` column
|
|
- [ ] `\di idx_users_test_account` shows the partial index
|
|
- [ ] Leaderboard query still works: `curl .../api/stats/leaderboard` returns entries
|
|
- [ ] `?include_test=true` parameter is accepted (no 422/500)
|
|
|
|
## Bring-up
|
|
|
|
- [ ] Invite code flagged with `marks_as_test=TRUE` on target environment
|
|
- [ ] `bun run seed` creates/updates accounts in `.env.stresstest`
|
|
- [ ] All seeded users show `is_test_account=TRUE` in the DB
|
|
|
|
## Smoke test
|
|
|
|
- [ ] `bash scripts/smoke.sh` exits 0 within 60s
|
|
|
|
## Scenarios
|
|
|
|
- [ ] `--scenario=populate --rooms=1 --games-per-room=1` completes cleanly
|
|
- [ ] `--scenario=populate --rooms=2 --games-per-room=2` runs multiple rooms and multiple games
|
|
- [ ] `--scenario=stress --games-per-room=3` logs `chaos_injected` events and completes
|
|
|
|
## Watch modes
|
|
|
|
- [ ] `--watch=none` produces JSONL on stdout, nothing else
|
|
- [ ] `--watch=dashboard` opens http://localhost:7777, grid renders, WS shows `healthy`
|
|
- [ ] Clicking a player tile opens the video modal with live JPEG frames
|
|
- [ ] Closing the modal (Esc or Close) stops the screencast (check logs for `screencast_stopped`)
|
|
- [ ] `--watch=tiled` opens native Chromium windows sized to show the full game table
|
|
|
|
## Failure handling
|
|
|
|
- [ ] Ctrl-C during a run → graceful shutdown, summary printed, exit code 2
|
|
- [ ] Double Ctrl-C → immediate hard exit (130)
|
|
- [ ] Health probes detect server down (3 consecutive failures → fatal abort)
|
|
- [ ] Artifacts directory contains screenshots + state JSON on failure
|
|
- [ ] Artifacts older than 7 days are pruned on next startup
|
|
|
|
## Server-side filtering
|
|
|
|
- [ ] `GET /api/stats/leaderboard` (default) hides soak accounts
|
|
- [ ] `GET /api/stats/leaderboard?include_test=true` shows soak accounts
|
|
- [ ] Admin panel user list shows `[Test]` badge on soak accounts
|
|
- [ ] Admin panel invite codes tab shows `[Test-seed]` badge
|
|
- [ ] "Include test accounts" checkbox toggles visibility in admin
|
|
|
|
## Staging bring-up
|
|
|
|
- [ ] `5VC2MCCN` flagged with `marks_as_test=TRUE` on staging DB
|
|
- [ ] 16 accounts seeded via `SOAK_INVITE_CODE=5VC2MCCN bun run seed`
|
|
- [ ] Populate run against staging completes with `--watch=dashboard`
|
|
- [ ] Staging leaderboard default does NOT show soak accounts
|
|
- [ ] Staging leaderboard with `?include_test=true` does show them
|