The shutdown drain send could block past the deadline on a wedged
media thread with a full command channel — same whole-round-trip
principle as readyz's 5ce18bf. Plus session_map doc coherence and a
plan deviation note pointing snippet-copiers at the landed code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
Add spawn_tap_teardown (tap_engine.rs) — hands the 750ms bounded
close_tx-then-abort sequence to a tokio task instead of running it
inline via block_on on the media thread. The old Delete arm blocked
the 20ms tick loop for EVERY live call on the node whenever one
session's brain was unresponsive; teardown is brain I/O and must
never run on the timing thread. Rewire MediaCmd::Delete to call it.
TDD: new test spawn_tap_teardown_returns_immediately_and_aborts_stuck_engine
asserts (a) the call returns in <100ms even with a permanently-stuck
engine task, and (b) the stuck task is aborted within the 750ms+slack
cap, observed via a Drop-guard channel.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
Task 5 review fixes: the 250ms readiness bound now wraps the whole
Stats round trip (a wedged thread can stall the send, not just the
reply), and the readyz integration test tears down its MediaThread
like the sibling test does.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
Implement task 5: liveness and readiness probes for the HTTP stack.
- GET /healthz: 200 OK always (process + HTTP stack up). Deliberately
does NOT consult the media thread — liveness and readiness have
different restart semantics.
- GET /readyz: 200 + MediaStats JSON iff media thread answers Stats
within 250ms AND node is not draining AND sessions < max_sessions;
else 503 + JSON or "media thread unresponsive" (zombie case:
dead thread while HTTP stack still answers).
Unit test: readyz_503_when_media_thread_gone_but_healthz_200 (routes.rs)
shows the zombie-node failure path from 2026-07-04 review.
Integration test: readyz_200_with_stats_json_when_thread_alive
verifies 200 + valid JSON from a live media thread.
Both handlers follow the brief's signatures; no seam touches.
All existing tests remain green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
MediaCmd::Drain flips a draining flag that rejects new Registers with
the "draining" contract string (routes already map this to 503, Task
3) while existing sessions keep ticking; the reply fires once the
session map empties, checked right after the eviction sweep so a tick
that closes the last session completes the drain in the same
iteration. MediaStats.draining now reports the real flag instead of
the Task 3 placeholder.
config::drain_deadline parses RUTSTER_DRAIN_DEADLINE_SECS, defaulting
to 0 (today's instant dev-loop shutdown). main.rs restructures the
serve/shutdown tail into two phases: on SIGTERM, send Drain while axum
keeps serving (in-flight calls still need DELETE/offer, and Task 5's
readyz will flip so the LB pulls the node), wait for drain-complete or
the deadline, THEN quiesce HTTP and call media_thread.shutdown() for
the hard stop. Calls are non-migratable by design, so drain-then-
terminate is the only graceful scale-in shape.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
Register now sheds the marginal call ("node full") once sessions.len()
>= max_sessions instead of degrading every in-flight call's 20ms
budget. The fixed sleep(META_TICK) tail is replaced by a compensated
sleep that tracks tick_overruns/last_tick_micros — the ADR-0010
benchmark's primary readout and the autoscaling signal. MediaCmd::Stats
exposes {sessions, max_sessions, draining, tick_overruns,
last_tick_micros} for the future /readyz handler. routes.rs maps
"node full" and "draining" to 503 so an LB retries elsewhere.
RUTSTER_MAX_SESSIONS (default 64) wires the cap from env.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
Multi-agent scalability & infra-fit review (docs/reviews/) and the
slice-5 seams plan it produced (docs/superpowers/plans/). Slice 5
plants the horizontal-scaling seams — MediaAddressConfig (B1),
admission/Stats (M2), drain (M1), EventSink (M3), non-blocking
teardown (M7), ADR-0009 amendment (M5), tap v2 reservations (M4) —
before further slices calcify around their absence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
