alee/rutster
0
0
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity

slice-5: scalability seams — addressing, admission, drain, events (review B1/M1-M7) #14

Merged
alee merged 13 commits from slice-5/scalability-seams into main 2026-07-05 04:35:43 +00:00
Conversation 0 Commits 13 Files Changed 20 +3223 -116
alee commented 2026-07-05 01:09:53 +00:00
Owner
Copy Link

What lands

  • MediaAddressConfig (review B1): bind/advertised address split + UDP port range; ICE candidate now carries the advertised IP; rtc_session.rs seam re-pinned (f47d63b9) in-commit, loop_driver.rs frozen.
  • Admission cap + tick-lag gauge + MediaCmd::Stats (M2): a node can now say "full" (503), tick overruns are measured (compensated sleep replaces the fixed one), and the ADR-0010 benchmark gets its primary readout.
  • Drain lifecycle (M1): SIGTERM → stop admitting + keep serving HTTP while calls bleed out → deadline → hard stop. RUTSTER_DRAIN_DEADLINE_SECS=0 default preserves dev-loop instant shutdown.
  • /healthz + /readyz: readiness round-trips Stats through the command channel (250ms bound covers send AND reply) — a wedged media thread flips readyz while healthz stays up.
  • Non-blocking tap teardown (M7): Delete no longer block_on's brain I/O inside the tick loop (was up to 750ms freeze for every live call).
  • EventSink + wall-clock started_at (M3): CDR-shaped lifecycle events (Registered/Connected/Ended + reason + tap metrics) through a trait the ADR-0005 Valkey publisher will implement; log-backed today.
  • ADR-0009 amendment (M5): in-process enforcement, shared accounting — pending your ratification; plus ADR-0005 cross-note and ARCHITECTURE.md gains "call placement" in the horizontal list.
  • Tap protocol v2 reservations (M4): resume token / resume-ack / terminal-bye documented in protocol.rs module docs.
  • Companion docs: the 2026-07-04 scalability review + the slice-5 plan (with post-execution deviation note on Task 5).

With no env vars set, single-node dev behavior is unchanged except the plan-mandated 64-session default admission cap (unreachable in the dev loop). New knobs: RUTSTER_HTTP_BIND, RUTSTER_MEDIA_BIND_IP / _ADVERTISED_IP / _PORT_RANGE, RUTSTER_MAX_SESSIONS, RUTSTER_DRAIN_DEADLINE_SECS.

Merge instructions

  • Default squash. 138 tests green, fmt/clippy clean, both seam hashes verified against ci.yml.
  • Note: local cargo deny check fails on deny.toml deserialization with cargo-deny 0.18.3 on main too (pre-existing tool-version mismatch, no dep changes in this PR) — CI's pinned action is the arbiter.

Reviews

  • docs/reviews/2026-07-04-scalability-infra-review.md (the motivating review; B1/M1–M7 map to commits)
  • Executed subagent-driven with per-task adversarial review + final whole-branch review (verdict: ready to merge). Punted minors logged in the PR branch's .superpowers/sdd/progress.md ledger; notable: port-range exhaustion currently maps to 500 not 503 (revisit at step-5 trunk work).
  • ADR-0009 amendment requires maintainer ratification — it's on this branch; merging = accepting, or strip the commit if you want to sit with it.

🤖 Generated with Claude Code

https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8

## What lands - **MediaAddressConfig** (review B1): bind/advertised address split + UDP port range; ICE candidate now carries the advertised IP; rtc_session.rs seam re-pinned (f47d63b9) in-commit, loop_driver.rs frozen. - **Admission cap + tick-lag gauge + MediaCmd::Stats** (M2): a node can now say "full" (503), tick overruns are measured (compensated sleep replaces the fixed one), and the ADR-0010 benchmark gets its primary readout. - **Drain lifecycle** (M1): SIGTERM → stop admitting + keep serving HTTP while calls bleed out → deadline → hard stop. RUTSTER_DRAIN_DEADLINE_SECS=0 default preserves dev-loop instant shutdown. - **/healthz + /readyz**: readiness round-trips Stats through the command channel (250ms bound covers send AND reply) — a wedged media thread flips readyz while healthz stays up. - **Non-blocking tap teardown** (M7): Delete no longer block_on's brain I/O inside the tick loop (was up to 750ms freeze for every live call). - **EventSink + wall-clock started_at** (M3): CDR-shaped lifecycle events (Registered/Connected/Ended + reason + tap metrics) through a trait the ADR-0005 Valkey publisher will implement; log-backed today. - **ADR-0009 amendment** (M5): in-process enforcement, **shared accounting** — pending your ratification; plus ADR-0005 cross-note and ARCHITECTURE.md gains "call placement" in the horizontal list. - **Tap protocol v2 reservations** (M4): resume token / resume-ack / terminal-bye documented in protocol.rs module docs. - Companion docs: the 2026-07-04 scalability review + the slice-5 plan (with post-execution deviation note on Task 5). With **no env vars set**, single-node dev behavior is unchanged except the plan-mandated 64-session default admission cap (unreachable in the dev loop). New knobs: RUTSTER_HTTP_BIND, RUTSTER_MEDIA_BIND_IP / _ADVERTISED_IP / _PORT_RANGE, RUTSTER_MAX_SESSIONS, RUTSTER_DRAIN_DEADLINE_SECS. ## Merge instructions - Default squash. 138 tests green, fmt/clippy clean, both seam hashes verified against ci.yml. - Note: local `cargo deny check` fails on deny.toml deserialization with cargo-deny 0.18.3 **on main too** (pre-existing tool-version mismatch, no dep changes in this PR) — CI's pinned action is the arbiter. ## Reviews - docs/reviews/2026-07-04-scalability-infra-review.md (the motivating review; B1/M1–M7 map to commits) - Executed subagent-driven with per-task adversarial review + final whole-branch review (verdict: ready to merge). Punted minors logged in the PR branch's .superpowers/sdd/progress.md ledger; notable: port-range exhaustion currently maps to 500 not 503 (revisit at step-5 trunk work). - **ADR-0009 amendment requires maintainer ratification** — it's on this branch; merging = accepting, or strip the commit if you want to sit with it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
alee added 13 commits 2026-07-05 01:09:55 +00:00
docs: slice-5 scalability review + implementation plan 5661111a44 
Multi-agent scalability & infra-fit review (docs/reviews/) and the
slice-5 seams plan it produced (docs/superpowers/plans/). Slice 5
plants the horizontal-scaling seams — MediaAddressConfig (B1),
admission/Stats (M2), drain (M1), EventSink (M3), non-blocking
teardown (M7), ADR-0009 amendment (M5), tap v2 reservations (M4) —
before further slices calcify around their absence.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
slice-5: RUTSTER_HTTP_BIND env config via new config module 04e6fec673 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
slice-5: MediaAddressConfig — advertised addr + port range (review B1) b97c083faf 
Re-pins the rtc_session.rs seam hash (loud by design). loop_driver.rs
unchanged. Default config reproduces pre-slice-5 behavior exactly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
slice-5: sweep stale comments from Task 2 review (re-pins seam hash) b64a23ae9c 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
slice-5: admission cap, tick-lag gauge, MediaCmd::Stats (review M2) 2e5a623b17 
Register now sheds the marginal call ("node full") once sessions.len()
>= max_sessions instead of degrading every in-flight call's 20ms
budget. The fixed sleep(META_TICK) tail is replaced by a compensated
sleep that tracks tick_overruns/last_tick_micros — the ADR-0010
benchmark's primary readout and the autoscaling signal. MediaCmd::Stats
exposes {sessions, max_sessions, draining, tick_overruns,
last_tick_micros} for the future /readyz handler. routes.rs maps
"node full" and "draining" to 503 so an LB retries elsewhere.
RUTSTER_MAX_SESSIONS (default 64) wires the cap from env.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
slice-5: drain lifecycle — SIGTERM bleeds out calls (review M1) 5fd324e971 
MediaCmd::Drain flips a draining flag that rejects new Registers with
the "draining" contract string (routes already map this to 503, Task
3) while existing sessions keep ticking; the reply fires once the
session map empties, checked right after the eviction sweep so a tick
that closes the last session completes the drain in the same
iteration. MediaStats.draining now reports the real flag instead of
the Task 3 placeholder.

config::drain_deadline parses RUTSTER_DRAIN_DEADLINE_SECS, defaulting
to 0 (today's instant dev-loop shutdown). main.rs restructures the
serve/shutdown tail into two phases: on SIGTERM, send Drain while axum
keeps serving (in-flight calls still need DELETE/offer, and Task 5's
readyz will flip so the LB pulls the node), wait for drain-complete or
the deadline, THEN quiesce HTTP and call media_thread.shutdown() for
the hard stop. Calls are non-migratable by design, so drain-then-
terminate is the only graceful scale-in shape.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
slice-5: /healthz + /readyz — readiness reads MediaCmd::Stats a83a902ebe 
Implement task 5: liveness and readiness probes for the HTTP stack.

- GET /healthz: 200 OK always (process + HTTP stack up). Deliberately
  does NOT consult the media thread — liveness and readiness have
  different restart semantics.

- GET /readyz: 200 + MediaStats JSON iff media thread answers Stats
  within 250ms AND node is not draining AND sessions < max_sessions;
  else 503 + JSON or "media thread unresponsive" (zombie case:
  dead thread while HTTP stack still answers).

Unit test: readyz_503_when_media_thread_gone_but_healthz_200 (routes.rs)
  shows the zombie-node failure path from 2026-07-04 review.

Integration test: readyz_200_with_stats_json_when_thread_alive
  verifies 200 + valid JSON from a live media thread.

Both handlers follow the brief's signatures; no seam touches.
All existing tests remain green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
slice-5: readyz timeout covers send; fix test thread leak 5ce18bf472 
Task 5 review fixes: the 250ms readiness bound now wraps the whole
Stats round trip (a wedged thread can stall the send, not just the
reply), and the readyz integration test tears down its MediaThread
like the sibling test does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
slice-5: Delete teardown off the tick loop (review M7) a53fb8a510 
Add spawn_tap_teardown (tap_engine.rs) — hands the 750ms bounded
close_tx-then-abort sequence to a tokio task instead of running it
inline via block_on on the media thread. The old Delete arm blocked
the 20ms tick loop for EVERY live call on the node whenever one
session's brain was unresponsive; teardown is brain I/O and must
never run on the timing thread. Rewire MediaCmd::Delete to call it.

TDD: new test spawn_tap_teardown_returns_immediately_and_aborts_stuck_engine
asserts (a) the call returns in <100ms even with a permanently-stuck
engine task, and (b) the stuck task is aborted within the 750ms+slack
cap, observed via a Drop-guard channel.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
slice-5: EventSink seam + wall-clock started_at (review M3) 418f29e3c7 
Lifecycle events now flow through a trait the ADR-0005 Valkey publisher
can implement; Channel carries SystemTime so a CDR can exist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
docs(adr): ADR-0009 amendment — shared spend accounting (review M5) df1ec3360d 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
docs(tap): reserve v2 resume/terminal-bye semantics (review M4) be167b46a3 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
slice-5: drain deadline covers the send; doc fixes (final review)
All checks were successful
CI / fmt (pull_request) Successful in 1m25s
Details
CI / clippy (pull_request) Successful in 1m44s
Details
CI / test (1.85) (pull_request) Successful in 3m56s
Details
CI / test (stable) (pull_request) Successful in 4m27s
Details
CI / deny (pull_request) Successful in 1m30s
Details
48e41f2ff5 
The shutdown drain send could block past the deadline on a wedged
media thread with a full command channel — same whole-round-trip
principle as readyz's 5ce18bf. Plus session_map doc coherence and a
plan deviation note pointing snippet-copiers at the landed code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
alee merged commit bdadfd9057 into main 2026-07-05 04:35:43 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
alee
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: alee/rutster#14
Delete Branch "slice-5/scalability-seams"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?