Files
rutster/docs/superpowers/plans/2026-07-04-slice-5-scalability-seams.md
Aaron D. Lee 48e41f2ff5
All checks were successful
CI / fmt (pull_request) Successful in 1m25s
CI / clippy (pull_request) Successful in 1m44s
CI / test (1.85) (pull_request) Successful in 3m56s
CI / test (stable) (pull_request) Successful in 4m27s
CI / deny (pull_request) Successful in 1m30s
slice-5: drain deadline covers the send; doc fixes (final review)
The shutdown drain send could block past the deadline on a wedged
media thread with a full command channel — same whole-round-trip
principle as readyz's 5ce18bf. Plus session_map doc coherence and a
plan deviation note pointing snippet-copiers at the landed code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
2026-07-04 21:06:56 -04:00

1830 lines
71 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Slice 5 — Scalability Seams — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Plant the horizontal-scaling seams identified in the
[2026-07-04 scalability & infra-fit review](../../reviews/2026-07-04-scalability-infra-review.md)
— media addressing (B1), admission/capacity signals (M2), drain lifecycle (M1), event
sink + wall-clock CDR fields (M3), non-blocking teardown (M7), health/readiness probes,
config surface, and the ADR-0009 accounting amendment — **before** further slices accrete
around their absence.
**Naming note:** this is *not* spearhead step 5 (rented-transport trunk). It is an
infra-seams slice that pre-paves step 4½ (ADR-0010 benchmark — Task 3's tick-lag gauge is
its primary readout) and step 5 (the trunk adapter inherits the advertised-address +
placement concepts).
**Architecture:** No new services, no Valkey yet, no fleet. Every change is a seam *inside
the existing fused vertical*: a config struct where a literal was, an enum variant on the
existing `MediaCmd` command channel, a trait where tracing lines were. Single-node behavior
with default config is byte-for-byte-equivalent-or-better; the deliverable is that N>1,
NAT, and autoscaling stop being *design changes* and become *plumbing*.
**Tech Stack:** existing workspace only — Rust stable + 1.85, `str0m` 0.21, `axum` 0.7,
`tokio`, `std::thread` media loop, `serde`. **No new dependencies.**
## Global Constraints
- **License:** GPL-3.0-or-later on every crate manifest (ADR-0004).
- **DCO:** every commit signed off — `git commit -s` (AGENTS.md Git workflow). Signoff
identity = the human maintainer's git config, not the agent.
- **Seam gate (CHANGED THIS SLICE):** `loop_driver.rs` stays **byte-identical**
(hash `744bf314edf7f4925c8bb3bd0f5176dbc88f8113`). `rtc_session.rs` is **re-pinned by
Task 2 only** — Task 2 updates `EXPECTED_RTC_SESSION` in `.github/workflows/ci.yml` in
the same commit as the code change. **No other task may touch either file.**
- **Hot-path policy:** never `?`-propagate on the 20 ms loop; match-and-continue;
"drop + observe." No `unwrap()`/`expect()` outside tests/const-init/cold-startup.
- **Learner-facing comments:** this project OVERRIDES the no-comments default. Every new
public item gets `///` docs; every new module gets `//!` docs citing the review finding
it closes (e.g. "closes 2026-07-04 review B1"). Snippets below show the load-bearing
comments; implementers keep that density.
- **Cold-path error strings are a wire contract within the binary:** routes match
`e.contains("not found")` today; Tasks 34 add `"node full"` and `"draining"` to that
contract. Don't reword one side without the other.
- **Env naming:** `RUTSTER_*`, parsed by pure functions in `crates/rutster/src/config.rs`
(testable without env mutation — take `Option<String>`/`&str` inputs).
- **CI gates:** `cargo fmt --check`, `cargo clippy --all -- -D warnings`,
`cargo test --all` (stable + 1.85), `cargo deny check`.
- **Branch/PR:** branch `slice-5/scalability-seams` off `main`; PR via `tea` (not `gh`).
## File Structure
### New files
| Path | Responsibility |
|---|---|
| `crates/rutster/src/config.rs` | Pure env-parsing helpers: `http_bind`, `parse_port_range`, `media_address_config`, `max_sessions`, `drain_deadline`. |
| `crates/rutster/src/event_sink.rs` | `EventSink` trait + `CallEvent`/`EndReason` + `TracingEventSink` (log-backed impl; Valkey impl lands with ADR-0005 wiring, later rung). |
### Modified files
| Path | What changes |
|---|---|
| `crates/rutster/src/main.rs` | Env-driven HTTP bind; builds `MediaThreadOpts` from env; two-phase shutdown (drain → deadline → hard stop). |
| `crates/rutster/src/lib.rs` | `pub mod config;` + `pub mod event_sink;`. |
| `crates/rutster-media/src/rtc_session.rs` | **Task 2 ONLY.** `MediaAddressConfig`, `new_with_config`, `bind_in_range`, `advertised_addr` field, candidate uses advertised addr. Re-pin CI hash. |
| `crates/rutster-media/src/lib.rs` | Re-export `MediaAddressConfig`. |
| `crates/rutster/src/media_thread.rs` | `MediaThreadOpts`; admission cap; tick-lag gauge + compensated sleep; `MediaCmd::{Stats, Drain}`; non-blocking Delete teardown; `EventSink` emissions. (Tasks 2, 3, 4, 6, 7 — **serial**.) |
| `crates/rutster/src/session_map.rs` | `spawn_media_thread_with(opts, …)`; existing `spawn_media_thread` delegates with defaults (test call sites unchanged). |
| `crates/rutster/src/routes.rs` | 503 mapping for `"node full"`/`"draining"`; `GET /healthz`; `GET /readyz`. |
| `crates/rutster/src/tap_engine.rs` | `spawn_tap_teardown` free function (moved out of the tick loop). |
| `crates/rutster-call-model/src/lib.rs` | `Channel.started_at: SystemTime` (wall-clock CDR anchor; `created_at: Instant` stays for idle math). |
| `crates/rutster/tests/api_integration.rs` | 503-when-full + healthz/readyz integration tests. |
| `.github/workflows/ci.yml` | Task 2 re-pins `EXPECTED_RTC_SESSION`. |
| `docs/QUICKSTART.md` | Env-var table replaces "edit main.rs to change the port". |
| `docs/adr/0009-spend-gate-honest-rescope.md` | Amendment 2026-07-04: enforcement locality vs accounting locality. |
| `docs/adr/0005-event-bus.md` | One-line cross-ref under Constraints. |
| `docs/ARCHITECTURE.md` | "call placement (session→node directory)" added to the horizontal-platform list. |
| `crates/rutster-tap/src/protocol.rs` | Module-doc "v2 reservations" section (resume token, terminal bye). Doc-only. |
### SEAM-INVARIANT files
- `crates/rutster-media/src/loop_driver.rs`**byte-identical, all tasks.**
- `crates/rutster-media/src/rtc_session.rs` — untouchable by every task except Task 2.
## Task ordering (for multi-agent dispatch)
`media_thread.rs` is the contention file — Tasks 3 → 4 → 6 → 7 are **strictly serial** on
one dev. Everything else parallelizes:
- **Task 1** — config.rs foundation + `RUTSTER_HTTP_BIND`. Land first (Task 2 extends config.rs).
- **Task 2** — `MediaAddressConfig` + seam re-pin. Depends on Task 1 (config.rs exists). Parallel with 3.
- **Task 3** — admission cap + tick-lag + `MediaCmd::Stats`. Depends on Task 2 (`MediaThreadOpts` exists — Task 2 introduces it).
- **Task 4** — drain lifecycle. Depends on Task 3 (extends `MediaStats` with `draining`).
- **Task 5** — healthz/readyz. Depends on Tasks 3+4.
- **Task 6** — non-blocking tap teardown. Depends on Task 4 (same file; serial).
- **Task 7** — `EventSink` + wall-clock. Depends on Task 6 (same file; serial).
- **Task 8** — ADR-0009 amendment + ADR-0005 note + ARCHITECTURE.md line. Parallel-safe anytime.
- **Task 9** — tap protocol v2 reservations doc. Parallel-safe anytime.
Parallelizable-now filler: QUICKSTART env table (after Task 1), LEARNING.md pointers to
`config.rs`/`event_sink.rs` (after Tasks 1/7), `cargo doc` render check (after Task 7).
---
### Task 1: `config.rs` + `RUTSTER_HTTP_BIND`
**Files:**
- Create: `crates/rutster/src/config.rs`
- Modify: `crates/rutster/src/lib.rs` (add `pub mod config;`)
- Modify: `crates/rutster/src/main.rs:44` (replace hardcoded bind)
- Modify: `docs/QUICKSTART.md` (env table; drop "edit main.rs" advice)
- Test: inline `#[cfg(test)]` in `config.rs`
**Interfaces:**
- Consumes: nothing new.
- Produces: `pub fn http_bind(raw: Option<String>) -> Result<std::net::SocketAddr, String>`
(later tasks add more parsers to this module).
- [ ] **Step 1: Write the failing tests**
Create `crates/rutster/src/config.rs`:
```rust
//! # config — pure env-parsing helpers (slice-5)
//!
//! Every knob is a pure function over `Option<String>` / `&str` so tests
//! never mutate process env (env mutation in tests races across the
//! parallel test harness — the same reason `api_key.rs` needs its
//! ENV_MUTEX). `main.rs` is the only caller that touches `std::env`.
//!
//! Closes 2026-07-04 scalability review "HTTP bind hardcoded" (minor).
use std::net::SocketAddr;
/// Resolve the HTTP bind address from `RUTSTER_HTTP_BIND`.
///
/// `None` → the historical default `0.0.0.0:8080`. Invalid input is a
/// hard error (fail-fast at startup — an operator typo must not silently
/// bind the default and hide behind an LB health check).
pub fn http_bind(raw: Option<String>) -> Result<SocketAddr, String> {
match raw {
None => Ok("0.0.0.0:8080".parse().expect("static default parses")),
Some(s) => s
.parse()
.map_err(|e| format!("RUTSTER_HTTP_BIND {s:?} is not a socket address: {e}")),
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn http_bind_defaults_when_unset() {
assert_eq!(
http_bind(None).unwrap(),
"0.0.0.0:8080".parse::<SocketAddr>().unwrap()
);
}
#[test]
fn http_bind_parses_override() {
assert_eq!(
http_bind(Some("127.0.0.1:9090".into())).unwrap(),
"127.0.0.1:9090".parse::<SocketAddr>().unwrap()
);
}
#[test]
fn http_bind_rejects_garbage_with_var_name_in_error() {
let err = http_bind(Some("not-an-addr".into())).unwrap_err();
assert!(err.contains("RUTSTER_HTTP_BIND"));
}
}
```
- [ ] **Step 2: Run tests to verify they fail (module not declared)**
Run: `cargo test -p rutster config::`
Expected: compile error — `config` not in `lib.rs` yet. Add to `crates/rutster/src/lib.rs`:
```rust
pub mod config;
```
- [ ] **Step 3: Run tests to verify they pass**
Run: `cargo test -p rutster config::`
Expected: `3 passed`
- [ ] **Step 4: Wire into `main.rs`**
Replace (main.rs:44):
```rust
let addr: SocketAddr = "0.0.0.0:8080".parse().expect("valid addr");
```
with:
```rust
// RUTSTER_HTTP_BIND: per-instance bind is the first thing any LB /
// compose template parametrizes (slice-5; matches the existing
// RUTSTER_TAP_BIND pattern in rutster-brain-realtime).
let addr: SocketAddr = rutster::config::http_bind(std::env::var("RUTSTER_HTTP_BIND").ok())
.expect("RUTSTER_HTTP_BIND must be host:port");
```
(`use std::net::SocketAddr;` already imported at main.rs:14.)
- [ ] **Step 5: Update QUICKSTART**
In `docs/QUICKSTART.md`, replace the troubleshooting row "edit `crates/rutster/src/main.rs`
to bind a different port" with:
```markdown
| Port 8080 already in use | Set `RUTSTER_HTTP_BIND`, e.g. `RUTSTER_HTTP_BIND=0.0.0.0:8090 cargo run -p rutster` |
```
- [ ] **Step 6: Full check + commit**
Run: `cargo fmt && cargo clippy --all -- -D warnings && cargo test -p rutster`
Expected: clean, all tests pass.
```bash
git add crates/rutster/src/config.rs crates/rutster/src/lib.rs crates/rutster/src/main.rs docs/QUICKSTART.md
git commit -s -m "slice-5: RUTSTER_HTTP_BIND env config via new config module"
```
---
### Task 2: `MediaAddressConfig` — bind/advertised address + port range (review B1)
**Files:**
- Modify: `crates/rutster-media/src/rtc_session.rs` (**the seam-frozen file — this task re-pins it**)
- Modify: `crates/rutster-media/src/lib.rs` (re-export)
- Modify: `crates/rutster/src/media_thread.rs` (introduce `MediaThreadOpts`; Register uses config)
- Modify: `crates/rutster/src/session_map.rs` (add `spawn_media_thread_with`)
- Modify: `crates/rutster/src/config.rs` (parsers), `crates/rutster/src/main.rs` (env wiring)
- Modify: `.github/workflows/ci.yml:54` (`EXPECTED_RTC_SESSION` re-pin, same commit)
- Test: inline tests in `rtc_session.rs` + `config.rs`
**Interfaces:**
- Consumes: `config.rs` module (Task 1).
- Produces:
- `pub struct MediaAddressConfig { pub bind_ip: IpAddr, pub advertised_ip: Option<IpAddr>, pub port_range: Option<(u16, u16)> }` + manual `Default` (loopback/None/None = today's behavior), re-exported from `rutster_media`.
- `RtcSession::new_with_config(cfg: &MediaAddressConfig) -> Result<Self, RtcSessionError>`; `RtcSession::new()` delegates with defaults.
- `pub struct MediaThreadOpts { pub media_cfg: MediaAddressConfig }` + manual `Default` (in `media_thread.rs`; Tasks 3/7 add fields — **manual** `Default`, no derive, so later `Arc<dyn EventSink>` fits).
- `MediaThread::spawn(default_tap_url: url::Url, opts: MediaThreadOpts, tokio_handle: tokio::runtime::Handle)` (signature change; existing test call sites updated in this task).
- `AppState::spawn_media_thread_with(self, opts: MediaThreadOpts, tokio_handle) -> Result<(Self, MediaThread), std::io::Error>`; existing `spawn_media_thread` delegates with `MediaThreadOpts::default()` so integration tests don't churn.
- `config::parse_port_range(&str) -> Result<(u16, u16), String>`, `config::media_address_config(bind_ip, advertised_ip, port_range: Option<String> ×3) -> Result<MediaAddressConfig, String>`.
- Env: `RUTSTER_MEDIA_BIND_IP` (default `127.0.0.1`), `RUTSTER_MEDIA_ADVERTISED_IP` (default: bind IP), `RUTSTER_MEDIA_PORT_RANGE` (`"lo-hi"` inclusive; default: OS-ephemeral).
- [ ] **Step 1: Write failing tests in `rtc_session.rs`** (append inside the existing `#[cfg(test)] mod tests`)
```rust
#[test]
fn default_config_binds_loopback_ephemeral() {
// Byte-for-byte behavioral compatibility: new() == new_with_config(default).
let s = RtcSession::new_with_config(&MediaAddressConfig::default()).expect("session");
assert!(s.local_addr.ip().is_loopback());
}
#[test]
fn port_range_is_respected_and_exhaustion_errors() {
let cfg = MediaAddressConfig {
bind_ip: "127.0.0.1".parse().unwrap(),
advertised_ip: None,
port_range: Some((41500, 41501)), // room for exactly two sessions
};
let a = RtcSession::new_with_config(&cfg).expect("first");
let b = RtcSession::new_with_config(&cfg).expect("second");
assert!((41500..=41501).contains(&a.local_addr.port()));
assert!((41500..=41501).contains(&b.local_addr.port()));
// Range full → cold-path Socket error, not a panic.
assert!(matches!(
RtcSession::new_with_config(&cfg),
Err(RtcSessionError::Socket(_))
));
}
#[test]
fn sdp_answer_advertises_the_advertised_ip_not_the_bind_ip() {
let cfg = MediaAddressConfig {
bind_ip: "127.0.0.1".parse().unwrap(),
advertised_ip: Some("203.0.113.7".parse().unwrap()), // TEST-NET-3
port_range: None,
};
let mut s = RtcSession::new_with_config(&cfg).expect("session");
let answer = s.accept_offer(BROWSER_SDP_OFFER).expect("answer");
assert!(answer.contains("203.0.113.7"), "candidate must carry advertised IP");
assert!(!answer.contains("127.0.0.1 "), "bind IP must not leak into the candidate");
}
#[test]
fn unspecified_bind_without_advertised_ip_is_rejected() {
// str0m's Candidate::host rejects 0.0.0.0 — fail at construction
// with a config error, not at offer-accept with an expect().
let cfg = MediaAddressConfig {
bind_ip: "0.0.0.0".parse().unwrap(),
advertised_ip: None,
port_range: None,
};
assert!(RtcSession::new_with_config(&cfg).is_err());
}
```
- [ ] **Step 2: Run to verify failure**
Run: `cargo test -p rutster-media rtc_session`
Expected: compile error — `MediaAddressConfig` / `new_with_config` undefined.
- [ ] **Step 3: Implement in `rtc_session.rs`**
Add after the `RtcSessionError` enum (below line 55):
```rust
use std::net::{IpAddr, SocketAddr, UdpSocket};
/// Where the media socket binds vs. what address peers are told to reach.
///
/// # Why bind and advertised are SEPARATE concepts (slice-5, review B1)
///
/// Behind cloud NAT / a 1:1 elastic IP, the address a peer must send RTP
/// to is NOT the local bind address — and RTP cannot ride the HTTP load
/// balancer, so every instance must advertise its own reachable address.
/// str0m also rejects the unspecified address (`0.0.0.0`) as a candidate,
/// which forces the split at the type level: you may bind wildcard, but
/// you must then say what to advertise.
///
/// `port_range` bounds the UDP ports (inclusive) so a fleet's security
/// groups can be written; `None` keeps the OS-ephemeral behavior.
#[derive(Debug, Clone)]
pub struct MediaAddressConfig {
pub bind_ip: IpAddr,
pub advertised_ip: Option<IpAddr>,
pub port_range: Option<(u16, u16)>,
}
impl Default for MediaAddressConfig {
/// Loopback + ephemeral — byte-for-byte the pre-slice-5 behavior, so
/// every existing test and the dev loop run unchanged.
fn default() -> Self {
Self {
bind_ip: IpAddr::from([127, 0, 0, 1]),
advertised_ip: None,
port_range: None,
}
}
}
/// Bind the first free UDP port in `[lo, hi]` (inclusive).
///
/// Sequential scan — O(range) worst case, cold path only (session
/// construction). The OS is the allocator: a failed bind means "taken,"
/// so no shared allocator state is needed across sessions or threads.
fn bind_in_range(ip: IpAddr, lo: u16, hi: u16) -> std::io::Result<UdpSocket> {
let mut last_err = None;
for port in lo..=hi {
match UdpSocket::bind(SocketAddr::new(ip, port)) {
Ok(s) => return Ok(s),
Err(e) => last_err = Some(e),
}
}
Err(last_err.unwrap_or_else(|| {
std::io::Error::new(std::io::ErrorKind::AddrInUse, "empty port range")
}))
}
```
Change the struct: add a field after `local_addr` (line 88):
```rust
/// The address written into our ICE host candidate — advertised IP
/// (if configured) + the actually-bound port. `local_addr` stays the
/// real socket address (loop_driver's `Receive::new` needs it);
/// `advertised_addr` is what the SDP answer tells the peer.
pub(crate) advertised_addr: SocketAddr,
```
Replace `new()` / `new_internal()` (lines 116199):
```rust
pub fn new() -> Result<Self, RtcSessionError> {
Self::new_with_config(&MediaAddressConfig::default())
}
/// Construct with explicit addressing (slice-5, review B1). The
/// binary threads `MediaThreadOpts.media_cfg` here; `new()` keeps the
/// loopback default for tests and the dev loop.
pub fn new_with_config(cfg: &MediaAddressConfig) -> Result<Self, RtcSessionError> {
// Fail-fast: a wildcard bind with nothing to advertise would die
// later inside accept_offer (str0m rejects 0.0.0.0 candidates).
// Cold path → a real error, not an expect().
if cfg.bind_ip.is_unspecified() && cfg.advertised_ip.is_none() {
return Err(RtcSessionError::Socket(std::io::Error::new(
std::io::ErrorKind::InvalidInput,
"RUTSTER_MEDIA_BIND_IP is unspecified (0.0.0.0/::) — set RUTSTER_MEDIA_ADVERTISED_IP",
)));
}
let socket = match cfg.port_range {
Some((lo, hi)) => bind_in_range(cfg.bind_ip, lo, hi)?,
None => UdpSocket::bind(SocketAddr::new(cfg.bind_ip, 0))?,
};
socket.set_nonblocking(true)?;
let local_addr = socket.local_addr()?;
let advertised_addr =
SocketAddr::new(cfg.advertised_ip.unwrap_or(cfg.bind_ip), local_addr.port());
let rtc = Rtc::new(Instant::now());
Ok(Self {
channel: Channel::new_inbound(),
rtc,
decoder: OpusDecoder::new()?,
encoder: OpusEncoder::new()?,
pipe: Box::new(EchoAudioPipe::new()),
socket,
local_addr,
advertised_addr,
audio_mid: None,
next_timeout: None,
last_rx: Instant::now(),
last_outbound_at: Instant::now(),
next_media_time: str0m::media::MediaTime::ZERO,
})
}
```
In `accept_offer` (line 236) change the candidate to the advertised address:
```rust
let candidate = str0m::Candidate::host(self.advertised_addr, "udp")
.expect("host candidate from validated advertised address");
// ^-- expect stays acceptable: new_with_config rejected the only
// input (unspecified IP) that str0m refuses, so this is const-ish.
```
Keep the old `use std::net::SocketAddr;` import (line 20) — fold into the new `use` line.
In `crates/rutster-media/src/lib.rs` add:
```rust
pub use rtc_session::MediaAddressConfig;
```
- [ ] **Step 4: Run media tests**
Run: `cargo test -p rutster-media`
Expected: all pass, including the four new tests.
- [ ] **Step 5: Introduce `MediaThreadOpts` and thread the config (binary crate)**
In `media_thread.rs`, after `CMD_CHANNEL_CAPACITY` (line 39):
```rust
/// Knobs the binary resolves from env and hands to the media thread.
/// Manual Default (not derive) because later slices add non-derivable
/// fields (Task 7's `Arc<dyn EventSink>`). Defaults reproduce the
/// pre-slice-5 single-node dev behavior exactly.
pub struct MediaThreadOpts {
pub media_cfg: rutster_media::MediaAddressConfig,
}
impl Default for MediaThreadOpts {
fn default() -> Self {
Self {
media_cfg: rutster_media::MediaAddressConfig::default(),
}
}
}
```
`MediaThread::spawn` (line 96) gains the param and forwards it:
```rust
pub fn spawn(
default_tap_url: url::Url,
opts: MediaThreadOpts,
tokio_handle: tokio::runtime::Handle,
) -> Result<Self, std::io::Error> {
let (cmd_tx, cmd_rx) = mpsc::channel(CMD_CHANNEL_CAPACITY);
let cmd_tx_for_thread = cmd_tx.clone();
let join = std::thread::Builder::new()
.name("rutster-media".into())
.spawn(move || {
run_media_thread(cmd_rx, default_tap_url, opts, tokio_handle, cmd_tx_for_thread);
})?;
Ok(Self {
cmd_tx,
join: Some(join),
})
}
```
`run_media_thread` (line 158) gains `opts: MediaThreadOpts` after `default_tap_url`, and
the Register arm (line 171) constructs with it:
```rust
MediaCmd::Register { tap_url, reply } => {
match RtcSession::new_with_config(&opts.media_cfg) {
```
(the rest of the arm is unchanged). Update the in-file test (line 316):
```rust
let thread =
MediaThread::spawn(url, MediaThreadOpts::default(), handle).expect("media thread spawn in test");
```
In `session_map.rs`, replace `spawn_media_thread` (lines 8895) with the pair:
```rust
/// Spawn with defaults — the test-facing convenience; production goes
/// through `spawn_media_thread_with` so env config reaches the thread.
pub fn spawn_media_thread(
self,
tokio_handle: tokio::runtime::Handle,
) -> Result<(Self, MediaThread), std::io::Error> {
self.spawn_media_thread_with(crate::media_thread::MediaThreadOpts::default(), tokio_handle)
}
pub fn spawn_media_thread_with(
mut self,
opts: crate::media_thread::MediaThreadOpts,
tokio_handle: tokio::runtime::Handle,
) -> Result<(Self, MediaThread), std::io::Error> {
let thread = MediaThread::spawn(self.default_tap_url.clone(), opts, tokio_handle)?;
self.cmd_tx = thread.cmd_tx();
Ok((self, thread))
}
```
- [ ] **Step 6: config.rs parsers + failing tests first**
Append tests to `config.rs`:
```rust
#[test]
fn port_range_parses_inclusive_pair() {
assert_eq!(parse_port_range("10000-20000").unwrap(), (10000, 20000));
}
#[test]
fn port_range_rejects_inverted_and_garbage() {
assert!(parse_port_range("20000-10000").is_err());
assert!(parse_port_range("10000").is_err());
assert!(parse_port_range("a-b").is_err());
}
#[test]
fn media_address_config_defaults_to_loopback_ephemeral() {
let cfg = media_address_config(None, None, None).unwrap();
assert!(cfg.bind_ip.is_loopback());
assert!(cfg.advertised_ip.is_none());
assert!(cfg.port_range.is_none());
}
#[test]
fn media_address_config_parses_all_three() {
let cfg = media_address_config(
Some("0.0.0.0".into()),
Some("203.0.113.7".into()),
Some("10000-10100".into()),
)
.unwrap();
assert!(cfg.bind_ip.is_unspecified());
assert_eq!(cfg.advertised_ip.unwrap().to_string(), "203.0.113.7");
assert_eq!(cfg.port_range, Some((10000, 10100)));
}
```
Run: `cargo test -p rutster config::` — expected: compile failure. Then implement:
```rust
use std::net::IpAddr;
use rutster_media::MediaAddressConfig;
/// Parse `RUTSTER_MEDIA_PORT_RANGE` — `"lo-hi"`, inclusive, lo ≤ hi.
pub fn parse_port_range(raw: &str) -> Result<(u16, u16), String> {
let (lo, hi) = raw
.split_once('-')
.ok_or_else(|| format!("RUTSTER_MEDIA_PORT_RANGE {raw:?}: expected \"lo-hi\""))?;
let lo: u16 = lo.trim().parse().map_err(|e| format!("RUTSTER_MEDIA_PORT_RANGE lo: {e}"))?;
let hi: u16 = hi.trim().parse().map_err(|e| format!("RUTSTER_MEDIA_PORT_RANGE hi: {e}"))?;
if lo > hi {
return Err(format!("RUTSTER_MEDIA_PORT_RANGE {raw:?}: lo > hi"));
}
Ok((lo, hi))
}
/// Assemble `MediaAddressConfig` from the three `RUTSTER_MEDIA_*` vars.
pub fn media_address_config(
bind_ip: Option<String>,
advertised_ip: Option<String>,
port_range: Option<String>,
) -> Result<MediaAddressConfig, String> {
let mut cfg = MediaAddressConfig::default();
if let Some(s) = bind_ip {
cfg.bind_ip = s.parse::<IpAddr>().map_err(|e| format!("RUTSTER_MEDIA_BIND_IP: {e}"))?;
}
if let Some(s) = advertised_ip {
cfg.advertised_ip =
Some(s.parse::<IpAddr>().map_err(|e| format!("RUTSTER_MEDIA_ADVERTISED_IP: {e}"))?);
}
if let Some(s) = port_range {
cfg.port_range = Some(parse_port_range(&s)?);
}
Ok(cfg)
}
```
Run: `cargo test -p rutster config::` — expected: all pass.
- [ ] **Step 7: Wire main.rs**
In `main.rs`, after the `default_tap_url` block (line 35), build opts and use the `_with`
spawn:
```rust
let media_cfg = rutster::config::media_address_config(
std::env::var("RUTSTER_MEDIA_BIND_IP").ok(),
std::env::var("RUTSTER_MEDIA_ADVERTISED_IP").ok(),
std::env::var("RUTSTER_MEDIA_PORT_RANGE").ok(),
)
.expect("RUTSTER_MEDIA_* env config invalid");
let opts = rutster::media_thread::MediaThreadOpts { media_cfg };
```
and change the spawn call (line 40):
```rust
let (app_state, media_thread) = app_state
.spawn_media_thread_with(opts, tokio::runtime::Handle::current())
.expect("media thread spawn");
```
- [ ] **Step 8: Re-pin the CI seam gate — SAME COMMIT as the code**
```bash
cargo fmt # hash must be of the final formatted content
git add crates/rutster-media/src/rtc_session.rs
git hash-object crates/rutster-media/src/rtc_session.rs
```
Paste the printed hash into `.github/workflows/ci.yml:54` (`EXPECTED_RTC_SESSION='<new>'`)
and replace the comment block (lines 4450) with:
```yaml
# Seam gate (slice-4 §7 #3; re-pinned slice-5):
#
# `loop_driver.rs` stays byte-identical to slice-3 — the hot-path
# poll loop is untouched. `rtc_session.rs` was re-pinned in slice-5
# for MediaAddressConfig (2026-07-04 scalability review, B1:
# bind/advertised address split + port range). If a legitimate
# change is needed, update the pinned blob hashes below in the same
# PR — loud and reviewable by design.
```
Verify locally exactly as CI will:
```bash
test "$(git hash-object crates/rutster-media/src/loop_driver.rs)" = "744bf314edf7f4925c8bb3bd0f5176dbc88f8113" && echo LOOP_DRIVER_OK
```
Expected: `LOOP_DRIVER_OK`.
- [ ] **Step 9: Full check + commit**
Run: `cargo fmt --check && cargo clippy --all -- -D warnings && cargo test --all`
Expected: clean.
```bash
git add -A
git commit -s -m "slice-5: MediaAddressConfig — advertised addr + port range (review B1)
Re-pins the rtc_session.rs seam hash (loud by design). loop_driver.rs
unchanged. Default config reproduces pre-slice-5 behavior exactly."
```
---
### Task 3: Admission cap + tick-lag gauge + `MediaCmd::Stats` (review M2)
**Files:**
- Modify: `crates/rutster/src/media_thread.rs` (cap, gauge, compensated sleep, Stats)
- Modify: `crates/rutster/src/routes.rs:56-60` (503 mapping)
- Modify: `crates/rutster/src/config.rs` + `crates/rutster/src/main.rs` (`RUTSTER_MAX_SESSIONS`)
- Test: `media_thread.rs` inline + `crates/rutster/tests/api_integration.rs`
**Interfaces:**
- Consumes: `MediaThreadOpts` (Task 2).
- Produces:
- `MediaThreadOpts.max_sessions: usize` (manual `Default``64`).
- `MediaCmd::Stats { reply: oneshot::Sender<MediaStats> }`.
- `#[derive(Debug, Clone, serde::Serialize)] pub struct MediaStats { pub sessions: usize, pub max_sessions: usize, pub draining: bool, pub tick_overruns: u64, pub last_tick_micros: u64 }``draining` is hardwired `false` until Task 4.
- Register error strings `"node full"` (routes contract → 503).
- `config::max_sessions(raw: Option<String>) -> Result<usize, String>`.
- [ ] **Step 1: Failing tests (media_thread.rs test module)**
```rust
#[test]
fn register_rejects_when_at_capacity_and_stats_reports() {
let rt = tokio::runtime::Runtime::new().unwrap();
let handle = rt.handle().clone();
let url = url::Url::parse("ws://127.0.0.1:8081/echo").unwrap();
let opts = MediaThreadOpts {
max_sessions: 1,
..Default::default()
};
let thread = MediaThread::spawn(url.clone(), opts, handle).expect("spawn");
let register = |thread: &MediaThread| {
let (reply, rx) = oneshot::channel();
thread
.cmd_tx()
.blocking_send(MediaCmd::Register {
tap_url: url::Url::parse("ws://127.0.0.1:1/echo").unwrap(),
reply,
})
.unwrap();
rx.blocking_recv().expect("reply")
};
assert!(register(&thread).is_ok(), "first session fits");
let err = register(&thread).expect_err("second must be rejected");
assert!(err.contains("node full"), "routes contract string: {err}");
let (reply, rx) = oneshot::channel();
thread.cmd_tx().blocking_send(MediaCmd::Stats { reply }).unwrap();
let stats = rx.blocking_recv().expect("stats");
assert_eq!(stats.sessions, 1);
assert_eq!(stats.max_sessions, 1);
assert!(!stats.draining);
thread.shutdown();
}
```
Run: `cargo test -p rutster media_thread` — expected: compile failure (no `max_sessions`,
no `Stats`).
- [ ] **Step 2: Implement**
`MediaThreadOpts` gains the field (Default → 64, with a comment: "64 is a placeholder
until the ADR-0010 benchmark produces the real per-node number — the gauge below is that
benchmark's primary readout"). Add to `MediaCmd`:
```rust
/// Cold-path capacity/health snapshot (readyz + the autoscaling
/// signal; review M2). Cheap: counters the loop already maintains.
Stats { reply: oneshot::Sender<MediaStats> },
```
Add next to `MediaCmd`:
```rust
/// What the node tells the platform about itself. `serde::Serialize`
/// because /readyz returns it verbatim as JSON.
#[derive(Debug, Clone, serde::Serialize)]
pub struct MediaStats {
pub sessions: usize,
pub max_sessions: usize,
pub draining: bool,
/// Ticks whose work exceeded META_TICK — the saturation signal.
/// Overload degrades EVERY call at once (missed 20ms deadlines), so
/// this must trend at ~0; an autoscaler scales out on its slope.
pub tick_overruns: u64,
pub last_tick_micros: u64,
}
```
In `run_media_thread`, before the loop:
```rust
let max_sessions = opts.max_sessions;
let mut tick_overruns: u64 = 0;
let mut last_tick_micros: u64 = 0;
```
Register arm gains the admission check as its first line:
```rust
MediaCmd::Register { tap_url, reply } => {
// Admission control (review M2): shed the marginal
// call with a crisp 503 instead of degrading every
// in-flight call's 20ms budget. "node full" is the
// routes-layer contract string.
if sessions.len() >= max_sessions {
let _ = reply.send(Err(format!(
"node full: {} sessions (max {max_sessions})",
sessions.len()
)));
continue;
}
match RtcSession::new_with_config(&opts.media_cfg) {
// ... existing arm body unchanged ...
```
Stats arm (in the command drain):
```rust
MediaCmd::Stats { reply } => {
let _ = reply.send(MediaStats {
sessions: sessions.len(),
max_sessions,
draining: false, // Task 4 wires the real flag
tick_overruns,
last_tick_micros,
});
}
```
Replace the tail of the loop (`std::thread::sleep(META_TICK);`, line 303) with the
instrumented, compensated sleep — the tick start is captured **before** the command drain
(first line inside `loop {`):
```rust
let tick_started = Instant::now();
```
and the tail becomes:
```rust
// === Step 3: compensated sleep (review M2). ===
// The old fixed sleep(META_TICK) silently stretched the effective
// tick to 10ms + work; under load that pushes every session past
// its 20ms outbound deadline SIMULTANEOUSLY. Sleep only the
// remainder, and count the ticks where there was none to sleep.
let elapsed = tick_started.elapsed();
last_tick_micros = elapsed.as_micros() as u64;
if elapsed >= META_TICK {
tick_overruns += 1;
}
std::thread::sleep(META_TICK.saturating_sub(elapsed));
```
- [ ] **Step 3: Run the new test**
Run: `cargo test -p rutster media_thread`
Expected: PASS (both media_thread tests).
- [ ] **Step 4: routes 503 mapping + integration test**
In `routes.rs` `create_session` (lines 5660), replace the error arm:
```rust
Err(e) => {
// Contract strings from the media thread (see media_thread.rs):
// capacity/drain rejections are retryable-elsewhere → 503 so an
// LB retries the next node; everything else stays 500.
if e.contains("node full") || e.contains("draining") {
return (StatusCode::SERVICE_UNAVAILABLE, e).into_response();
}
tracing::error!(error = ?e, "session create failed");
StatusCode::INTERNAL_SERVER_ERROR.into_response()
}
```
Append to `crates/rutster/tests/api_integration.rs` (match the file's existing
`tower::ServiceExt::oneshot` pattern for building requests):
```rust
#[tokio::test]
async fn create_session_returns_503_when_node_full() {
use rutster::media_thread::MediaThreadOpts;
let state = rutster::session_map::AppState::default();
let opts = MediaThreadOpts {
max_sessions: 0, // a node that can never admit — the LB-shed path
..Default::default()
};
let (state, _thread) = state
.spawn_media_thread_with(opts, tokio::runtime::Handle::current())
.expect("spawn");
let app = rutster::routes::router(state);
let resp = app
.oneshot(
http::Request::builder()
.method("POST")
.uri("/v1/sessions")
.body(axum::body::Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(resp.status(), http::StatusCode::SERVICE_UNAVAILABLE);
}
```
Run: `cargo test -p rutster --test api_integration`
Expected: PASS.
- [ ] **Step 5: `RUTSTER_MAX_SESSIONS` in config.rs + main.rs**
config.rs (test first: `max_sessions(None) == Ok(64)`, `Some("128") == Ok(128)`,
`Some("zero")` errs mentioning the var name):
```rust
/// `RUTSTER_MAX_SESSIONS` — admission cap. Default 64 (placeholder until
/// the ADR-0010 benchmark measures the real per-node ceiling).
pub fn max_sessions(raw: Option<String>) -> Result<usize, String> {
match raw {
None => Ok(64),
Some(s) => s.parse().map_err(|e| format!("RUTSTER_MAX_SESSIONS {s:?}: {e}")),
}
}
```
main.rs opts construction becomes:
```rust
let opts = rutster::media_thread::MediaThreadOpts {
media_cfg,
max_sessions: rutster::config::max_sessions(std::env::var("RUTSTER_MAX_SESSIONS").ok())
.expect("RUTSTER_MAX_SESSIONS must be a number"),
};
```
- [ ] **Step 6: Full check + commit**
Run: `cargo fmt --check && cargo clippy --all -- -D warnings && cargo test --all`
```bash
git add -A
git commit -s -m "slice-5: admission cap, tick-lag gauge, MediaCmd::Stats (review M2)"
```
---
### Task 4: Drain lifecycle (review M1)
**Files:**
- Modify: `crates/rutster/src/media_thread.rs` (Drain variant, draining flag)
- Modify: `crates/rutster/src/main.rs` (two-phase shutdown)
- Modify: `crates/rutster/src/config.rs` (`RUTSTER_DRAIN_DEADLINE_SECS`)
- Test: `media_thread.rs` inline
**Interfaces:**
- Consumes: Task 3's `MediaStats`/Register-reject shape.
- Produces:
- `MediaCmd::Drain { reply: oneshot::Sender<()> }` — flips draining on; reply fires when the session map empties (immediately if already empty).
- Register rejection string `"draining"` (routes 503 contract — mapping already landed in Task 3).
- `MediaStats.draining` reports the real flag.
- `config::drain_deadline(raw: Option<String>) -> Result<std::time::Duration, String>` — default `0s` (dev loop keeps today's instant shutdown; operators opt into real drain).
- [ ] **Step 1: Failing test**
```rust
#[test]
fn drain_rejects_new_sessions_and_completes_when_map_empties() {
let rt = tokio::runtime::Runtime::new().unwrap();
let handle = rt.handle().clone();
let url = url::Url::parse("ws://127.0.0.1:8081/echo").unwrap();
let thread =
MediaThread::spawn(url.clone(), MediaThreadOpts::default(), handle).expect("spawn");
// one live session
let (reply, rx) = oneshot::channel();
thread
.cmd_tx()
.blocking_send(MediaCmd::Register {
tap_url: url::Url::parse("ws://127.0.0.1:1/echo").unwrap(),
reply,
})
.unwrap();
let id = rx.blocking_recv().unwrap().unwrap();
// drain: must NOT complete while the session lives
let (drain_reply, mut drain_rx) = oneshot::channel();
thread
.cmd_tx()
.blocking_send(MediaCmd::Drain { reply: drain_reply })
.unwrap();
std::thread::sleep(Duration::from_millis(50));
assert!(
drain_rx.try_recv().is_err(),
"drain must wait for in-flight sessions"
);
// new registers are shed with the contract string
let (reply, rx) = oneshot::channel();
thread
.cmd_tx()
.blocking_send(MediaCmd::Register {
tap_url: url::Url::parse("ws://127.0.0.1:1/echo").unwrap(),
reply,
})
.unwrap();
let err = rx.blocking_recv().unwrap().expect_err("draining rejects");
assert!(err.contains("draining"), "{err}");
// deleting the last session completes the drain
let (reply, rx) = oneshot::channel();
thread
.cmd_tx()
.blocking_send(MediaCmd::Delete { id, reply })
.unwrap();
rx.blocking_recv().unwrap();
assert!(
drain_rx.blocking_recv().is_ok(),
"drain completes once the map is empty"
);
thread.shutdown();
}
```
Run: `cargo test -p rutster media_thread` — expected: compile failure (no `Drain`).
- [ ] **Step 2: Implement**
`MediaCmd` gains:
```rust
/// Enter drain (review M1): reject new Registers ("draining" → 503),
/// keep ticking existing sessions, fire `reply` when the map empties.
/// The deadline lives with the CALLER (main.rs times out and proceeds
/// to Shutdown) — the thread just reports emptiness; that keeps
/// "how long is too long" an operator policy, not thread logic.
Drain { reply: oneshot::Sender<()> },
```
`run_media_thread` state, next to the counters:
```rust
let mut draining = false;
let mut drain_done: Option<oneshot::Sender<()>> = None;
```
Register arm — the draining check goes FIRST (before the capacity check):
```rust
if draining {
let _ = reply.send(Err("draining: not accepting new sessions".into()));
continue;
}
```
Drain arm (in the command drain):
```rust
MediaCmd::Drain { reply } => {
draining = true;
if sessions.is_empty() {
let _ = reply.send(());
} else {
drain_done = Some(reply);
}
}
```
Stats arm: `draining: false``draining,`. After the eviction sweep
(`for id in closed_ids { ... }`, line ~297), append:
```rust
// Drain completion check — after eviction so a tick that closes
// the last session completes the drain in the same iteration.
if draining && sessions.is_empty() {
if let Some(done) = drain_done.take() {
let _ = done.send(());
}
}
```
- [ ] **Step 3: Run test**
Run: `cargo test -p rutster media_thread` — expected: PASS.
- [ ] **Step 4: config parser + two-phase shutdown in main.rs**
config.rs (test first: `None → 0s`, `Some("300") → 300s`, garbage errs with var name):
```rust
/// `RUTSTER_DRAIN_DEADLINE_SECS` — how long shutdown waits for in-flight
/// calls before the hard stop. Default 0 = today's instant shutdown (dev
/// loop); production sets minutes. Calls are non-migratable by design, so
/// drain-then-terminate is the ONLY graceful scale-in shape.
pub fn drain_deadline(raw: Option<String>) -> Result<std::time::Duration, String> {
match raw {
None => Ok(std::time::Duration::ZERO),
Some(s) => s
.parse::<u64>()
.map(std::time::Duration::from_secs)
.map_err(|e| format!("RUTSTER_DRAIN_DEADLINE_SECS {s:?}: {e}")),
}
}
```
main.rs — restructure the serve/shutdown tail (lines 4452). The key ordering change:
**drain runs while axum still serves** (in-flight calls still need DELETE/offer during
bleed-out; the LB pulls the node via readyz, Task 5), and only then does HTTP quiesce:
```rust
let addr: SocketAddr = rutster::config::http_bind(std::env::var("RUTSTER_HTTP_BIND").ok())
.expect("RUTSTER_HTTP_BIND must be host:port");
let drain_deadline =
rutster::config::drain_deadline(std::env::var("RUTSTER_DRAIN_DEADLINE_SECS").ok())
.expect("RUTSTER_DRAIN_DEADLINE_SECS must be seconds");
info!(%addr, "listening");
let listener = tokio::net::TcpListener::bind(addr).await.unwrap();
// Two-phase shutdown (slice-5, review M1):
// SIGTERM → Drain (stop admitting; keep serving HTTP for in-flight
// calls' DELETE/offer; readyz flips so the LB pulls us) → deadline
// → HTTP quiesce → media_thread.shutdown() (the hard stop).
let (http_stop_tx, http_stop_rx) = tokio::sync::oneshot::channel::<()>();
let drain_cmd_tx = app_state.cmd_tx.clone();
tokio::spawn(async move {
shutdown_signal().await;
if !drain_deadline.is_zero() {
let (reply, drained) = tokio::sync::oneshot::channel();
if drain_cmd_tx
.send(rutster::media_thread::MediaCmd::Drain { reply })
.await
.is_ok()
{
match tokio::time::timeout(drain_deadline, drained).await {
Ok(_) => info!("drain complete; proceeding to shutdown"),
Err(_) => info!(?drain_deadline, "drain deadline hit; shutting down anyway"),
}
}
}
let _ = http_stop_tx.send(());
});
axum::serve(listener, router(app_state))
.with_graceful_shutdown(async {
let _ = http_stop_rx.await;
})
.await
.unwrap();
media_thread.shutdown();
```
(The `shutdown_signal` doc comment at main.rs:5559 — delete the sentence "No in-flight
call preservation story in the dev loop." and describe the two-phase flow instead.)
- [ ] **Step 5: Full check + commit**
Run: `cargo fmt --check && cargo clippy --all -- -D warnings && cargo test --all`
```bash
git add -A
git commit -s -m "slice-5: drain lifecycle — SIGTERM bleeds out calls (review M1)"
```
---
### Task 5: `GET /healthz` + `GET /readyz`
> **Deviation note (post-review):** the snippets below contain two bugs found and fixed
> in commit 5ce18bf during execution — the readyz timeout must wrap the SEND as well as
> the reply await, and the integration test must shut its `MediaThread` down. Copy from
> the landed code, not these snippets.
**Files:**
- Modify: `crates/rutster/src/routes.rs` (two routes + handlers)
- Test: `routes.rs` inline (default-AppState zombie case) + `crates/rutster/tests/api_integration.rs` (live case)
**Interfaces:**
- Consumes: `MediaCmd::Stats`, `MediaStats` (Tasks 34).
- Produces:
- `GET /healthz` → 200 `"ok"` always (liveness: process + HTTP stack up).
- `GET /readyz` → 200 + `MediaStats` JSON iff the media thread answers Stats within 250 ms AND `!draining` AND `sessions < max_sessions`; else 503 (body: the JSON if available, `"media thread unresponsive"` if not). Readiness = "can this node take a NEW call."
- [ ] **Step 1: Failing tests**
`routes.rs` test module (the zombie case — this is the *point* of readyz: a dead media
thread must flip readiness off while the HTTP listener still answers):
```rust
#[tokio::test]
async fn readyz_503_when_media_thread_gone_but_healthz_200() {
use tower::ServiceExt;
// Default AppState: closed placeholder channel = dead media thread.
let app = router(AppState::default());
let ready = app
.clone()
.oneshot(
axum::http::Request::builder()
.uri("/readyz")
.body(axum::body::Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(ready.status(), StatusCode::SERVICE_UNAVAILABLE);
let health = app
.oneshot(
axum::http::Request::builder()
.uri("/healthz")
.body(axum::body::Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(health.status(), StatusCode::OK);
}
```
`api_integration.rs` (live case):
```rust
#[tokio::test]
async fn readyz_200_with_stats_json_when_thread_alive() {
let state = rutster::session_map::AppState::default();
let (state, _thread) = state
.spawn_media_thread(tokio::runtime::Handle::current())
.expect("spawn");
let app = rutster::routes::router(state);
let resp = app
.oneshot(
http::Request::builder()
.uri("/readyz")
.body(axum::body::Body::empty())
.unwrap(),
)
.await
.unwrap();
assert_eq!(resp.status(), http::StatusCode::OK);
let body = axum::body::to_bytes(resp.into_body(), 64 * 1024).await.unwrap();
let v: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert_eq!(v["draining"], false);
assert!(v["max_sessions"].as_u64().unwrap() > 0);
}
```
Run: `cargo test -p rutster` — expected: FAIL (404 on the routes).
- [ ] **Step 2: Implement handlers + routes**
routes.rs:
```rust
/// GET /healthz — liveness only: the process and HTTP stack are up.
/// Deliberately does NOT consult the media thread — liveness and
/// readiness are different probes with different restart semantics.
pub async fn healthz() -> Response {
(StatusCode::OK, "ok").into_response()
}
/// GET /readyz — "can this node accept a NEW call right now?"
/// LB target membership + autoscaler both read this. 503 when draining,
/// at capacity, or when the media thread doesn't answer Stats in 250ms
/// (a wedged/dead media thread previously left GET / answering 200 —
/// the zombie-node failure from the 2026-07-04 review).
pub async fn readyz(State(state): State<AppState>) -> Response {
let (reply, rx) = tokio::sync::oneshot::channel();
if state
.cmd_tx
.send(crate::media_thread::MediaCmd::Stats { reply })
.await
.is_err()
{
return (StatusCode::SERVICE_UNAVAILABLE, "media thread unresponsive").into_response();
}
let stats = match tokio::time::timeout(std::time::Duration::from_millis(250), rx).await {
Ok(Ok(s)) => s,
_ => {
return (StatusCode::SERVICE_UNAVAILABLE, "media thread unresponsive")
.into_response();
}
};
let ready = !stats.draining && stats.sessions < stats.max_sessions;
let code = if ready { StatusCode::OK } else { StatusCode::SERVICE_UNAVAILABLE };
(code, Json(stats)).into_response()
}
```
Router additions (in `router()`, before `.with_state`):
```rust
.route("/healthz", get(healthz))
.route("/readyz", get(readyz))
```
- [ ] **Step 3: Run tests**
Run: `cargo test -p rutster && cargo test -p rutster --test api_integration`
Expected: PASS.
- [ ] **Step 4: Commit**
```bash
git add -A
git commit -s -m "slice-5: /healthz + /readyz — readiness reads MediaCmd::Stats"
```
---
### Task 6: Non-blocking tap teardown (review M7)
**Files:**
- Modify: `crates/rutster/src/tap_engine.rs` (new `spawn_tap_teardown` free fn + test)
- Modify: `crates/rutster/src/media_thread.rs:201-223` (Delete arm calls it)
**Interfaces:**
- Consumes: `TapConn` (existing).
- Produces: `pub fn spawn_tap_teardown(tokio_handle: &tokio::runtime::Handle, id: ChannelId, conn: TapConn)` — returns immediately; the 750 ms bounded close_tx→join→abort sequence runs on a tokio task.
- [ ] **Step 1: Failing test (tap_engine.rs test module)**
```rust
#[test]
fn spawn_tap_teardown_returns_immediately_and_aborts_stuck_engine() {
let rt = tokio::runtime::Runtime::new().unwrap();
// A "stuck brain": the engine task never finishes on its own. Its
// guard's Drop fires on abort — our proof the teardown completed.
struct DropGuard(std::sync::mpsc::Sender<()>);
impl Drop for DropGuard {
fn drop(&mut self) {
let _ = self.0.send(());
}
}
let (dropped_tx, dropped_rx) = std::sync::mpsc::channel();
let join = rt.spawn(async move {
let _guard = DropGuard(dropped_tx);
std::future::pending::<()>().await;
});
let (close_tx, _close_rx) = oneshot::channel();
let conn = TapConn {
close_tx,
join,
metrics: TapMetrics::new(),
flush_rx: None,
rx_function_call: None,
tx_function_call_output: None,
tool_registry: Arc::new(Mutex::new(ToolRegistry::default())),
};
let started = std::time::Instant::now();
spawn_tap_teardown(rt.handle(), rutster_call_model::ChannelId::new(), conn);
// The whole point (review M7): the caller — the media TICK LOOP —
// must not wait out the 750ms brain timeout.
assert!(
started.elapsed() < Duration::from_millis(100),
"teardown must not block the caller"
);
// …but the stuck engine task must still get aborted after the cap.
dropped_rx
.recv_timeout(Duration::from_secs(2))
.expect("stuck engine task was aborted (guard dropped)");
}
```
Run: `cargo test -p rutster tap_engine` — expected: compile failure (`spawn_tap_teardown`
undefined).
- [ ] **Step 2: Implement in tap_engine.rs**
```rust
/// Tear down one session's tap engine WITHOUT blocking the caller.
///
/// # Why this must never run inline on the media thread (review M7)
///
/// The old Delete arm did `tokio_handle.block_on(timeout(750ms, join))`
/// inside the tick loop: one teardown with an unresponsive brain froze
/// the 20 ms loop for EVERY live call on the node (~37 missed frames),
/// and batched Deletes stacked sequentially. Teardown is brain I/O —
/// exactly what the tick loop must never wait on. Same discipline as the
/// tap pipe's try_send posture.
pub fn spawn_tap_teardown(
tokio_handle: &tokio::runtime::Handle,
id: ChannelId,
mut conn: TapConn,
) {
tokio_handle.spawn(async move {
// Gentle path first (AGENTS.md: close_tx is the documented
// trigger, abort is the safety net).
let _ = conn.close_tx.send(());
match tokio::time::timeout(Duration::from_millis(750), &mut conn.join).await {
Ok(Ok(())) => {
info!(channel_id = %id, "tap engine torn down (graceful)");
}
_ => {
conn.join.abort();
info!(channel_id = %id, "tap engine torn down (abort after timeout)");
}
}
});
}
```
Note: `close_tx` is consumed by `send`, but `conn` is only partially moved inside the
async block — Rust allows the field-by-field moves because the closure owns `conn`. If
the borrow checker objects on the `&mut conn.join` + later `conn.join.abort()` pair,
destructure instead: `let TapConn { close_tx, mut join, .. } = conn;`.
- [ ] **Step 3: Rewire the Delete arm (media_thread.rs:201-223)**
```rust
MediaCmd::Delete { id, reply } => {
if let Some(mut s) = sessions.remove(&id) {
if let Some(conn) = s.tap_conn.take() {
// Hand the 750ms bounded teardown to tokio —
// the tick loop never waits on brain I/O
// (review M7; see spawn_tap_teardown docs).
crate::tap_engine::spawn_tap_teardown(&tokio_handle, id, conn);
}
s.rtc.channel.tap = None;
s.rtc.channel.state = rutster_call_model::ChannelState::Closed;
}
let _ = reply.send(());
}
```
- [ ] **Step 4: Run the full suite (the existing tap/barge/realtime integration tests are the regression net for live-engine teardown)**
Run: `cargo test --all`
Expected: PASS — `tap_integration`, `barge_in_integration`, `realtime_integration` all green.
- [ ] **Step 5: Commit**
```bash
git add -A
git commit -s -m "slice-5: Delete teardown off the tick loop (review M7)"
```
---
### Task 7: `EventSink` + wall-clock `started_at` (review M3)
**Files:**
- Modify: `crates/rutster-call-model/src/lib.rs` (Channel gains `started_at: SystemTime`)
- Create: `crates/rutster/src/event_sink.rs`
- Modify: `crates/rutster/src/lib.rs` (`pub mod event_sink;`)
- Modify: `crates/rutster/src/media_thread.rs` (opts gains sink; emissions at Register/Connected/Delete/evict/Shutdown)
- Test: call-model inline; `media_thread.rs` inline with a channel-backed TestSink
**Interfaces:**
- Consumes: `MetricsSnapshot` (rutster-tap), `ChannelId`, `MediaThreadOpts`.
- Produces:
- `Channel.started_at: std::time::SystemTime` (set in `new_inbound`; `created_at: Instant` STAYS — monotonic for idle math, wall-clock for CDR; the doc comments explain the pairing).
- `pub trait EventSink: Send + Sync { fn emit(&self, event: CallEvent); }`**contract:** `emit` is called from the media `std::thread` and MUST NOT block or do I/O inline (a Valkey impl buffers via channel to a tokio task; that impl lands with ADR-0005 wiring, later rung).
- `#[derive(Debug, Clone)] pub enum CallEvent { SessionRegistered { id: ChannelId, at: SystemTime }, SessionConnected { id: ChannelId, at: SystemTime }, SessionEnded { id: ChannelId, started_at: SystemTime, ended_at: SystemTime, reason: EndReason, tap_metrics: Option<MetricsSnapshot> } }`.
- `#[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum EndReason { Deleted, Closed, Shutdown }`.
- `pub struct TracingEventSink;` — emits one structured `tracing::info!(target: "rutster::events", …)` per event (the log-backed placeholder for the bus).
- `MediaThreadOpts.sink: std::sync::Arc<dyn EventSink>` (Default → `Arc::new(TracingEventSink)`).
- [ ] **Step 1: call-model — failing test then field**
Test (call-model tests module):
```rust
#[test]
fn channel_carries_wall_clock_start() {
let before = std::time::SystemTime::now();
let ch = Channel::new_inbound();
let after = std::time::SystemTime::now();
assert!(ch.started_at >= before && ch.started_at <= after);
}
```
Implement: `Channel` gains
```rust
/// Wall-clock start (slice-5, review M3). `created_at: Instant` above
/// measures elapsed time (idle timeout); THIS field anchors the CDR —
/// "when did the call start" as a timestamp a bill or an audit can
/// use. A monotonic Instant cannot be converted to wall-clock after
/// the fact, which is why both exist: Instant for arithmetic,
/// SystemTime for the record.
pub started_at: std::time::SystemTime,
```
and `new_inbound()` sets `started_at: std::time::SystemTime::now(),`.
Run: `cargo test -p rutster-call-model` — expected: PASS.
- [ ] **Step 2: event_sink.rs (types + TracingEventSink)**
```rust
//! # event_sink — durable-lifecycle seam (slice-5, review M3)
//!
//! ADR-0005 promises call lifecycle → Valkey streams → durable CDR
//! pipeline. Until that lands, lifecycle events die with the process —
//! an OOM-killed node erases all evidence of its calls. This module is
//! the SEAM: the media thread emits `CallEvent`s through `EventSink`;
//! today's impl logs, the ADR-0005 impl will publish. The emission
//! points and the event shape are the part that calcifies — the backend
//! is plumbing.
use std::sync::Arc;
use std::time::SystemTime;
use rutster_call_model::ChannelId;
use rutster_tap::MetricsSnapshot;
/// CDR-shaped lifecycle events. `SessionEnded` carries both timestamps so
/// a consumer computes duration without correlating two events (a node
/// can die between them — the whole point of emitting durably).
#[derive(Debug, Clone)]
pub enum CallEvent {
SessionRegistered { id: ChannelId, at: SystemTime },
SessionConnected { id: ChannelId, at: SystemTime },
SessionEnded {
id: ChannelId,
started_at: SystemTime,
ended_at: SystemTime,
reason: EndReason,
/// Tap counters at teardown — fulfills the tap_engine.rs promise
/// ("read a MetricsSnapshot for log/CDR emission on teardown").
/// Snapshot is taken just before the engine task is torn down, so
/// counts may trail the final frame by a tick — fine for a CDR.
tap_metrics: Option<MetricsSnapshot>,
},
}
/// Why the session ended — the CDR disposition embryo.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum EndReason {
/// DELETE /v1/sessions/:id (API-driven hangup).
Deleted,
/// The session closed itself (peer close / idle timeout).
Closed,
/// Node shutdown dropped it (the mass-hangup path drain exists to avoid).
Shutdown,
}
/// The seam. CONTRACT: `emit` is called from the media std::thread —
/// implementations MUST NOT block or perform I/O inline. Buffer to your
/// own task (the future Valkey impl does channel → tokio publisher).
pub trait EventSink: Send + Sync {
fn emit(&self, event: CallEvent);
}
/// Log-backed sink: structured tracing on target "rutster::events". Not
/// durable — the placeholder that makes the emission points real today.
pub struct TracingEventSink;
impl EventSink for TracingEventSink {
fn emit(&self, event: CallEvent) {
tracing::info!(target: "rutster::events", event = ?event, "call event");
}
}
/// Convenience alias for the opts field / spawn signature.
pub type SharedEventSink = Arc<dyn EventSink>;
```
Add `pub mod event_sink;` to `crates/rutster/src/lib.rs`.
(`MetricsSnapshot` must be re-exported from `rutster_tap` — it lives in `metrics.rs`;
check `crates/rutster-tap/src/lib.rs` re-exports it, add `pub use metrics::MetricsSnapshot;`
if absent.)
- [ ] **Step 3: Failing media_thread test with a TestSink**
```rust
#[test]
fn lifecycle_events_emit_registered_then_ended_deleted() {
use crate::event_sink::{CallEvent, EndReason, EventSink};
struct TestSink(std::sync::Mutex<Vec<CallEvent>>);
impl EventSink for TestSink {
fn emit(&self, event: CallEvent) {
self.0.lock().unwrap().push(event);
}
}
let sink = std::sync::Arc::new(TestSink(std::sync::Mutex::new(Vec::new())));
let rt = tokio::runtime::Runtime::new().unwrap();
let handle = rt.handle().clone();
let url = url::Url::parse("ws://127.0.0.1:8081/echo").unwrap();
let opts = MediaThreadOpts {
sink: sink.clone(),
..Default::default()
};
let thread = MediaThread::spawn(url, opts, handle).expect("spawn");
let (reply, rx) = oneshot::channel();
thread
.cmd_tx()
.blocking_send(MediaCmd::Register {
tap_url: url::Url::parse("ws://127.0.0.1:1/echo").unwrap(),
reply,
})
.unwrap();
let id = rx.blocking_recv().unwrap().unwrap();
let (reply, rx) = oneshot::channel();
thread
.cmd_tx()
.blocking_send(MediaCmd::Delete { id, reply })
.unwrap();
rx.blocking_recv().unwrap();
thread.shutdown();
let events = sink.0.lock().unwrap();
assert!(
matches!(&events[0], CallEvent::SessionRegistered { id: eid, .. } if *eid == id)
);
assert!(events.iter().any(|e| matches!(
e,
CallEvent::SessionEnded { id: eid, reason: EndReason::Deleted, .. } if *eid == id
)));
}
```
Run: `cargo test -p rutster media_thread` — expected: compile failure (no `sink` field).
- [ ] **Step 4: Implement emissions**
`MediaThreadOpts` gains `pub sink: crate::event_sink::SharedEventSink,`; `Default` sets
`sink: std::sync::Arc::new(crate::event_sink::TracingEventSink),`. In `run_media_thread`
let-bind `let sink = opts.sink.clone();` before the loop. Emissions:
- Register arm, after `sessions.insert(...)` / before `reply`:
```rust
sink.emit(crate::event_sink::CallEvent::SessionRegistered {
id,
at: started_at,
});
```
(read `let started_at = session.rtc.channel.started_at;` — take it before the
`sessions.insert` moves the session, i.e. right after `let id = session.channel_id();`).
- Connected transition — after the existing `info!(channel_id = %id, "tap engine + reflex
+ local VAD wired on Connected");` and **before the `continue;`** that ends the block:
```rust
sink.emit(crate::event_sink::CallEvent::SessionConnected {
id: *id,
at: std::time::SystemTime::now(),
});
```
- Delete arm — full rewritten arm (ORDER MATTERS: the metrics snapshot must be read
**before** `s.tap_conn.take()` empties the field, or `tap_metrics` is silently `None`):
```rust
MediaCmd::Delete { id, reply } => {
if let Some(mut s) = sessions.remove(&id) {
// Snapshot BEFORE take(): after the take the conn
// (and its metrics Arc) belongs to the teardown task.
let tap_metrics = s.tap_conn.as_ref().map(|c| c.metrics.snapshot());
sink.emit(crate::event_sink::CallEvent::SessionEnded {
id,
started_at: s.rtc.channel.started_at,
ended_at: std::time::SystemTime::now(),
reason: crate::event_sink::EndReason::Deleted,
tap_metrics,
});
if let Some(conn) = s.tap_conn.take() {
crate::tap_engine::spawn_tap_teardown(&tokio_handle, id, conn);
}
s.rtc.channel.tap = None;
s.rtc.channel.state = rutster_call_model::ChannelState::Closed;
}
let _ = reply.send(());
}
```
- Eviction sweep — change the loop to fetch state before removal:
```rust
for id in closed_ids {
if let Some(s) = sessions.remove(&id) {
sink.emit(crate::event_sink::CallEvent::SessionEnded {
id,
started_at: s.rtc.channel.started_at,
ended_at: std::time::SystemTime::now(),
reason: crate::event_sink::EndReason::Closed,
tap_metrics: s.tap_conn.as_ref().map(|c| c.metrics.snapshot()),
});
}
debug!(channel_id = %id, "session evicted after close");
}
```
- Shutdown arm, before `sessions.clear()`:
```rust
for (id, s) in sessions.iter() {
sink.emit(crate::event_sink::CallEvent::SessionEnded {
id: *id,
started_at: s.rtc.channel.started_at,
ended_at: std::time::SystemTime::now(),
reason: crate::event_sink::EndReason::Shutdown,
tap_metrics: s.tap_conn.as_ref().map(|c| c.metrics.snapshot()),
});
}
```
main.rs opts construction gains nothing (Default supplies TracingEventSink) — add a
comment noting the seam: `// sink: TracingEventSink via Default — ADR-0005 Valkey impl replaces it at the same seam.`
- [ ] **Step 5: Run + commit**
Run: `cargo fmt --check && cargo clippy --all -- -D warnings && cargo test --all`
Expected: PASS.
```bash
git add -A
git commit -s -m "slice-5: EventSink seam + wall-clock started_at (review M3)
Lifecycle events now flow through a trait the ADR-0005 Valkey publisher
can implement; Channel carries SystemTime so a CDR can exist."
```
---
### Task 8: ADR-0009 amendment + ADR-0005 note + ARCHITECTURE.md placement line
**Files:**
- Modify: `docs/adr/0009-spend-gate-honest-rescope.md` (append amendment before References)
- Modify: `docs/adr/0005-event-bus.md` (one-line note under Constraints)
- Modify: `docs/ARCHITECTURE.md:34-36` (horizontal list)
- [ ] **Step 1: Append to ADR-0009** (between "Consequences" and "References"):
```markdown
## Amendment 2026-07-04 — enforcement locality ≠ accounting locality
- **Status:** Proposed (pending maintainer ratification)
- **Origin:** [2026-07-04 scalability & infra-fit review](../reviews/2026-07-04-scalability-infra-review.md),
finding M5.
Guarantee 2 above prescribes where the **check** sits — in-process with the tap and the
provider call-control client. It is silent on where the **accounting state** lives, and at
N>1 core instances that silence becomes a correctness trap: per-instance counters make
every spend/pacing cap silently N× the fleet intent, and toll-fraud thresholds never trip
because attempts spread across instances.
Clarification, binding on the step-6 implementation:
1. **Enforcement is in-process** (unchanged — constitutive; guarantee 2 stands).
2. **Accounting is shared.** The gate is built against a ledger trait with atomic
check-and-reserve semantics from day one: an in-memory implementation for single-node,
a Valkey-backed one ([ADR-0005](0005-event-bus.md)) for fleets. The check path never
assumes counter locality.
3. **ADR-0005 constraint 2 is not a counter-argument.** "The bus is not the source of
truth for billing- or call-loss-critical state" governs the *durable CDR*. Live
enforcement counters (spend/pacing/rate state) are ephemeral control state — exactly
what Valkey KV is for. Losing them on a Valkey restart degrades fail-safe (re-count
from zero, provider-side caps as the backstop per this ADR's deployment guidance) —
not to billing corruption.
```
- [ ] **Step 2: ADR-0005 cross-ref** — append to the "Constraints" list:
```markdown
3. **Constraint 2 governs the durable CDR, not enforcement counters.** The spend gate's
live accounting state (spend/pacing/rate counters) is ephemeral control state and DOES
belong in Valkey KV at N>1 — see the
[ADR-0009 amendment 2026-07-04](0009-spend-gate-honest-rescope.md#amendment-2026-07-04--enforcement-locality--accounting-locality).
```
- [ ] **Step 3: ARCHITECTURE.md** — in the "Horizontal platform" sentence (lines 3436), change
```markdown
inventory, billing rollup, analytics, multi-region orchestration, the management API, and the agent
brain itself.
```
to
```markdown
inventory, billing rollup, analytics, multi-region orchestration, call placement (the
session→node directory — which node owns which live call, the routing layer above N cores),
the management API, and the agent brain itself.
```
- [ ] **Step 4: Verify links render + commit**
Run: `ls docs/adr/ && grep -c "Amendment 2026-07-04" docs/adr/0009-spend-gate-honest-rescope.md`
Expected: `1`.
```bash
git add docs/adr/0009-spend-gate-honest-rescope.md docs/adr/0005-event-bus.md docs/ARCHITECTURE.md
git commit -s -m "docs(adr): ADR-0009 amendment — shared spend accounting (review M5)"
```
---
### Task 9: Tap protocol v2 reservations (doc-only; review M4)
**Files:**
- Modify: `crates/rutster-tap/src/protocol.rs` (module doc `//!` section only — NO code)
- [ ] **Step 1: Append to the module doc** (after the existing wire-contract prose):
```rust
//! ## v2 reservations (documented 2026-07-04; implemented at the version bump)
//!
//! The 2026-07-04 scalability review (finding M4) identified the missing
//! reconnect semantics as the most calcifying gap in the system — wire
//! protocols are the hardest surface to retrofit. v1 stays frozen; the
//! v2 negotiation MUST carry:
//!
//! - **`hello.resume_token`** — an opaque token the brain returned on a
//! prior accept for this session, plus a **connection epoch** counter.
//! Lets a brain fleet distinguish "resume this conversation" (state
//! held / recoverable) from "unknown session" (context lost).
//! - **`hello_ack.resume: accepted | rejected`** — the brain's explicit
//! answer. A rejected resume tells the core the brain is amnesiac, so
//! the core (not the caller's ears) decides: re-prime, escalate, or end.
//! - **`bye.reason: terminal | transient`** — today every close funnels
//! into infinite re-dial (5s cap, forever); a brain that deliberately
//! ends a session is re-dialed for the rest of the call. `terminal`
//! exits the retry loop.
//!
//! Anything in this list changes BOTH sides of the wire — which is why it
//! is reserved here, in the protocol's own doc, rather than in a review
//! doc nobody re-reads at implementation time.
```
- [ ] **Step 2: Verify doc renders + commit**
Run: `cargo doc --no-deps -p rutster-tap 2>&1 | tail -2 && cargo test -p rutster-tap`
Expected: doc builds clean; tests untouched and green.
```bash
git add crates/rutster-tap/src/protocol.rs
git commit -s -m "docs(tap): reserve v2 resume/terminal-bye semantics (review M4)"
```
---
## Done criteria (slice level)
1. `cargo fmt --check && cargo clippy --all -- -D warnings && cargo test --all && cargo deny check` — green on stable + 1.85.
2. Seam gate: `loop_driver.rs` hash unchanged; `rtc_session.rs` re-pinned once (Task 2).
3. With **no env vars set**, the dev loop behaves exactly as before slice-5 (loopback media, port 8080, instant Ctrl-C shutdown) — the seams cost nothing until configured.
4. With `RUTSTER_MEDIA_ADVERTISED_IP` + `RUTSTER_MEDIA_PORT_RANGE` + `RUTSTER_HTTP_BIND` + `RUTSTER_MAX_SESSIONS` + `RUTSTER_DRAIN_DEADLINE_SECS` set, the binary is deployable behind NAT with an LB reading `/readyz` — the review's B1/M1/M2 exit criteria.
5. Every call leaves a `rutster::events` record with wall-clock start/end + reason + tap counters (M3's seam, log-backed).
6. PR via `tea`, squash-merge, DCO-signed commits throughout.
```