From fae9fcc82aa52164c0fe3606e5a2316c076daf7a Mon Sep 17 00:00:00 2001 From: "Aaron D. Lee" Date: Sun, 5 Jul 2026 21:37:17 -0400 Subject: [PATCH] =?UTF-8?q?docs(plans):=20four=20deployment-epoch=20implem?= =?UTF-8?q?entation=20plans=20(slices=20A=E2=80=93G)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plan A: engine hygiene (TCP_NODELAY via axum 0.7.9 Serve::tcp_nodelay, trunk WS pings, webhook-base TwiML derivation, trusted-proxy posture). Plan B: packaging + CI (four images, s6 all-in-one, compose, Caddyfile, smoke suite incl. reload-during-call, slice-F publish workflow). Plan C: in-binary features (rustls Phase 1 BYO-cert, /metrics, ValkeyEventSink + smoke hook fill). Plan G: docs/deploy tree + ADR-0011 drafting. Authored by parallel plan-writers (this session + omo rescue after a session-limit interruption); verified: placeholder scan, cited-path existence, cross-plan image/env-var consistency, seam-gate compliance. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8 Signed-off-by: Aaron D. Lee --- .../2026-07-05-deploy-a-engine-hygiene.md | 1676 +++++++++ .../plans/2026-07-05-deploy-b-packaging-ci.md | 3114 +++++++++++++++++ .../2026-07-05-deploy-c-binary-features.md | 2677 ++++++++++++++ .../2026-07-05-deploy-g-docs-adr-0011.md | 1610 +++++++++ 4 files changed, 9077 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-05-deploy-a-engine-hygiene.md create mode 100644 docs/superpowers/plans/2026-07-05-deploy-b-packaging-ci.md create mode 100644 docs/superpowers/plans/2026-07-05-deploy-c-binary-features.md create mode 100644 docs/superpowers/plans/2026-07-05-deploy-g-docs-adr-0011.md diff --git a/docs/superpowers/plans/2026-07-05-deploy-a-engine-hygiene.md b/docs/superpowers/plans/2026-07-05-deploy-a-engine-hygiene.md new file mode 100644 index 0000000..0bc43b2 --- /dev/null +++ b/docs/superpowers/plans/2026-07-05-deploy-a-engine-hygiene.md @@ -0,0 +1,1676 @@ +# Deploy slice A — engine hygiene: TCP_NODELAY, app-level WS pings, trunk config fixes — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use `superpowers:subagent-driven-development` (recommended) or `superpowers:executing-plans` to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Land the three unconditional hygiene items from the deployment-topology spec +([§5.1, §5.2, §5.3](../specs/2026-07-05-deployment-topology-design.md)) — TCP_NODELAY on every +accepted HTTP/WS socket with a CI-regressed latency assertion, engine-originated WS keepalive +pings on the trunk media-stream WS, and the trunk config fixes (derived TwiML Stream URL, wired +`config::twilio_credentials` startup validation, trusted-proxy public-URL reconstruction) — so +slices B (artifacts) and C (rustls Phase 1) build on a listener that is actually fit for a +CPaaS edge. + +**Architecture:** All changes ride the existing seams. A new `rutster::serve` module wraps +`axum::serve(...).tcp_nodelay(true)` so `main.rs` and the sim-bench assertion exercise the same +serve path (axum 0.7.9's `Serve::tcp_nodelay` is verified present in the locked version and +carried through `with_graceful_shutdown`). The WS ping is a third `tokio::select!` branch in the +existing trunk pump loop (`crates/rutster-trunk/src/twilio_media_streams.rs`). The trusted-proxy +URL-reconstruction helper lands in `crates/rutster-trunk` (the future consumer — Twilio signature +validation — lives there; **signature validation itself does not exist yet and is NOT built in +this slice**), with the CIDR-list env parser in the house `config.rs` pattern. + +**Tech Stack:** Rust stable + 1.85 (CI matrix), axum 0.7.9 (locked; has `tcp_nodelay`), the +existing workspace. ONE new direct dep: `ipnet = "2"` (already in `Cargo.lock` transitively via +reqwest — this adds only the direct edge). + +## Global Constraints + +- **License:** GPL-3.0-or-later on every crate manifest (ADR-0004). +- **DCO:** every commit signed off — `git commit -s` (AGENTS.md Git workflow). +- **SEAM GATE (UNCHANGED):** `crates/rutster-media/src/loop_driver.rs` and + `crates/rutster-media/src/rtc_session.rs` stay **byte-identical** — CI pins their blob hashes + (`744bf314edf7f4925c8bb3bd0f5176dbc88f8113` / `f47d63b9a2883d37066a93c9daa0e2cf8816bec4` in + `.github/workflows/ci.yml`). NO task in this plan touches them. This slice is the HTTP/WS + listener and config surface, not the RTP path. +- **Hot-path policy:** never `?`-propagate on the 20 ms tick; match-and-continue; "drop + + observe." No `unwrap()`/`expect()` outside tests/startup (`main.rs` fail-fast `expect`s at + boot are the existing house pattern). +- **Code style:** `cargo fmt` is the single whitespace source of truth; + `cargo clippy --all --all-targets -- -D warnings` is the lint bar. +- **MSRV:** CI matrix runs stable + 1.85. No syntax/std APIs newer than 1.85. (The rcgen 0.14.7 + pin in `Cargo.lock` protects this — do not `cargo update` wholesale.) +- **Workspace dep pinning:** new deps go in the ROOT `Cargo.toml` `[workspace.dependencies]`; + members reference with `dep.workspace = true`. +- **cargo-deny:** `cargo deny check` must stay green (license gate). `ipnet` is dual + MIT/Apache-2.0 and already vetted transitively. +- **sim-bench CI:** the `sim-bench` job runs + `cargo test --all --features=sim-bench -- --test-threads=1` — `--test-threads=1` is + load-bearing (shared tick-lag gauge). The new assertion joins that job unchanged. +- **Branch/PR:** branch `deploy-a/engine-hygiene`; PR to `main` via `tea`. This plan is the + dependency root: slices B and C branch after it merges. + +## File Structure + +### New files + +| Path | Responsibility | +|---|---| +| `crates/rutster/src/serve.rs` | `serve_with_nodelay` — the one production serve path: `axum::serve` + `tcp_nodelay(true)` + graceful shutdown. | +| `crates/rutster/tests/serve_nodelay.rs` | Integration test: the helper serves a request over a real socket and quiesces on the shutdown signal. | +| `crates/rutster-trunk/tests/ws_ping.rs` | Integration test: engine-originated Ping frames arrive on the trunk media-stream WS at the configured cadence. | +| `crates/rutster-sim/src/nodelay.rs` | sim-bench latency assertion: p99 send→recv of 20 ms-cadence WS frames through the production serve path ≤ threshold. | +| `crates/rutster-trunk/src/public_url.rs` | `reconstruct_public_url` — trusted-proxy-gated `X-Forwarded-Proto/Host` public-URL reconstruction (spec §3.1 invariant 5), exported for the trunk slice's future signature validation. | + +### Modified files + +| Path | What changes | +|---|---| +| `Cargo.toml` (root) | Add `ipnet = "2"` to `[workspace.dependencies]`. | +| `crates/rutster/src/lib.rs` | Add `pub mod serve;`. | +| `crates/rutster/src/main.rs` | Serve via `serve_with_nodelay`; parse `RUTSTER_WS_PING_SECS`, `RUTSTER_TWILIO_*` (startup validation — currently parsed-but-never-called), `RUTSTER_TRUSTED_PROXIES`; thread webhook base + trusted proxies into `AppState`. | +| `crates/rutster/src/config.rs` | Add `ws_ping_interval` + `trusted_proxies` parsers (house fail-fast pattern). | +| `crates/rutster/src/routes.rs` | `twilio_inbound_webhook` derives the TwiML Stream URL from the configured webhook base — kills the hardcoded `ws://127.0.0.1:8080` at line 265; 503 when trunk unconfigured. | +| `crates/rutster/src/session_map.rs` | `AppState` gains `trunk_webhook_base: Option` + `trusted_proxies: Vec` fields + builder methods. | +| `crates/rutster/Cargo.toml` | Add `ipnet.workspace = true`. | +| `crates/rutster-trunk/src/twilio_media_streams.rs` | Router state grows a `ping_interval`; the pump `select!` gains a ping branch. | +| `crates/rutster-trunk/src/lib.rs` | Add `pub mod public_url;`. | +| `crates/rutster-trunk/Cargo.toml` | Add `ipnet.workspace = true`; add `tokio` with `net` to dev-deps for the ping test. | +| `crates/rutster-sim/src/lib.rs` | Declare `nodelay` module; re-export the new threshold const. | +| `crates/rutster-sim/src/thresholds.rs` | Add `WS_FRAME_SEND_TO_RECV_P99_MS` const. | +| `crates/rutster-sim/Cargo.toml` | Add dev-deps for the nodelay test (rutster-trunk, rutster-call-model, tokio-tungstenite, futures-util, tokio/net). | + +### SEAM-INVARIANT files (DO NOT TOUCH) + +- `crates/rutster-media/src/loop_driver.rs` — byte-identical, all tasks. +- `crates/rutster-media/src/rtc_session.rs` — byte-identical, all tasks. + +## Task ordering + +- **Task 1** — `serve_with_nodelay` + main.rs wiring. Foundation: Task 3 consumes the helper. +- **Task 2** — WS ping (config parser + pump branch + router signature). Task 3's client must + skip Ping frames, so this lands first. +- **Task 3** — sim-bench NODELAY latency assertion. Depends on Tasks 1 + 2. +- **Task 4** — TwiML Stream URL derivation + `twilio_credentials` startup wiring. Independent of + 1–3; can run in parallel with Task 3. +- **Task 5** — trusted-proxy posture: `RUTSTER_TRUSTED_PROXIES` + `reconstruct_public_url`. + Independent of 1–3; touches `main.rs`/`session_map.rs` after Task 4 (sequential with 4). +- **Task 6** — final verification sweep (no code). + +--- + +### Task 1: TCP_NODELAY on every accepted socket — the `serve` helper + +The locked axum 0.7.9 already carries the fix for its own #2521: `Serve::tcp_nodelay(true)` +sets the option on every accepted connection and the flag survives +`.with_graceful_shutdown(...)` (verified in the vendored `axum-0.7.9/src/serve.rs`). No manual +accept loop, no hyper-util dep. The only structural work is extracting the serve call into the +library so the sim-bench assertion (Task 3) exercises the *production* path — if a future edit +drops the flag from the helper, CI goes red. + +**Files:** +- Create: `crates/rutster/src/serve.rs` +- Create: `crates/rutster/tests/serve_nodelay.rs` +- Modify: `crates/rutster/src/lib.rs` (module list, currently lines 25–31) +- Modify: `crates/rutster/src/main.rs` (the `axum::serve` block, currently lines 136–141) + +**Interfaces:** +- Consumes: `axum::serve` (0.7.9), `tokio::net::TcpListener`, `axum::Router`. +- Produces: + `pub async fn serve_with_nodelay(listener: tokio::net::TcpListener, app: axum::Router, shutdown: F) -> std::io::Result<()> where F: Future + Send + 'static` + +- [ ] **Step 1: Write the failing integration test** + +`crates/rutster/tests/serve_nodelay.rs` (complete file): + +```rust +//! Integration test for `rutster::serve::serve_with_nodelay` (deploy +//! slice A §5.1): the production serve helper round-trips a request over +//! a real socket and quiesces when the shutdown future resolves. +//! +//! The NODELAY *behavior* (no Nagle stall on 20 ms-cadence WS frames) is +//! asserted in the CI-regressed sim-bench +//! (`crates/rutster-sim/src/nodelay.rs`, Task 3) — this test pins the +//! helper's serve/shutdown contract only. + +use tokio::io::{AsyncReadExt, AsyncWriteExt}; + +#[tokio::test] +async fn serves_healthz_and_stops_on_shutdown_signal() { + let app = rutster::routes::router(rutster::session_map::AppState::default()); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let (stop_tx, stop_rx) = tokio::sync::oneshot::channel::<()>(); + let server = tokio::spawn(rutster::serve::serve_with_nodelay(listener, app, async { + let _ = stop_rx.await; + })); + + // Raw HTTP/1.1 over a plain TcpStream — no client dep needed for a + // liveness round trip. `Connection: close` so read_to_end terminates. + let mut stream = tokio::net::TcpStream::connect(addr).await.unwrap(); + stream + .write_all(b"GET /healthz HTTP/1.1\r\nHost: test\r\nConnection: close\r\n\r\n") + .await + .unwrap(); + let mut buf = Vec::new(); + stream.read_to_end(&mut buf).await.unwrap(); + let response = String::from_utf8_lossy(&buf); + assert!(response.starts_with("HTTP/1.1 200"), "got: {response}"); + + stop_tx.send(()).unwrap(); + tokio::time::timeout(std::time::Duration::from_secs(5), server) + .await + .expect("serve future resolves after the shutdown signal") + .expect("serve task did not panic") + .expect("serve returned Ok"); +} +``` + +- [ ] **Step 2: Run it — expect a compile failure** + +```bash +cargo test -p rutster --test serve_nodelay +``` + +Expected: compile error `E0433`-class — `could not find `serve` in `rutster`` (the module does +not exist yet). + +- [ ] **Step 3: Implement the helper** + +`crates/rutster/src/serve.rs` (complete file): + +```rust +//! # serve — the production HTTP/WS listener path (deploy slice A §5.1) +//! +//! One wrapper over `axum::serve` so `main.rs` and the sim-bench latency +//! assertion (`crates/rutster-sim/src/nodelay.rs`) exercise the SAME +//! serve path. The single behavioral addition over plain `axum::serve`: +//! **TCP_NODELAY on every accepted socket.** +//! +//! ## Why TCP_NODELAY (a piece of TCP history worth teaching) +//! +//! Nagle's algorithm (RFC 896, 1984) holds back small writes while a +//! previously-sent segment is still unacknowledged — great for 1984 +//! telnet, poison for real-time media. It interacts with the peer's +//! delayed-ACK timer: a stream of sub-MSS WS frames at fixed 20 ms +//! cadence (exactly the trunk media stream) degenerates into bursts with +//! up-to-40 ms stalls (axum #2521 / #1961). This listener serves such +//! frames for hours per call behind *any* proxy, so Nagle is pure harm +//! here and the option is unconditional — no env knob. +//! +//! axum 0.7.9's `Serve::tcp_nodelay(true)` applies the option to every +//! accepted connection, and the flag is carried through +//! `with_graceful_shutdown` (verified against the locked version). + +use std::future::Future; + +use axum::Router; +use tokio::net::TcpListener; + +/// Serve `app` on `listener` with TCP_NODELAY set on every accepted +/// socket, shutting down gracefully once `shutdown` resolves (stop +/// accepting, let in-flight connections finish — same semantics `main.rs` +/// already relied on with plain `axum::serve`). +pub async fn serve_with_nodelay( + listener: TcpListener, + app: Router, + shutdown: F, +) -> std::io::Result<()> +where + F: Future + Send + 'static, +{ + axum::serve(listener, app) + .tcp_nodelay(true) + .with_graceful_shutdown(shutdown) + .await +} +``` + +Add the module to `crates/rutster/src/lib.rs` — the declaration list becomes: + +```rust +pub mod config; +pub mod event_sink; +pub mod media_thread; +pub mod routes; +pub mod serve; +pub mod session_map; +pub mod tap_engine; +pub mod tool_registry; +``` + +- [ ] **Step 4: Wire `main.rs` through the helper** + +In `crates/rutster/src/main.rs`, replace (current lines 136–141): + +```rust + axum::serve(listener, app) + .with_graceful_shutdown(async { + let _ = http_stop_rx.await; + }) + .await + .unwrap(); +``` + +with: + +```rust + // Deploy slice A §5.1: TCP_NODELAY on every accepted socket, via the + // shared serve helper so the sim-bench latency assertion regresses + // the SAME path this binary runs (crates/rutster-sim/src/nodelay.rs). + rutster::serve::serve_with_nodelay(listener, app, async { + let _ = http_stop_rx.await; + }) + .await + .unwrap(); +``` + +- [ ] **Step 5: Run tests — expect pass** + +```bash +cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings +cargo test -p rutster --test serve_nodelay +cargo test --all +``` + +Expected: all green. + +- [ ] **Step 6: Commit** + +```bash +git add crates/rutster/src/serve.rs crates/rutster/tests/serve_nodelay.rs crates/rutster/src/lib.rs crates/rutster/src/main.rs +git commit -s -m "feat(serve): TCP_NODELAY on every accepted HTTP/WS socket (deploy-A §5.1) + +Nagle + the peer's delayed-ACK timer turns sub-MSS WS frames at 20ms +cadence into bursts with up-to-40ms stalls (axum #2521) — on the +plaintext :8080 listener behind ANY proxy. axum 0.7.9 (locked) carries +Serve::tcp_nodelay; the new rutster::serve::serve_with_nodelay wrapper +is the single production serve path so the sim-bench latency assertion +(Task 3) regresses exactly what main.rs runs. + +Seam gate untouched: this is the HTTP/WS listener, not the RTP path." +``` + +--- + +### Task 2: App-level WS ping on the trunk media-stream WS (`RUTSTER_WS_PING_SECS`) + +The trunk media-stream WS is the only long-lived WS the engine serves today (WebRTC signaling +is plain HTTP). Neither Twilio nor Telnyx documents any WS keepalive of their own — keepalive +is entirely our job (spec §3.1 invariant 3), and the caller→engine direction can legitimately +be the only traffic for hours, letting every idle timer between the FOB and the CPaaS kill the +socket. Engine-originated Ping frames, interval `RUTSTER_WS_PING_SECS`, default 20. + +**Files:** +- Modify: `crates/rutster/src/config.rs` (new parser; tests in the existing `mod tests`) +- Modify: `crates/rutster-trunk/src/twilio_media_streams.rs` (router state + pump branch; the + in-module test `router_constructs_with_register_channel` at lines 342–346 gains the new arg) +- Modify: `crates/rutster/src/main.rs` (parse env, pass to the router call at lines 131–132) +- Modify: `crates/rutster-trunk/Cargo.toml` (dev-deps: `tokio` with `net` for the test listener) +- Create: `crates/rutster-trunk/tests/ws_ping.rs` + +**Interfaces:** +- Consumes: the existing pump loop's `tokio::select!`, `axum::extract::ws::Message::Ping`. +- Produces: + - `pub fn config::ws_ping_interval(raw: Option) -> Result` (default 20 s; rejects 0 and garbage) + - `TwilioMediaStreamsServer::router(register_tx: mpsc::Sender, ping_interval: Duration) -> Router` (signature change; both call sites updated) + +- [ ] **Step 1: Write the failing config-parser tests** + +Append to the `mod tests` block in `crates/rutster/src/config.rs`: + +```rust + #[test] + fn ws_ping_interval_defaults_to_20s_when_unset() { + assert_eq!( + ws_ping_interval(None).unwrap(), + std::time::Duration::from_secs(20) + ); + } + + #[test] + fn ws_ping_interval_parses_override() { + assert_eq!( + ws_ping_interval(Some("5".into())).unwrap(), + std::time::Duration::from_secs(5) + ); + } + + #[test] + fn ws_ping_interval_rejects_zero() { + let err = ws_ping_interval(Some("0".into())).unwrap_err(); + assert!(err.contains("RUTSTER_WS_PING_SECS")); + } + + #[test] + fn ws_ping_interval_rejects_garbage_with_var_name_in_error() { + let err = ws_ping_interval(Some("often".into())).unwrap_err(); + assert!(err.contains("RUTSTER_WS_PING_SECS")); + } +``` + +- [ ] **Step 2: Run — expect compile failure** + +```bash +cargo test -p rutster config::tests +``` + +Expected: compile error — `cannot find function `ws_ping_interval` in this scope`. + +- [ ] **Step 3: Implement the parser** + +Add to `crates/rutster/src/config.rs` (after `drain_deadline`, before `twilio_credentials` — +same pure-`Option` convention as the whole module): + +```rust +/// `RUTSTER_WS_PING_SECS` — engine-originated app-level WS ping interval +/// on long-lived WebSockets. Today that is exactly one surface: the trunk +/// media-stream WS (WebRTC signaling is plain HTTP). Default 20 s. +/// +/// Belt-and-braces against every idle timer between the FOB and the +/// CPaaS: neither Twilio nor Telnyx documents WS keepalive of their own, +/// so keepalive is entirely our job (deploy spec §3.1 invariant 3), and +/// the caller→engine direction may be the only traffic for hours. +/// +/// `0` is rejected fail-fast: it would busy-loop the ping timer. There is +/// deliberately no "off" spelling — an operator who wants no pings is an +/// operator about to learn about 60 s proxy idle defaults mid-call. +pub fn ws_ping_interval(raw: Option) -> Result { + match raw { + None => Ok(std::time::Duration::from_secs(20)), + Some(s) => { + let secs: u64 = s + .parse() + .map_err(|e| format!("RUTSTER_WS_PING_SECS {s:?}: {e}"))?; + if secs == 0 { + return Err( + "RUTSTER_WS_PING_SECS must be >= 1 (0 would busy-loop the ping timer)" + .to_string(), + ); + } + Ok(std::time::Duration::from_secs(secs)) + } + } +} +``` + +Run `cargo test -p rutster config::tests` — expect pass. + +- [ ] **Step 4: Write the failing WS-ping integration test** + +`crates/rutster-trunk/tests/ws_ping.rs` (complete file): + +```rust +//! App-level WS ping emission on the trunk media-stream WS (deploy +//! slice A §5.2). +//! +//! Spins the real router on an ephemeral listener, performs the Twilio +//! connected/start handshake as a tokio-tungstenite client, then asserts +//! an engine-originated Ping frame arrives well within CI slack of the +//! configured interval. Twilio documents no keepalive of its own — this +//! test is the contract that WE originate the pings. + +use futures_util::{SinkExt, StreamExt}; +use rutster_trunk::twilio_media_streams::{ + RegisterTrunkInboundChannel, TwilioMediaStreamsServer, +}; +use tokio::sync::mpsc; +use tokio_tungstenite::tungstenite::Message; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn engine_originates_ws_ping_at_configured_interval() { + let (register_tx, mut register_rx) = mpsc::channel::(4); + let app = + TwilioMediaStreamsServer::router(register_tx, std::time::Duration::from_millis(100)); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + + // Stub media thread: ack the registration and hold the channel ends + // alive so the pump loop stays up for the whole test. + let registered = tokio::spawn(async move { + let register = register_rx.recv().await.expect("pump registers"); + let _ = register.reply.send(rutster_call_model::ChannelId::new()); + register + }); + + let (mut ws, _resp) = + tokio_tungstenite::connect_async(format!("ws://{addr}/twilio/media-stream")) + .await + .expect("WS connect"); + ws.send(Message::Text( + r#"{"event":"connected","protocol":"twilio-media-stream","version":"1.0.0"}"#.into(), + )) + .await + .unwrap(); + ws.send(Message::Text( + r#"{"event":"start","start":{"streamSid":"MZping","callSid":"CAping"}}"#.into(), + )) + .await + .unwrap(); + let _register = registered.await.unwrap(); + + // 100 ms interval → a Ping must land within 2 s (20 intervals of + // slack for a loaded CI runner; the point is emission, not cadence + // precision). + let got_ping = tokio::time::timeout(std::time::Duration::from_secs(2), async { + while let Some(msg) = ws.next().await { + if matches!(msg.expect("WS frame"), Message::Ping(_)) { + return true; + } + } + false + }) + .await + .expect("a Ping frame must arrive before the deadline"); + assert!(got_ping, "expected an engine-originated Ping frame"); +} +``` + +Add to `crates/rutster-trunk/Cargo.toml` `[dev-dependencies]` (the test binds a real listener; +the crate's own `tokio` feature list has no `net`, though `net` IS already available via the +workspace `tokio = { features = ["full"] }` — this dev-dep makes the test's `net` dependency +explicit at the crate level rather than inheriting it through workspace feature unification): + +```toml +tokio = { workspace = true, features = ["macros", "rt-multi-thread", "sync", "time", "net"] } +``` + +- [ ] **Step 5: Run — expect compile failure** + +```bash +cargo test -p rutster-trunk --test ws_ping +``` + +Expected: compile error — `router` takes 1 argument but 2 were supplied (the signature change +does not exist yet). + +- [ ] **Step 6: Implement the ping branch + router signature** + +In `crates/rutster-trunk/src/twilio_media_streams.rs`: + +(a) Add `Duration` to the imports (top of file): + +```rust +use std::time::Duration; +``` + +(b) Replace the router/state plumbing (currently lines 31–38 + the handler at 75–80): + +```rust +/// Router state: the media-thread register channel + the app-level +/// keepalive interval (`RUTSTER_WS_PING_SECS`, deploy slice A §5.2). +#[derive(Clone)] +struct MediaStreamState { + register_tx: mpsc::Sender, + ping_interval: Duration, +} + +pub struct TwilioMediaStreamsServer; + +impl TwilioMediaStreamsServer { + pub fn router( + register_tx: mpsc::Sender, + ping_interval: Duration, + ) -> Router { + Router::new() + .route("/twilio/media-stream", get(handle_media_stream)) + .with_state(MediaStreamState { + register_tx, + ping_interval, + }) + } +} +``` + +and: + +```rust +async fn handle_media_stream( + ws: WebSocketUpgrade, + State(state): State, +) -> impl IntoResponse { + ws.on_upgrade(move |socket| { + run_media_stream(socket, state.register_tx, state.ping_interval) + }) +} +``` + +(c) Thread the interval into the pump. Change the signature of `run_media_stream` +(currently line 82): + +```rust +async fn run_media_stream( + mut socket: WebSocket, + register_tx: mpsc::Sender, + ping_interval: Duration, +) { +``` + +(d) Immediately before the `loop {` (after the `info!(%channel_id, ...)` registration log, +currently line 151), create the timer: + +```rust + // App-level keepalive (deploy slice A §5.2): engine-originated pings + // at RUTSTER_WS_PING_SECS cadence. `interval_at` skips the immediate + // first tick (a fresh call doesn't need a ping at t=0); + // MissedTickBehavior::Delay because a late ping must not be followed + // by a compensating burst. The pre-registration handshake recvs above + // are un-pinged — Twilio sends connected+start immediately on + // connect, so that window is sub-second by protocol. + let mut ping = tokio::time::interval_at( + tokio::time::Instant::now() + ping_interval, + ping_interval, + ); + ping.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); +``` + +(e) Add a third branch to the `tokio::select!` (after the `frame = outbound_rx.recv()` arm, +before the closing brace of the select at line 223): + +```rust + _ = ping.tick() => { + // Vec::new(): an empty application payload — the frame + // itself is the keepalive; peers must Pong per RFC 6455. + if socket.send(Message::Ping(Vec::new())).await.is_err() { + warn!(%call_sid, "twilio WS ping send failed; ending pump"); + break; + } + }, +``` + +(f) Update the in-module router test (currently lines 342–346): + +```rust + #[tokio::test] + async fn router_constructs_with_register_channel() { + let (register_tx, _register_rx) = mpsc::channel::(4); + let _app = TwilioMediaStreamsServer::router(register_tx, Duration::from_secs(20)); + } +``` + +- [ ] **Step 7: Wire `main.rs`** + +In `crates/rutster/src/main.rs`, replace the router construction (currently lines 131–132): + +```rust + let trunk_router = + rutster_trunk::twilio_media_streams::TwilioMediaStreamsServer::router(trunk_register_tx); +``` + +with: + +```rust + // Deploy slice A §5.2: app-level keepalive on the only long-lived WS + // surface. Fail-fast on garbage, like every RUTSTER_* knob. + let ws_ping_interval = + rutster::config::ws_ping_interval(std::env::var("RUTSTER_WS_PING_SECS").ok()) + .expect("RUTSTER_WS_PING_SECS must be a positive integer (seconds)"); + let trunk_router = rutster_trunk::twilio_media_streams::TwilioMediaStreamsServer::router( + trunk_register_tx, + ws_ping_interval, + ); +``` + +- [ ] **Step 8: Run tests — expect pass** + +```bash +cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings +cargo test -p rutster-trunk --test ws_ping +cargo test --all +``` + +Expected: all green (including the pre-existing trunk tests — the pump's behavior is additive). + +- [ ] **Step 9: Commit** + +```bash +git add crates/rutster/src/config.rs crates/rutster/src/main.rs crates/rutster-trunk/src/twilio_media_streams.rs crates/rutster-trunk/Cargo.toml crates/rutster-trunk/tests/ws_ping.rs +git commit -s -m "feat(trunk): app-level WS ping on the media-stream WS, RUTSTER_WS_PING_SECS default 20 (deploy-A §5.2) + +Neither Twilio nor Telnyx documents WS keepalive — keepalive is entirely +our job (spec §3.1 invariant 3), and the caller→engine direction can be +the only traffic for hours, letting any 60s proxy idle timer kill a live +call. Engine-originated Ping frames as a third select! branch in the +existing pump loop; interval via the config.rs fail-fast parser pattern +(0 rejected — no 'off' spelling by design). + +Router signature grows the interval; both call sites updated. WebRTC +signaling is plain HTTP today, so this is the only long-lived WS." +``` + +--- + +### Task 3: NODELAY latency assertion in the CI-regressed sim-bench + +The S1–S8 sweep asserts tick-side latency; nothing regresses the *listener*. This test drives +the real trunk WS route through `serve_with_nodelay` over a real loopback socket at the real +20 ms cadence and asserts p99 send→recv per frame. With NODELAY the path measures ~1–2 ms; a +Nagle regression stalls frames ~40 ms+ (Nagle holds frame N+1 until frame N's ACK; the client +kernel delays that ACK) — 20 ms separates the two regimes with an order of magnitude on each +side, the same deterministic-but-not-flaky posture as the existing consts. + +**Files:** +- Modify: `crates/rutster-sim/src/thresholds.rs` (new const, alongside the existing ones) +- Modify: `crates/rutster-sim/src/lib.rs` (module declaration + re-export lists, currently + lines 53–70) +- Modify: `crates/rutster-sim/Cargo.toml` (dev-deps) +- Create: `crates/rutster-sim/src/nodelay.rs` + +**Interfaces:** +- Consumes: `rutster::serve::serve_with_nodelay` (Task 1), + `TwilioMediaStreamsServer::router(_, Duration)` (Task 2), `rutster_media::PcmFrame`, + `rutster_call_model::ChannelId::new()`. +- Produces: `pub const WS_FRAME_SEND_TO_RECV_P99_MS: f64 = 20.0;` + one + `#[cfg(all(test, feature = "sim-bench"))]` test in the S7 style. + +- [ ] **Step 1: Add the threshold const** + +Append to the const block in `crates/rutster-sim/src/thresholds.rs` (after +`SWEEP_CONCURRENCIES`, before the `bench_assertions` module): + +```rust +/// Deploy slice A §5.1: the TCP_NODELAY regression tripwire. p99 +/// send→recv latency for small server-originated WS frames at the trunk's +/// 20 ms cadence, over loopback, through the PRODUCTION serve path +/// (`rutster::serve::serve_with_nodelay` — the exact call `main.rs` +/// makes). Healthy: ~1–2 ms. A Nagle regression (axum #2521 class — +/// e.g. someone drops `.tcp_nodelay(true)` from the helper) interacts +/// with the peer's delayed-ACK timer and stalls frames ~40 ms+. 20 ms +/// sits an order of magnitude above healthy and comfortably below the +/// failure mode — deterministic-but-not-flaky on a slow CI runner, same +/// slack posture as the consts above. +pub const WS_FRAME_SEND_TO_RECV_P99_MS: f64 = 20.0; +``` + +- [ ] **Step 2: Declare the module + re-export** + +In `crates/rutster-sim/src/lib.rs`, the module list (currently lines 53–60) becomes: + +```rust +pub mod concurrency; +pub mod latency; +#[cfg(all(test, feature = "sim-bench"))] +mod nodelay; +pub mod runner; +pub mod scenario; +pub mod sim_audio_pipe; +pub mod thresholds; +pub mod tick_lag; +``` + +and the `thresholds` re-export (currently lines 66–69) becomes: + +```rust +pub use thresholds::{ + BARGE_IN_KILL_TIME_P99_MS, MOUTH_TO_EAR_P99_MS, SWEEP_CONCURRENCIES, TICK_LAG_MAX_MS, + TICK_OVERRUN_PCT_MAX, WS_FRAME_SEND_TO_RECV_P99_MS, +}; +``` + +- [ ] **Step 3: Add the dev-deps** + +Append to `crates/rutster-sim/Cargo.toml`: + +```toml +[dev-dependencies] +# nodelay.rs (deploy slice A §5.1): drives the real trunk WS route through +# the production serve path over a real loopback socket. Test-only — the +# sim's own harness stays mpsc-pure. +rutster-trunk = { path = "../rutster-trunk" } +rutster-call-model = { path = "../rutster-call-model" } +tokio = { workspace = true, features = ["macros", "rt-multi-thread", "sync", "time", "net"] } +tokio-tungstenite = { workspace = true } +futures-util = { workspace = true } +``` + +- [ ] **Step 4: Write the assertion (failing-by-construction until it passes against Task 1's helper)** + +`crates/rutster-sim/src/nodelay.rs` (complete file): + +```rust +//! # nodelay — TCP_NODELAY latency assertion (deploy slice A §5.1) +//! +//! The S1–S8 sweep regresses tick-side latency; this test regresses the +//! LISTENER: server-originated WS frames at the trunk's 20 ms cadence, +//! through `rutster::serve::serve_with_nodelay` (the production serve +//! path) over a real loopback TCP socket, measured send→recv per frame. +//! +//! Runs ONLY under `--features=sim-bench` (the CI-regressed job, +//! `--test-threads=1`). A regression here means Nagle is back on the +//! accepted sockets (axum #2521 class) and every proxied trunk call +//! would jitter by the peer's delayed-ACK timer. + +use std::time::{Duration, Instant}; + +use futures_util::{SinkExt, StreamExt}; +use rutster_media::PcmFrame; +use rutster_trunk::twilio_media_streams::{ + RegisterTrunkInboundChannel, TwilioMediaStreamsServer, +}; +use tokio::sync::mpsc; +use tokio_tungstenite::tungstenite::Message; + +use crate::thresholds::WS_FRAME_SEND_TO_RECV_P99_MS; + +/// 200 frames × 20 ms ≈ 4 s of wall clock — enough samples for a +/// meaningful p99 (index 197) without dominating the sim-bench job. +const FRAMES: usize = 200; +const CADENCE: Duration = Duration::from_millis(20); + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn ws_frame_send_to_recv_p99_stays_under_nodelay_threshold() { + // 1. Real listener + the PRODUCTION serve path (Task 1's helper). + let (register_tx, mut register_rx) = mpsc::channel::(4); + let app = TwilioMediaStreamsServer::router(register_tx, Duration::from_secs(20)); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let (_stop_tx, stop_rx) = tokio::sync::oneshot::channel::<()>(); + tokio::spawn(rutster::serve::serve_with_nodelay(listener, app, async { + let _ = stop_rx.await; + })); + + // 2. Stub media thread: ack the registration; keep the channel ends. + let register_task = tokio::spawn(async move { + let register = register_rx.recv().await.expect("pump registers"); + let _ = register.reply.send(rutster_call_model::ChannelId::new()); + register + }); + + // 3. Twilio-side client: connected + start handshake. + let (mut ws, _resp) = + tokio_tungstenite::connect_async(format!("ws://{addr}/twilio/media-stream")) + .await + .expect("WS connect"); + ws.send(Message::Text( + r#"{"event":"connected","protocol":"twilio-media-stream","version":"1.0.0"}"#.into(), + )) + .await + .unwrap(); + ws.send(Message::Text( + r#"{"event":"start","start":{"streamSid":"MZsim","callSid":"CAsim"}}"#.into(), + )) + .await + .unwrap(); + let register = register_task.await.expect("register task"); + + // 4. Server-originated frames at the trunk cadence. mpsc, the pump + // loop, and the WS all preserve order, so sent frame k IS received + // frame k — instants pair by index; no payload tagging needed + // (µ-law would quantize a tag anyway). + let outbound = register.outbound_to_twilio_tx.clone(); + let sender = tokio::spawn(async move { + let mut send_instants = Vec::with_capacity(FRAMES); + let mut cadence = tokio::time::interval(CADENCE); + cadence.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + for _ in 0..FRAMES { + cadence.tick().await; + send_instants.push(Instant::now()); + outbound + .send(PcmFrame::zeroed()) + .await + .expect("pump alive for the whole run"); + } + send_instants + }); + + let mut recv_instants = Vec::with_capacity(FRAMES); + while recv_instants.len() < FRAMES { + let msg = tokio::time::timeout(Duration::from_secs(10), ws.next()) + .await + .expect("frame within 10s") + .expect("WS stream open") + .expect("WS frame ok"); + // Only Text frames are media envelopes; skip the §5.2 keepalive + // Ping (and anything else) so pairing stays index-aligned. + if matches!(msg, Message::Text(_)) { + recv_instants.push(Instant::now()); + } + } + let send_instants = sender.await.expect("sender task"); + + let mut latencies_ms: Vec = send_instants + .iter() + .zip(&recv_instants) + .map(|(s, r)| r.duration_since(*s).as_secs_f64() * 1000.0) + .collect(); + latencies_ms.sort_by(|a, b| a.partial_cmp(b).expect("no NaN latencies")); + let p99 = latencies_ms[(latencies_ms.len() * 99).div_ceil(100) - 1]; + assert!( + p99 <= WS_FRAME_SEND_TO_RECV_P99_MS, + "p99 WS frame send→recv {p99:.2}ms > {WS_FRAME_SEND_TO_RECV_P99_MS}ms \ + (TCP_NODELAY regression: Nagle/delayed-ACK is stalling the 20ms \ + trunk cadence; axum #2521 class; deploy spec §5.1)" + ); +} +``` + +- [ ] **Step 5: Prove the assertion catches the regression, then run green** + +First, temporarily flip the helper to `tcp_nodelay(false)` in `crates/rutster/src/serve.rs` and +run: + +```bash +cargo test -p rutster-sim --features=sim-bench nodelay -- --test-threads=1 +``` + +Expected: **FAIL** with a p99 in the tens of ms (`p99 WS frame send→recv ... > 20ms`). If it +passes with Nagle on, STOP — the assertion is not load-bearing; investigate before proceeding +(loopback MTU/quickack behavior varies; the fix is more/smaller frames, not a looser +threshold). Then revert to `tcp_nodelay(true)` and re-run: + +```bash +cargo test -p rutster-sim --features=sim-bench nodelay -- --test-threads=1 +``` + +Expected: pass, p99 in low single-digit ms. + +- [ ] **Step 6: Full gates + commit** + +```bash +cargo fmt --all --check +cargo clippy --all --all-targets --features=sim-bench -- -D warnings +cargo test --all +cargo test --all --features=sim-bench -- --test-threads=1 +git add crates/rutster-sim/src/thresholds.rs crates/rutster-sim/src/nodelay.rs crates/rutster-sim/src/lib.rs crates/rutster-sim/Cargo.toml +git commit -s -m "test(sim): WS-frame send→recv p99 assertion — the TCP_NODELAY tripwire (deploy-A §5.1) + +Joins the CI-regressed sim-bench sweep (S7 style: threshold const with +budget+slack rationale, cfg(sim-bench) test, --test-threads=1 job). +Drives the real trunk WS route through the PRODUCTION serve path +(rutster::serve::serve_with_nodelay) over a real loopback socket at the +real 20ms cadence. Healthy ~1-2ms; a Nagle regression stalls ~40ms+; +threshold 20ms splits the regimes by an order of magnitude each way. +Verified load-bearing: fails when tcp_nodelay(false)." +``` + +--- + +### Task 4: TwiML Stream URL from `RUTSTER_TWILIO_WEBHOOK_BASE` + wire `config::twilio_credentials` into startup + +Two halves of the same fix. `config::twilio_credentials` (crates/rutster/src/config.rs:119) has +been parsed-but-never-called since slice-5 — an operator with partial/malformed +`RUTSTER_TWILIO_*` config today boots silently WebRTC-only. And the webhook handler +(crates/rutster/src/routes.rs:261) answers every inbound call with a hardcoded +`ws://127.0.0.1:8080/twilio/media-stream` Stream URL (line 265) that no CPaaS can ever dial. + +**Files:** +- Modify: `crates/rutster/src/session_map.rs` (`AppState` field + builder) +- Modify: `crates/rutster/src/routes.rs` (derivation fn + rewired handler + tests) +- Modify: `crates/rutster/src/main.rs` (startup validation + threading) + +**Interfaces:** +- Consumes: `config::twilio_credentials` (exists, verified — returns + `Result, String>`; errors on partial config), `TwilioCredentials + { webhook_base: url::Url, .. }` (crates/rutster-trunk/src/provider/mod.rs:143). +- Produces: + - `AppState.trunk_webhook_base: Option` + `AppState::with_trunk_webhook_base(self, Option) -> Self` + - `fn stream_url_from_webhook_base(base: &url::Url) -> Result` (private to `routes.rs`) + - `POST /v1/trunk/webhook` → 200 TwiML with derived `wss://.../twilio/media-stream` when configured; 503 when not. + +- [ ] **Step 1: Write the failing tests** + +Append to `mod tests` in `crates/rutster/src/routes.rs`: + +```rust + #[test] + fn stream_url_derivation_maps_https_to_wss_and_keeps_authority() { + let base = url::Url::parse("https://pbx.example.com:8443").unwrap(); + let u = stream_url_from_webhook_base(&base).unwrap(); + assert_eq!(u.as_str(), "wss://pbx.example.com:8443/twilio/media-stream"); + } + + #[test] + fn stream_url_derivation_maps_http_to_ws_for_dev() { + let base = url::Url::parse("http://localhost:8080").unwrap(); + let u = stream_url_from_webhook_base(&base).unwrap(); + assert_eq!(u.as_str(), "ws://localhost:8080/twilio/media-stream"); + } + + #[test] + fn stream_url_derivation_rejects_non_http_schemes() { + let base = url::Url::parse("ftp://pbx.example.com").unwrap(); + let err = stream_url_from_webhook_base(&base).unwrap_err(); + assert!(err.contains("http"), "msg: {err}"); + } + + #[tokio::test] + async fn webhook_derives_stream_url_from_configured_base() { + let state = AppState::default().with_trunk_webhook_base(Some( + url::Url::parse("https://pbx.example.com").unwrap(), + )); + let app = router(state); + let resp = app + .oneshot( + axum::http::Request::builder() + .method("POST") + .uri("/v1/trunk/webhook") + .body(axum::body::Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(resp.status(), StatusCode::OK); + let body = axum::body::to_bytes(resp.into_body(), 4096).await.unwrap(); + let twiml = String::from_utf8(body.to_vec()).unwrap(); + assert!( + twiml.contains(r#""#), + "twiml: {twiml}" + ); + assert!( + !twiml.contains("127.0.0.1"), + "hardcoded loopback must be gone: {twiml}" + ); + } + + #[tokio::test] + async fn webhook_503_when_trunk_unconfigured() { + // Default AppState: trunk_webhook_base = None (no RUTSTER_TWILIO_*). + let app = router(AppState::default()); + let resp = app + .oneshot( + axum::http::Request::builder() + .method("POST") + .uri("/v1/trunk/webhook") + .body(axum::body::Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE); + } +``` + +- [ ] **Step 2: Run — expect compile failure** + +```bash +cargo test -p rutster routes::tests +``` + +Expected: compile errors — `cannot find function `stream_url_from_webhook_base`` and no method +`with_trunk_webhook_base` on `AppState`. + +- [ ] **Step 3: `AppState` field + builder** + +In `crates/rutster/src/session_map.rs`, the struct (currently lines 23–27) becomes: + +```rust +#[derive(Clone)] +pub struct AppState { + pub cmd_tx: mpsc::Sender, + pub default_tap_url: url::Url, + /// Public base URL the CPaaS calls back on + /// (`RUTSTER_TWILIO_WEBHOOK_BASE`). `None` = trunk unconfigured — the + /// webhook answers 503 instead of minting a TwiML Stream URL nobody + /// can dial (deploy slice A §5.3). + pub trunk_webhook_base: Option, +} +``` + +`new` keeps its two-argument signature (many call sites); the field defaults to `None` and a +builder sets it — replace the `new` body (currently lines 36–41) and add the builder: + +```rust + pub fn new(cmd_tx: mpsc::Sender, default_tap_url: url::Url) -> Self { + Self { + cmd_tx, + default_tap_url, + trunk_webhook_base: None, + } + } + + /// Thread the operator's public webhook base (from + /// `config::twilio_credentials`) into the routes. Builder-style so the + /// two-arg `new` keeps its many existing call sites. + pub fn with_trunk_webhook_base(mut self, base: Option) -> Self { + self.trunk_webhook_base = base; + self + } +``` + +(`Default` at lines 123–136 goes through `Self::new`, so it needs no change.) + +- [ ] **Step 4: Derivation fn + rewired handler** + +In `crates/rutster/src/routes.rs`, replace the stub handler `twilio_inbound_webhook` +(currently lines 245–274) with: + +```rust +/// Derive the TwiML `` URL from the operator's public webhook +/// base (`RUTSTER_TWILIO_WEBHOOK_BASE`): `https://…` → `wss://…`, +/// `http://…` → `ws://…` (dev loop), path fixed to the WS route's mount +/// point. +/// +/// Authority-only by design: any path on the base is dropped, because +/// `TwilioMediaStreamsServer::router` mounts the WS route absolutely at +/// `/twilio/media-stream` — a path-prefixing edge must strip its prefix +/// before the FOB, exactly like it must for every other route. +fn stream_url_from_webhook_base(base: &url::Url) -> Result { + let ws_scheme = match base.scheme() { + "https" => "wss", + "http" => "ws", + other => { + return Err(format!( + "webhook base must be http(s); got scheme {other:?}" + )); + } + }; + let mut stream_url = base + .join("/twilio/media-stream") + .map_err(|e| format!("webhook base rejects a path join: {e}"))?; + stream_url + .set_scheme(ws_scheme) + .map_err(|()| format!("cannot rewrite scheme to {ws_scheme}"))?; + Ok(stream_url) +} + +/// POST /v1/trunk/webhook — Twilio's inbound-call signaling webhook +/// receiver (deploy slice A §5.3). Responds with TwiML instructing Twilio +/// to open a Media Streams fork against our `/twilio/media-stream` WS +/// endpoint, at the PUBLIC address derived from +/// `RUTSTER_TWILIO_WEBHOOK_BASE` — replacing the slice-5 placeholder's +/// hardcoded `ws://127.0.0.1:8080`, which no CPaaS could ever dial. +/// +/// 503 when the trunk is unconfigured: a webhook arriving without +/// `RUTSTER_TWILIO_*` set is an operator misconfiguration (Twilio was +/// pointed here, the binary wasn't told its public name) — failing the +/// call loudly beats returning a loopback Stream URL that dies silently +/// 10 s later. No XML escaping needed: the derived URL is +/// scheme+authority+fixed-path, no query, and `url` forbids `"` in +/// authorities. +pub async fn twilio_inbound_webhook(State(state): State) -> Response { + let Some(base) = state.trunk_webhook_base.as_ref() else { + return ( + StatusCode::SERVICE_UNAVAILABLE, + "trunk not configured: set the RUTSTER_TWILIO_* env vars \ + (webhook received, but there is no RUTSTER_TWILIO_WEBHOOK_BASE \ + to derive the public Stream URL from)", + ) + .into_response(); + }; + let stream_url = match stream_url_from_webhook_base(base) { + Ok(u) => u, + Err(e) => { + // Defensive: config::twilio_credentials validated the URL at + // startup, so this arm is unreachable in a correctly-booted + // binary. 500, never a panic. + tracing::error!(error = %e, "webhook base failed Stream-URL derivation"); + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + } + }; + let twiml = format!( + r#" + + + + +"# + ); + ( + StatusCode::OK, + [(header::CONTENT_TYPE, "application/xml")], + twiml, + ) + .into_response() +} +``` + +- [ ] **Step 5: Startup validation in `main.rs`** + +In `crates/rutster/src/main.rs`, replace the `AppState` construction (currently lines 38–39): + +```rust + let (placeholder_tx, _placeholder_rx) = mpsc::channel(1); + let app_state = AppState::new(placeholder_tx, default_tap_url); +``` + +with: + +```rust + let (placeholder_tx, _placeholder_rx) = mpsc::channel(1); + // Deploy slice A §5.3: config::twilio_credentials was parsed-but- + // never-called since slice-5 — wired into startup validation here. + // Partial or malformed RUTSTER_TWILIO_* config now fails the boot + // loudly instead of silently running WebRTC-only. (The credentials + // themselves stay green-zone: only webhook_base is threaded into the + // routes; TwilioCredentials' Debug redacts auth_token — ADR-0009.) + let twilio_credentials = rutster::config::twilio_credentials( + std::env::var("RUTSTER_TWILIO_ACCOUNT_SID").ok(), + std::env::var("RUTSTER_TWILIO_AUTH_TOKEN").ok(), + std::env::var("RUTSTER_TWILIO_MEDIA_BIND").ok(), + std::env::var("RUTSTER_TWILIO_WEBHOOK_BASE").ok(), + ) + .expect("RUTSTER_TWILIO_* env config invalid"); + if let Some(creds) = twilio_credentials.as_ref() { + info!(webhook_base = %creds.webhook_base, "trunk configured"); + } + let app_state = AppState::new(placeholder_tx, default_tap_url).with_trunk_webhook_base( + twilio_credentials.as_ref().map(|c| c.webhook_base.clone()), + ); +``` + +- [ ] **Step 6: Run tests — expect pass** + +```bash +cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings +cargo test -p rutster routes::tests +cargo test --all +``` + +Expected: all green (the pre-existing webhook behavior had no test asserting the loopback URL — +verified: `crates/rutster/tests/` does not exercise `/v1/trunk/webhook`). + +- [ ] **Step 7: Commit** + +```bash +git add crates/rutster/src/session_map.rs crates/rutster/src/routes.rs crates/rutster/src/main.rs +git commit -s -m "fix(trunk): derive TwiML Stream URL from RUTSTER_TWILIO_WEBHOOK_BASE; wire twilio_credentials into startup (deploy-A §5.3) + +Kills the slice-5 placeholder's hardcoded ws://127.0.0.1:8080 Stream URL +(routes.rs) — no CPaaS could ever dial it. The webhook now answers with +wss:///twilio/media-stream derived from the operator's +configured base (authority-only; https→wss, http→ws for the dev loop), +or 503 when the trunk is unconfigured. + +config::twilio_credentials existed since slice-5 but was never called: +partial/malformed RUTSTER_TWILIO_* config booted silently WebRTC-only. +main.rs now fail-fasts at startup, matching every other RUTSTER_* knob." +``` + +--- + +### Task 5: Trusted-proxy posture — `RUTSTER_TRUSTED_PROXIES` + `reconstruct_public_url` + +Spec §3.1 invariant 5: `X-Twilio-Signature` is an HMAC over the URL *as Twilio saw it* — +`https://pbx.example.com/v1/trunk/webhook`, not the plaintext `http://10.0.0.5:8080/...` the +FOB sees behind its TLS-terminating edge. Validation must reconstruct the public URL from +`X-Forwarded-Proto/Host`, but ONLY when the direct peer is the operator-configured edge — +otherwise any internet peer chooses the URL its own forgery is checked against. + +**Scoped honestly:** signature validation does not exist anywhere in the tree today (verified — +no HMAC/signature code in `crates/rutster-trunk` or `crates/rutster`; `hmac` isn't even in +`Cargo.lock`). This task lands the trusted-proxy half — the env parser, the pure reconstruction +helper with tests, exported from `rutster-trunk` for the trunk slice's future validation to +consume — plus the `AppState` access point, so the remaining trunk work is an implementation, +not a plumbing refactor. + +**Files:** +- Modify: `Cargo.toml` (root — `[workspace.dependencies]`, currently ends at the rcgen note, + line 81) +- Modify: `crates/rutster-trunk/Cargo.toml` (`[dependencies]`) +- Modify: `crates/rutster/Cargo.toml` (`[dependencies]`) +- Create: `crates/rutster-trunk/src/public_url.rs` +- Modify: `crates/rutster-trunk/src/lib.rs` (module list, currently lines 30–36) +- Modify: `crates/rutster/src/config.rs` (parser + tests) +- Modify: `crates/rutster/src/session_map.rs` (field + builder) +- Modify: `crates/rutster/src/main.rs` (parse + thread) + +**Interfaces:** +- Consumes: `ipnet::IpNet` (new direct workspace dep; already in `Cargo.lock` via reqwest). +- Produces: + - `pub fn config::trusted_proxies(raw: Option) -> Result, String>` (empty/unset → `vec![]` = forwarded headers IGNORED) + - `pub fn rutster_trunk::public_url::reconstruct_public_url(peer: IpAddr, forwarded_proto: Option<&str>, forwarded_host: Option<&str>, trusted_proxies: &[IpNet], fallback_base: &Url, path_and_query: &str) -> Result` + - `AppState.trusted_proxies: Vec` + `AppState::with_trusted_proxies(self, Vec) -> Self` + +- [ ] **Step 1: Add the workspace dep** + +Append to `[workspace.dependencies]` in the root `Cargo.toml` (after the `toml` entry, before +the rcgen NOTE comment): + +```toml +# ipnet 2: CIDR parsing for RUTSTER_TRUSTED_PROXIES (deploy slice A §5.3). +# Already in Cargo.lock transitively (via reqwest) — this adds only the +# direct edge; MIT/Apache-2.0, cargo-deny clean. +ipnet = "2" +``` + +Add to `crates/rutster-trunk/Cargo.toml` under `[dependencies]` (green-zone block, after `url`): + +```toml +ipnet = { workspace = true } +``` + +Add to `crates/rutster/Cargo.toml` under `[dependencies]` (after `url`): + +```toml +ipnet = { workspace = true } +``` + +- [ ] **Step 2: Write the failing helper tests + implementation file** + +`crates/rutster-trunk/src/public_url.rs` (complete file — tests included; write the tests +first, confirm the compile failure, then fill the fn): + +```rust +//! # public_url — reconstruct the URL as the CPaaS saw it +//! (deploy spec §3.1 invariant 5; deploy slice A §5.3) +//! +//! `X-Twilio-Signature` is an HMAC over the URL *as Twilio requested it* +//! — `https://pbx.example.com/v1/trunk/webhook` — while a FOB behind a +//! TLS-terminating edge sees `http://10.0.0.5:8080/v1/trunk/webhook`. +//! Validation must rebuild the public URL from the edge's +//! `X-Forwarded-Proto`/`X-Forwarded-Host`… but ONLY when the direct TCP +//! peer is the operator-configured edge (`RUTSTER_TRUSTED_PROXIES`). +//! Honoring those headers from arbitrary peers would let an attacker +//! choose the exact URL their own forged signature is checked against. +//! +//! ## Scope (honest) +//! +//! Signature validation itself does NOT exist yet — the trunk webhook is +//! a slice-5-era handler with no HMAC anywhere in the tree. This module +//! lands the trusted-proxy half with tests so the trunk slice consumes +//! it instead of growing its own. It is a pure function over extracted +//! values (no axum types) — the future caller extracts the peer IP +//! (`ConnectInfo`) and header strings and gets back the URL to HMAC. +//! +//! Fail-closed contract: `Err` from this function means "the request's +//! public URL cannot be trusted" — a signature-validating caller must +//! reject the request, never fall back silently. + +use std::net::IpAddr; + +use ipnet::IpNet; +use url::Url; + +/// Rebuild the public URL for a request that arrived from `peer`. +/// +/// - `peer` ∉ `trusted_proxies` (including the empty-list default): +/// forwarded headers are IGNORED — the URL is `fallback_base` + +/// `path_and_query`. An untrusted peer never influences the result. +/// - `peer` ∈ `trusted_proxies` and BOTH headers present: the URL is +/// `{proto}://{host}{path_and_query}`. Chained proxies append +/// comma-separated values; the FIRST entry is what the outermost edge +/// (the hop the CPaaS actually dialed) saw, so that one wins. +/// - `peer` trusted but either header missing: fall back to +/// `fallback_base` — a half-forwarding edge must not half-win. +/// - Trusted peer + malformed values (non-http(s) proto, unparseable +/// host): `Err` — fail closed, see module docs. +/// +/// `path_and_query` is the request's origin-form target, e.g. +/// `/v1/trunk/webhook` or `/v1/trunk/webhook?a=b` (Twilio signs query +/// strings too on webhooks — Media Streams URLs never carry them, spec +/// §3.1 invariant 4). +pub fn reconstruct_public_url( + peer: IpAddr, + forwarded_proto: Option<&str>, + forwarded_host: Option<&str>, + trusted_proxies: &[IpNet], + fallback_base: &Url, + path_and_query: &str, +) -> Result { + let peer_trusted = trusted_proxies.iter().any(|net| net.contains(&peer)); + if peer_trusted { + if let (Some(proto), Some(host)) = (forwarded_proto, forwarded_host) { + let proto = proto.split(',').next().unwrap_or("").trim(); + let host = host.split(',').next().unwrap_or("").trim(); + if proto != "http" && proto != "https" { + return Err(format!( + "X-Forwarded-Proto {proto:?} from trusted peer {peer} is not http(s)" + )); + } + return Url::parse(&format!("{proto}://{host}{path_and_query}")) + .map_err(|e| format!("forwarded public URL does not parse: {e}")); + } + // Trusted peer, incomplete headers: fall through to the fallback. + } + fallback_base + .join(path_and_query) + .map_err(|e| format!("fallback URL join: {e}")) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn base() -> Url { + Url::parse("https://pbx.example.com").unwrap() + } + + fn edge_net() -> Vec { + vec!["10.0.0.0/24".parse().unwrap()] + } + + #[test] + fn untrusted_peer_headers_are_ignored() { + let u = reconstruct_public_url( + "203.0.113.9".parse().unwrap(), + Some("https"), + Some("evil.example.net"), + &edge_net(), + &base(), + "/v1/trunk/webhook", + ) + .unwrap(); + assert_eq!(u.as_str(), "https://pbx.example.com/v1/trunk/webhook"); + } + + #[test] + fn empty_trusted_list_default_ignores_headers() { + // RUTSTER_TRUSTED_PROXIES unset → empty list → headers IGNORED, + // even from loopback (deploy spec §5.7: empty default is the + // fail-closed posture, not a convenience). + let u = reconstruct_public_url( + "127.0.0.1".parse().unwrap(), + Some("https"), + Some("evil.example.net"), + &[], + &base(), + "/v1/trunk/webhook", + ) + .unwrap(); + assert_eq!(u.as_str(), "https://pbx.example.com/v1/trunk/webhook"); + } + + #[test] + fn trusted_peer_headers_are_honored_with_port_and_query() { + let u = reconstruct_public_url( + "10.0.0.7".parse().unwrap(), + Some("https"), + Some("pbx.example.com:8443"), + &edge_net(), + &base(), + "/v1/trunk/webhook?CallSid=CA123", + ) + .unwrap(); + assert_eq!( + u.as_str(), + "https://pbx.example.com:8443/v1/trunk/webhook?CallSid=CA123" + ); + } + + #[test] + fn trusted_peer_comma_chained_values_take_the_outermost() { + let u = reconstruct_public_url( + "10.0.0.7".parse().unwrap(), + Some("https, http"), + Some("pbx.example.com, internal.local"), + &edge_net(), + &base(), + "/v1/trunk/webhook", + ) + .unwrap(); + assert_eq!(u.as_str(), "https://pbx.example.com/v1/trunk/webhook"); + } + + #[test] + fn trusted_peer_with_missing_header_falls_back() { + let u = reconstruct_public_url( + "10.0.0.7".parse().unwrap(), + Some("https"), + None, + &edge_net(), + &base(), + "/v1/trunk/webhook", + ) + .unwrap(); + assert_eq!(u.as_str(), "https://pbx.example.com/v1/trunk/webhook"); + } + + #[test] + fn trusted_peer_with_garbage_proto_fails_closed() { + let err = reconstruct_public_url( + "10.0.0.7".parse().unwrap(), + Some("gopher"), + Some("pbx.example.com"), + &edge_net(), + &base(), + "/v1/trunk/webhook", + ) + .unwrap_err(); + assert!(err.contains("X-Forwarded-Proto"), "msg: {err}"); + } +} +``` + +Add the module to `crates/rutster-trunk/src/lib.rs` — the declaration list (currently lines +30–36) becomes: + +```rust +pub mod g711; +pub mod loop_driver; +mod mulaw_decode_table; +mod mulaw_encode_table; +pub mod provider; +pub mod public_url; +pub mod session; +pub mod twilio_media_streams; +``` + +- [ ] **Step 3: Run the helper tests** + +```bash +cargo test -p rutster-trunk public_url +``` + +Expected: all six pass. (If written tests-first as instructed, the intermediate run fails with +`cannot find function `reconstruct_public_url``.) + +- [ ] **Step 4: Write the failing config-parser tests, then the parser** + +Append to `mod tests` in `crates/rutster/src/config.rs`: + +```rust + #[test] + fn trusted_proxies_empty_when_unset() { + assert!(trusted_proxies(None).unwrap().is_empty()); + } + + #[test] + fn trusted_proxies_empty_string_is_empty_list() { + assert!(trusted_proxies(Some("".into())).unwrap().is_empty()); + } + + #[test] + fn trusted_proxies_parses_cidrs_and_bare_ips() { + let nets = trusted_proxies(Some("10.0.0.0/24, 192.168.1.7".into())).unwrap(); + assert_eq!(nets.len(), 2); + assert!(nets[0].contains(&"10.0.0.99".parse::().unwrap())); + // Bare IP == host-length prefix (/32). + assert!(nets[1].contains(&"192.168.1.7".parse::().unwrap())); + assert!(!nets[1].contains(&"192.168.1.8".parse::().unwrap())); + } + + #[test] + fn trusted_proxies_rejects_garbage_with_var_name_in_error() { + let err = trusted_proxies(Some("10.0.0.0/24,not-a-cidr".into())).unwrap_err(); + assert!(err.contains("RUTSTER_TRUSTED_PROXIES")); + assert!(err.contains("not-a-cidr")); + } +``` + +Run `cargo test -p rutster config::tests` — expect the compile failure naming +`trusted_proxies`. Then add the parser to `crates/rutster/src/config.rs` (after +`ws_ping_interval`): + +```rust +/// `RUTSTER_TRUSTED_PROXIES` — comma-separated CIDR list of edge peers +/// whose `X-Forwarded-Proto`/`X-Forwarded-Host` are believed +/// (deploy slice A §5.3; consumed via +/// `rutster_trunk::public_url::reconstruct_public_url`). +/// +/// Empty or unset ⇒ empty list ⇒ forwarded headers IGNORED — fail-closed: +/// an internet peer must never choose the public URL that Twilio +/// signature validation reconstructs (deploy spec §3.1 invariant 5). +/// Bare IPs are accepted as host-length prefixes +/// (`10.0.0.7` ≡ `10.0.0.7/32`). +pub fn trusted_proxies(raw: Option) -> Result, String> { + let Some(s) = raw else { + return Ok(Vec::new()); + }; + let mut nets = Vec::new(); + for entry in s.split(',') { + let entry = entry.trim(); + if entry.is_empty() { + continue; + } + let net = entry + .parse::() + .or_else(|_| entry.parse::().map(ipnet::IpNet::from)) + .map_err(|e| format!("RUTSTER_TRUSTED_PROXIES entry {entry:?}: {e}"))?; + nets.push(net); + } + Ok(nets) +} +``` + +(`IpAddr` is already imported at the top of `config.rs`.) Run +`cargo test -p rutster config::tests` — expect pass. + +- [ ] **Step 5: `AppState` access point + `main.rs` threading** + +In `crates/rutster/src/session_map.rs`, extend the struct/builder from Task 4: + +```rust + /// CIDR list of edge peers whose forwarded headers are believed + /// (`RUTSTER_TRUSTED_PROXIES`; empty = ignored, fail-closed). No + /// route consumes this yet — it is the declared access point for the + /// trunk slice's signature validation and slice C's TLS listener, so + /// that work is an implementation, not a plumbing refactor. + pub trusted_proxies: Vec, +``` + +(add the field to the struct, `trusted_proxies: Vec::new()` to `new`'s `Self { .. }`, and the +builder:) + +```rust + pub fn with_trusted_proxies(mut self, nets: Vec) -> Self { + self.trusted_proxies = nets; + self + } +``` + +In `crates/rutster/src/main.rs`, chain onto the Task-4 `AppState` construction: + +```rust + let trusted_proxies = + rutster::config::trusted_proxies(std::env::var("RUTSTER_TRUSTED_PROXIES").ok()) + .expect("RUTSTER_TRUSTED_PROXIES must be a comma-separated CIDR list"); + let app_state = AppState::new(placeholder_tx, default_tap_url) + .with_trunk_webhook_base( + twilio_credentials.as_ref().map(|c| c.webhook_base.clone()), + ) + .with_trusted_proxies(trusted_proxies); +``` + +- [ ] **Step 6: Run the full gates** + +```bash +cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings +cargo test --all +cargo deny check +``` + +Expected: all green; `cargo deny` accepts `ipnet` (already vetted transitively). + +- [ ] **Step 7: Commit** + +```bash +git add Cargo.toml Cargo.lock crates/rutster-trunk/Cargo.toml crates/rutster-trunk/src/public_url.rs crates/rutster-trunk/src/lib.rs crates/rutster/Cargo.toml crates/rutster/src/config.rs crates/rutster/src/session_map.rs crates/rutster/src/main.rs +git commit -s -m "feat(trunk): trusted-proxy posture — RUTSTER_TRUSTED_PROXIES + public-URL reconstruction helper (deploy-A §5.3) + +X-Twilio-Signature is an HMAC over the URL as Twilio saw it; behind a +TLS-terminating edge the FOB must rebuild that URL from +X-Forwarded-Proto/Host — but ONLY from the operator-configured edge +(CIDR list; empty default = headers IGNORED, fail-closed), or any +internet peer chooses the URL its own forgery is checked against +(spec §3.1 invariant 5). + +Scoped honestly: signature validation does not exist in the tree yet. +This lands the trusted-proxy half — pure reconstruct_public_url in +rutster-trunk (fail-closed Err contract, outermost-hop comma handling) +with tests, the config.rs fail-fast parser, and the AppState access +point — so the trunk slice consumes it instead of growing its own. + +New direct dep ipnet 2 (workspace-pinned; already in Cargo.lock via +reqwest; MIT/Apache-2.0, cargo-deny clean)." +``` + +--- + +### Task 6: Final verification sweep + +No code. Evidence before assertions — run every gate the CI runs, plus the seam-gate check, +before calling the slice done. + +- [ ] **Step 1: Full gate run** + +```bash +cargo fmt --all --check +cargo clippy --all --all-targets -- -D warnings +cargo clippy --all --all-targets --features=sim-bench -- -D warnings +cargo test --all +cargo test --all --features=sim-bench -- --test-threads=1 +cargo deny check +cargo doc --no-deps +``` + +Expected: every command exits 0. + +- [ ] **Step 2: Seam gate verification** + +```bash +git hash-object crates/rutster-media/src/loop_driver.rs +git hash-object crates/rutster-media/src/rtc_session.rs +``` + +Expected output, exactly: + +``` +744bf314edf7f4925c8bb3bd0f5176dbc88f8113 +f47d63b9a2883d37066a93c9daa0e2cf8816bec4 +``` + +If either hash differs, STOP — a task touched a seam-gated file; restore with +`git checkout main -- crates/rutster-media/src/loop_driver.rs crates/rutster-media/src/rtc_session.rs` +and re-run the gates. + +- [ ] **Step 3: MSRV spot-check (matches the CI 1.85 leg)** + +```bash +cargo +1.85 test --all +``` + +Expected: green. (If the 1.85 toolchain is not installed locally, note it and rely on the CI +matrix — do not skip silently.) + +--- + +## Final acceptance checklist + +After all tasks land: + +- [ ] `cargo fmt --check`, `clippy -D warnings` (default + `--features=sim-bench`), + `cargo test --all`, sim-bench sweep (`--test-threads=1`), `cargo deny check`, + `cargo doc --no-deps` all clean. +- [ ] Seam gate: `loop_driver.rs` + `rtc_session.rs` blob hashes unchanged (Step 2 above). +- [ ] Every accepted socket on the HTTP/WS listener has TCP_NODELAY (via + `serve_with_nodelay`, the single production serve path) and the sim-bench assertion was + **proven load-bearing** (fails with `tcp_nodelay(false)` — Task 3 Step 5). +- [ ] Trunk media-stream WS emits engine-originated Pings at `RUTSTER_WS_PING_SECS` (default + 20 s); `0`/garbage fail the boot. +- [ ] `POST /v1/trunk/webhook` returns TwiML with `wss:///twilio/media-stream`; the hardcoded `ws://127.0.0.1:8080` is gone from + `routes.rs`; unconfigured trunk → 503. +- [ ] Partial/malformed `RUTSTER_TWILIO_*` config fails startup (`config::twilio_credentials` + is actually called now). +- [ ] `reconstruct_public_url` exported from `rutster-trunk` with the fail-closed contract + documented + tested; `RUTSTER_TRUSTED_PROXIES` parsed fail-fast, empty default = headers + ignored. +- [ ] Spec §9 code-test bullet "signature validation behind forwarded headers (trusted and + untrusted source cases)" is covered by Task 5's `reconstruct_public_url` tests + (`untrusted_peer_headers_are_ignored` — untrusted source case; `empty_trusted_list_default_ignores_headers`; + `trusted_peer_headers_are_honored_with_port_and_query` — trusted source case; plus + `trusted_peer_comma_chained_values_take_the_outermost`, + `trusted_peer_with_missing_header_falls_back`, and + `trusted_peer_with_garbage_proto_fails_closed`). HMAC signature validation itself is + NOT built in slice A — spec §1.1 lists only "trusted-proxy header posture" as in-scope + code; the reconstruction helper is the tested seam the trunk slice consumes when it adds + HMAC. Six tests, both source cases, both fail-closed paths. +- [ ] Env additions match spec §5.7 rows: `RUTSTER_WS_PING_SECS` (default 20), + `RUTSTER_TRUSTED_PROXIES` (empty default). No other knobs invented + (`RUTSTER_DRAIN_DEADLINE_SECS` is unchanged from slice-5/seams in slice A; its 600 s + `.env.example` value ships in slice B per spec §5.7 row + §8). +- [ ] PR opened via `tea pulls create --head deploy-a/engine-hygiene --base main --title + "deploy slice A (engine hygiene): TCP_NODELAY + WS pings + trunk config fixes"` with the + spec's §5.1–§5.3 as the description skeleton. Slices B and C branch after this merges. diff --git a/docs/superpowers/plans/2026-07-05-deploy-b-packaging-ci.md b/docs/superpowers/plans/2026-07-05-deploy-b-packaging-ci.md new file mode 100644 index 0000000..0d1fe0e --- /dev/null +++ b/docs/superpowers/plans/2026-07-05-deploy-b-packaging-ci.md @@ -0,0 +1,3114 @@ +# Deploy slice B + F — `deploy/` artifacts (Dockerfile/compose/Caddyfile/.env) + image-build + container smoke CI + tag-push publish — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use `superpowers:subagent-driven-development` (recommended) or `superpowers:executing-plans` to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Dependency:** This plan depends on **plan A** (`2026-07-05-deploy-a-engine-hygiene.md`) landing +on `main` first. Plan A's three hygiene items are what make the containerized PSTN path honest: +**TCP_NODELAY** (`serve_with_nodelay`) so the 20 ms WS cadence isn't Nagle-coalesced behind the +edge, **app-level WS pings** so neither Twilio's nor Telnyx's idle timers can kill a quiet +mid-call WS, and the **trunk config fixes** (`RUTSTER_TWILIO_WEBHOOK_BASE` → derived TwiML Stream +URL + `config::twilio_credentials` startup validation + `RUTSTER_TRUSTED_PROXIES` +trusted-X-Forwarded-Proto/Host reconstruction) so a container that takes a real Twilio call is +actually answering the public URL the CPaaS dialed, not `ws://127.0.0.1:8080` from slice-5's +stub. Slice C (rustls Phase 1) also branches after A; B and C are order-independent. **Do not +start Task 1 of this plan until `git log main --oneline | grep -c 'deploy-A'` >= 1.** + +**Goal:** Land the `deploy/` artifact tree (multi-stage Dockerfile producing four first-party +images, `compose.yaml`, `Caddyfile`, `.env.example`) and the containerized CI muscle that +proves them — image-build job, all-in-one TLS smoke through Caddy's internal CA, compose-health +smoke, and Caddy config-reload-during-live-call — then add the slice-F tag-push publish workflow +to the `git.adlee.work/alee/` registry namespace (spec §2.1, §2.2, §3.2, §6, §9, §11). + +**Architecture:** One multi-stage `deploy/Dockerfile` with named build targets (`rutster-engine`, +`rutster-brain`, `rutster-edge`, `rutster-allinone`) so `docker buildx build --target …` produces +each image from the same Dockerfile — eliminating the per-image Dockerfile drift that broke +Asterisk-era integrators. The builder stage compiles against `rust:1.85-slim-bookworm` + +`libopus-dev` (matching `rust-toolchain.toml` + dev loop exactly) and is a single target shared +across engine/brain (both derive from the same `cargo build --bins`); the all-in-one image adds +s6-overlay v3 + the upstream `valkey/valkey` server binary + the custom Caddy from a parallel +xcaddy builder. CI is added to the existing `.github/workflows/ci.yml` (Gitea Actions +consuming GitHub-workflow YAML) as two new jobs (`image-build` + `smoke`) plus a third workflow +file (`.github/workflows/publish-images.yml`) for the slice-F tag-push publish. The smoke job +reuses the WS-handshake protocol sequence (the `{"event":"connected"…}` + `{"event":"start"…}` +frames) proven in `crates/rutster/tests/trunk_sim_e2e.rs`'s TODO body and the in-tree +`ws_ping.rs`/`nodelay.rs` integration tests — but driving it through the *real* edge→FOB path +inside a booted container, against Caddy's internal CA (no ACME in CI). The +`ValkeyEventSink`-lands-in-stream assertion is explicitly a slice-C concern; this plan leaves a +**named TODO hook** comment in the compose-smoke step and does NOT implement it. + +**Tech Stack:** Docker (BuildKit multi-stage), `rust:1.85-slim-bookworm` builder (pinned to +`rust-toolchain.toml`), `debian:bookworm-slim` runtime (matches dev: `apt-get install libopus0` +in runtime, `libopus-dev` in builder — spec §6.1 "distro libopus0 over static-bundled"), s6-overlay +v3.2.0.0 (ISC), xcaddy with the curated DNS-plugin set (cloudflare/route53/porkbun/hetzner/desec +— duckdns excluded, no license file per spec §3.2), upstream `valkey/valkey:latest` (BSD-3), +Caddy 2.8 builder image. CI: Gitea Actions (GitHub Actions YAML syntax), `docker` CLI, Python 3 +for the smoke-test WS client (stdlib only — no `pip install` step in CI). Zero new Rust crates. +Zero changes to `cargo` graphs. + +## Global Constraints + +House rules (every task's requirements implicitly include this section): + +- **License:** GPL-3.0-or-later on every crate manifest (ADR-0004). No Rust crates touched in + this slice; the rule stands for any task that would. +- **DCO:** every commit signed off — `git commit -s` (AGENTS.md Git workflow). +- **SEAM GATE (UNCHANGED):** `crates/rutster-media/src/loop_driver.rs` and + `crates/rutster-media/src/rtc_session.rs` stay **byte-identical** — CI pins their blob hashes + (`744bf314edf7f4925c8bb3bd0f5176dbc88f8113` and `f47d63b9a2883d37066a93c9daa0e2cf8816bec4` + respectively, in `.github/workflows/ci.yml` lines 60–61). NO task in this plan touches them. + This slice is artifacts + CI, not the RTP path. Verify unchanged hashes in the final sweep + (Task 11 Step 2). +- **Hot-path policy:** never `?`-propagate on the 20 ms tick; no `unwrap()`/`expect()` outside + tests/startup. (No Rust code added; every smoke-test Python script honors it: WS read failures + `assert` loudly with a counter snapshot — they never silently loop.) +- **Code style:** `cargo fmt --all --check` + `cargo clippy --all --all-targets -- -D warnings` + must stay green. (This slice adds no Rust; CI's clippy job stays the source of truth.) +- **MSRV:** CI matrix runs stable + 1.85 (per `rust-toolchain.toml`). The Dockerfile builder + stage MUST pin `1.85` exactly — not `stable` — so image builds reproduce what the MSRV matrix + gates. No `cargo update` wholesale (rcgen 0.14.7 lock rationale per root `Cargo.toml`). +- **Workspace dep pinning:** new deps go in the ROOT `Cargo.toml` `[workspace.dependencies]`; + members reference with `dep.workspace = true`. (None added here — all artifact deps are apt + packages or upstream Docker images, not crates.) +- **cargo-deny license gate:** `cargo deny check` must stay green. The bundled-binary licenses + in the images (Caddy Apache-2.0, valkey BSD-3, s6-overlay ISC, Debian GPL/MIT stack) are + **aggregation-clean** vs GPL-3 — Rust's cargo-deny operates on the Rust crate graph only and + is unaffected. The image license scan is a future cargo-deny-adjacent CI muscle (spec §6.1), + not this slice's concern; this slice notes the license posture in `deploy/.env.example`'s + header comment. +- **sim-bench CI:** the `sim-bench` job runs + `cargo test --all --features=sim-bench -- --test-threads=1` — `--test-threads=1` is + load-bearing (shared tick-lag gauge). Unaffected by this slice; the new smoke jobs are + independent Gitea Actions jobs, not additional `cargo test` invocations. +- **Branch/PR:** branch `deploy-b/packaging-ci`; PR to `main` via `tea`. Slices A landed first + (this plan's dependency root); slice C branches after A merges and is order-independent + with B. The slice-F publish workflow lives in the same PR (the file is inert until a tag is + pushed — no risk of running before artifacts exist). + +Slice-B-specific constraints: + +- **The four image names are pinned** to spec §6.1: `rutster-allinone`, `rutster-engine`, + `rutster-brain`, `rutster-edge`. The CI image-build job asserts each `--target` produces a + tagged image with exactly that name. Do not invent variants (`rutster-fob`, `rutster-core`, + etc.) — the docs tree (plan G) hard-codes these four names. +- **The publish registry namespace is `git.adlee.work/alee/rutster-*`** (matches the `alee` + tea login per AGENTS.md Git workflow; the root `Cargo.toml`'s + `repository = "https://git.adlee.work/alee/rutster"` already commits to this). If your Gitea + registry uses a different owner segment, replace `alee` throughout the publish workflow + + `deploy/.env.example`'s registry-namespace comment. +- **linux/amd64 only.** arm64 is a named deferral (spec §1.2, ADR-0011); the publish workflow + pins `platforms: linux/amd64`. +- **Caddy's `local_certs` (internal CA) is the CI TLS path.** No ACME in CI (no public DNS, + no rate-limit budget, hits Let's Encrypt's duplicate-cert path the instant a job re-runs). + Production Caddyfiles default to ACME; CI overrides via `RUTSTER_LOCAL_CERTS=true` env in the + smoke job. The `rutster-edge` image is the same in CI and prod — only the runtime env differs. +- **The smoke WS client is Python stdlib only.** No `pip install` step in CI (Gitea runners + ship Python 3 + stdlib; `websocket-client`/`websockets` would require a `pip install` network + round-trip per job — a friction the slice-A sim-bench posture avoids). The WS handshake is ten + lines of code on top of `socket` + `ssl`. +- **The `ValkeyEventSink` smoke assertion belongs to plan C.** Leave a `# TODO slice-C: assert + lifecycle event in Valkey stream` named hook comment in the compose-smoke step. Do NOT + implement the assertion; the Valkey service ships in compose (so the docs + the smoke shape + the contract) but no event-lands check runs in this slice. +- **Image build cache:** the `image-build` job uses `docker/build-push-action@v5` with + `cache-from: type=gha` + `cache-to: type=gha,mode=max`. The cache scope includes the Rust + target dir; warm caches take image-build from ~6 min to ~90 s. Documented in the CI job. + +## Spec ambiguities resolved (load-bearing — flagged so the implementing engineer doesn't +re-derive them) + +1. **"the T8 e2e stub"** (referenced in the assignment): there is no `t8*` file in the tree. + The closest match is `crates/rutster/tests/trunk_sim_e2e.rs` — a `#[ignore]`'d TODO test + whose body sketches a full PSTN e2e against `MockTwilioMediaStreamsServer` + + `MediaCmd::RegisterTrunk` through the binary's `MediaThread`. That file's TODO text is + exactly the slice-B all-in-one smoke, except this slice drives it against the *real* + containerized edge→FOB WS path through Caddy's internal CA, rather than a + `MockTwilioMediaStreamsServer`. The WS-handshake JSON sequence (`connected` + `start` + frames) is borrowed verbatim from `crates/rutster-trunk/tests/ws_ping.rs` and + `crates/rutster-sim/src/nodelay.rs` (both slice-A additions). The `trunk_sim_e2e.rs` file + itself stays untouched — its `#[ignore]` TODO is a unit-test concern (in-process + `MediaThread` driving); the CI smoke is a black-box container concern (whole binary booted, + whole Caddy in front, external Python client). They are different test layers. +2. **`rutster-brain` runtime image libopus0:** spec §6.1 table lists `debian-slim` for the + brain (no `libopus0`). But `crates/rutster-brain-realtime/Cargo.toml` depends on + `rutster-media`, which depends on the `opus` crate (system FFI against libopus.so.0). Even + though the brain binary never calls an opus symbol, the libopus shared library is recorded + in the binary ELF as `NEEDED: libopus.so.0` because the linker resolves every Rust dep's + `[dependencies]` — a missing `.so.0` at startup would make the brain crash with + `error while loading shared libraries: libopus.so.0`. Resolution: install `libopus0` in the + brain runtime image too (one extra `apt-get install -y libopus0`, ~500 KB + additional). This is a defensive deviation from spec §6.1's brain row; flagged in the + Dockerfile inline comment with the linker rationale so a future brain-crate refactor that + drops the `rutster-media` dep can drop this package too. +3. **`RUTSTER_DOMAIN` + `RUTSTER_ACME_EMAIL` env-var names:** plan G's `quickstart-docker.md` + uses these two names (they are artifact-level names owned by slice B's Caddyfile — the spec + does not fix them). Adopted as the canonical names; the Caddyfile reads them via env + interpolation. The `.env.example` documents both. The `rutster-edge` Caddy binary has zero + knowledge of `RUTSTER_*` semantics — Caddy reads `{$DOMAIN}` + `{$ACME_EMAIL}` from its + environment per Caddyfile env-substitution syntax. Wait — the Caddyfile uses + `{$RUTSTER_DOMAIN}` and `{$RUTSTER_ACME_EMAIL}` to match the rutster convention, even though + Caddy is consuming them. This keeps the operator's `RUTSTER_*` namespace uniform across + both processes. +4. **`stop_grace_period` exactly 660 s (not 600 s):** the spec pins 660 s + drain 600 s + (grace-exceeds-drain invariant — spec §8). 60 s is the slack so the orchestrator doesn't + cut a drain that's legitimately waiting on a 599-second in-flight call. Pinned exactly; + do not "tune." +5. **Caddy reload-during-live-call smoke:** the spec §3.2 sentence "verified protocol-unaware + WS tunneling (no frame buffering)" + §9 bullet 4 ("Caddy config reload during a live + simulated call, asserting zero drops") load the heaviest claim in this slice. The smoke + asserts: during a live WS streaming media frames at 20 ms cadence through Caddy to the FOB, + a `caddy reload` (the operator's config-edit path) drops zero frames. The mitigation + under test is `stream_close_delay 24h` (Caddyfile) — above max call duration. The upstream + bug trail (caddy #6420, #7222) is documented in the TLS brief §5 risk 1 — the smoke is the + "don't trust untested" mitigation made CI-regressed. + +## File Structure + +### New files + +| Path | Responsibility | +|---|---| +| `deploy/Dockerfile` | Multi-stage: one builder (`rust:1.85-slim-bookworm` + `libopus-dev`) producing `rutster` + `rutster-brain-realtime` binaries; one xcaddy builder producing `caddy` with the curated DNS-plugin set; one valkey binary source stage; four runtime targets: `rutster-engine`, `rutster-brain`, `rutster-edge`, `rutster-allinone` (s6-overlay v3 supervising caddy + FOB + brain + valkey-server). | +| `deploy/.dockerignore` | Build-context pruning: excludes `target/`, `.git/`, `node_modules/`, `*.md` (other than README), `.env*`, `crates/*/target/`, `docs/`, `.github/`, `.opencode/`. Halves image-build context size. | +| `deploy/Caddyfile` | ~6-line site block + global options: tuned timeouts (`read_body 30s`, `read_header 30s`, `write 0`, `idle 24h`), honest `X-Forwarded-Proto`/`X-Forwarded-Host` (no `X-Forwarded-For`-trust of client-controlled values — only the honest hop), `stream_close_delay 24h` (above max call duration; spec §2.1 / TLS brief §3(a)), reverse_proxy to `127.0.0.1:8080` (loopback-only FOB). | +| `deploy/compose.yaml` | Four services — `caddy` (image `rutster-edge`), `engine` (image `rutster-engine`), `brain` (image `rutster-brain`, `network_mode: "service:engine"` so the loopback-only tap posture survives), `valkey` (upstream `valkey/valkey`). `stop_grace_period: 660s`. Volumes for caddy `/data` + valkey `/data`. | +| `deploy/.env.example` | All `RUTSTER_*` vars incl. `RUTSTER_DRAIN_DEADLINE_SECS=600`, `RUTSTER_DOMAIN` + `RUTSTER_ACME_EMAIL`, the four `RUTSTER_TWILIO_*`, `RUTSTER_MEDIA_ADVERTISED_IP`, `RUTSTER_MEDIA_PORT_RANGE`, `RUTSTER_WS_PING_SECS`, `RUTSTER_TRUSTED_PROXIES`. Each line commented with the slice that owns it + the failing-fast behavior. | +| `deploy/s6-rc.d/{caddy,engine,brain,valkey-database}/run` + dependencies.d | s6-overlay v3 service definitions — four `run` exec scripts + their dependency edges (engine before brain; valkey before engine; caddy before engine, etc.). | +| `deploy/s6-overlay-noarch.tar.xz` + `deploy/s6-overlay-x86_64.tar.xz` | Vendored s6-overlay v3.2.0.0 binaries (ISC). Pinned at vendoring time per AGENTS.md reproducibility posture; SHA256 recorded alongside. | +| `.github/workflows/publish-images.yml` | Slice F: tag-push workflow. Builds + pushes all four images to `git.adlee.work/alee/rutster-*` on `v*` tag push. `linux/amd64` only. Gated on the existing `ci.yml` pipeline via workflow_run trigger (publish re-runs the fmt/clippy/test/deny/sim-bench + image-build as a gate, then publishes). | +| `deploy/smoke/allinone_smoke.py` | Python-stdlib WS client (no pip deps): boots `rutster-allinone`, extracts Caddy's internal-CA root cert from `/data`, opens a wss:// WS to `/twilio/media-stream`, sends Twilio's `connected` + `start` handshake frames, streams 200 PCM frames at 20 ms cadence, asserts received frames count within tolerance. **No ValkeyEventSink assertion — TODO slice-C hook is in this file as a comment.** | +| `deploy/smoke/compose_smoke.sh` | Bash: `docker compose up -d --wait`, then `curl /healthz` against each service, then `docker compose ps` JSON parse for healthy statuses. The `# TODO slice-C: assert lifecycle event in Valkey stream` named hook is here. | +| `deploy/smoke/reload_during_call.py` | Python-stdlib WS client that holds the WS open streaming frames while a concurrent `caddy reload` fires mid-call; asserts zero frames dropped across the reload window. | + +### Modified files + +| Path | What changes | +|---|---| +| `.github/workflows/ci.yml` | Add three new jobs: `image-build` (after existing `test` + `deny` + `sim-bench`), `smoke` (after `image-build`), `smoke-compose` + `smoke-reload` (after `smoke`). The existing `fmt` / `clippy` / `test` / `deny` / `sim-bench` / `twilio-live` jobs are unchanged. | + +### SEAM-INVARIANT files (DO NOT TOUCH) + +- `crates/rutster-media/src/loop_driver.rs` — byte-identical, all tasks. +- `crates/rutster-media/src/rtc_session.rs` — byte-identical, all tasks. + +## Interfaces + +**Consumes (no new code; reuses what slice A + slice-5 built):** +- `rutster::serve::serve_with_nodelay` (plan A Task 1) — the production serve path the + all-in-one binary runs. +- `rutster_trunk::twilio_media_streams::TwilioMediaStreamsServer::router(register_tx, + ping_interval)` (plan A Task 2) — the trunk WS route mounted on the main axum listener. +- `rutster::config::twilio_credentials(...)` (plan A Task 4) — startup validation of the four + `RUTSTER_TWILIO_*` vars. +- `rutster::session_map::AppState::with_trunk_webhook_base(_).with_trusted_proxies(_)` + (plan A Tasks 4+5) — the `AppState` builder chain. +- Plan A's slice env additions: `RUTSTER_WS_PING_SECS` (default 20), `RUTSTER_TRUSTED_PROXIES` + (empty default). +- Existing config.rs env parsers (slice-5): `RUTSTER_HTTP_BIND`, `RUTSTER_MEDIA_BIND_IP`, + `RUTSTER_MEDIA_ADVERTISED_IP`, `RUTSTER_MEDIA_PORT_RANGE`, `RUTSTER_MAX_SESSIONS`, + `RUTSTER_DRAIN_DEADLINE_SECS`, the four `RUTSTER_TWILIO_*`. +- Existing routes: `/healthz` (200 = liveness), `/readyz` (200 = can-accept-call, 503 = + draining / at admission cap), `/v1/trunk/webhook` (POST → TwiML), `/twilio/media-stream` (WS + upgrade). +- The WebRTC static client at `/` (slice-1 — booted binary serves it via axum + `.fallback_service`). + +**Produces:** +- Four first-party Docker images, built from one `deploy/Dockerfile` via named `--target`s: + - `rutster-engine:latest` — FOB only (binary `rutster` + `libopus0` runtime). + - `rutster-brain:latest` — brain only (binary `rutster-brain-realtime` + `libopus0` runtime + — see spec-ambiguity #2 above for the presence of libopus0 in this image). + - `rutster-edge:latest` — custom xcaddy build with curated DNS plugins + (cloudflare/route53/porkbun/hetzner/desec — duckdns excluded per spec §3.2 + TLS brief + §3(a)). + - `rutster-allinone:latest` — s6-overlay v3 supervising `caddy` + `rutster` (FOB) + + `rutster-brain-realtime` + `valkey-server` (T1 per spec §2.1). +- A composed `docker compose up -d` that runs the T2 reference deployment (spec §2.2) — the + `caddy` service is `rutster-edge`, `engine` is `rutster-engine`, `brain` is `rutster-brain` + (sharing the engine's netns via `network_mode: "service:engine"`), `valkey` is upstream + `valkey/valkey`. +- Three CI jobs in `.github/workflows/ci.yml`: `image-build`, `smoke` (all-in-one TLS + + reload-during-call), `smoke-compose` — gated on the existing fmt/clippy/test/deny/sim-bench + matrix. +- One new workflow `.github/workflows/publish-images.yml` (slice F): tag-push publish of all + four images to `git.adlee.work/alee/rutster-*`, `linux/amd64` only, gated on the + image-build job. + +## Task ordering + +- **Task 1** — `deploy/Dockerfile` (multi-stage, four targets) + vendor s6-overlay tarballs + + create s6-rc service definitions. Foundational: all CI smoke jobs build images from this + file. No deps on later tasks. +- **Task 2** — `deploy/.dockerignore`. Tiny; lands alongside Task 1 (separate file so the diff + is reviewable). +- **Task 3** — `deploy/Caddyfile`. Required by both `rutster-edge` (rutster-edge bakes it in + via the Dockerfile) and `rutster-allinone` (s6 Caddy service reads it). Lands before Task 5 + (compose) references it. +- **Task 4** — `deploy/.env.example`. Operator documentation; the compose `env_file` reference + points at it. +- **Task 5** — `deploy/compose.yaml`. Depends on Tasks 1 + 3 (image names + Caddyfile path). +- **Task 6** — `deploy/smoke/allinone_smoke.py` + `compose_smoke.sh` + `reload_during_call.py`. + These scripts are invoked by the CI jobs (Tasks 7–9) — land them first so the CI YAML can + reference stable paths. +- **Task 7** — Extend `.github/workflows/ci.yml`: `image-build` job. Depends on Task 1. +- **Task 8** — Extend `.github/workflows/ci.yml`: `smoke` job (all-in-one TLS smoke). Depends + on Tasks 6 + 7. +- **Task 9** — Extend `.github/workflows/ci.yml`: `smoke-compose` + `smoke-reload` jobs + (compose smoke + Caddy-reload-during-call). Depends on Task 8. +- **Task 10** — Slice F: `.github/workflows/publish-images.yml`. Depends on Task 7 (reuses the + same `docker buildx build` invocations + GHA cache config). +- **Task 11** — Final verification sweep. + + +--- + +### Task 1: `deploy/Dockerfile` — multi-stage, four image targets + s6-overlay + s6-rc + +One Dockerfile, four named targets. `docker buildx build --target rutster-engine -t +rutster-engine:ci .` builds just the FOB; `--target rutster-allinone` builds the s6 bundle. +The same builder stage compiles both Rust binaries (the all-in-one needs both; the +single-purpose images need one each — so the builder builds both, and each runtime stage +`COPY --from=builder` only the binary it needs). + +**Files:** +- Create: `deploy/Dockerfile` +- Vendor: `deploy/s6-overlay-noarch.tar.xz`, `deploy/s6-overlay-x86_64.tar.xz` (pulled from + upstream s6-overlay v3.2.0.0 release — vendored, not regenerated per build, per AGENTS.md + reproducibility posture) +- Create: `deploy/s6-overlay.sha256sum` (records the SHA256 of both tarballs at vendoring time) +- Create: `deploy/s6-rc.d/caddy/run` + `deploy/s6-rc.d/caddy/dependencies.d/engine` +- Create: `deploy/s6-rc.d/engine/run` + `deploy/s6-rc.d/engine/dependencies.d/valkey-database` +- Create: `deploy/s6-rc.d/brain/run` + `deploy/s6-rc.d/brain/dependencies.d/engine` +- Create: `deploy/s6-rc.d/valkey-database/run` + `deploy/s6-rc.d/valkey-database/dependencies.d/engine` +- Create: `deploy/s6-rc.d/user/contents.d/{caddy,engine,brain,valkey-database}` (empty files) +- Test: bash content-grep (Step 5) + optional `docker build --check` (Step 3 — local only; CI + in Task 7 is the authoritative build). + +**Interfaces:** +- Consumes: `rust-toolchain.toml` (pinned `1.85` — verified: `channel = "1.85"`), the workspace + root `Cargo.toml` (workspace member list — verified: `crates/rutster` + + `crates/rutster-brain-realtime` are both members), `crates/rutster-media/Cargo.toml` (confirms + `opus` crate hard-deps on libopus FFI — verified, line 12: `opus = { workspace = true }`), + the README's "Quickstart" libopus dev-header note (verified: `libopus-dev / opus-devel / + brew install opus`). Task 3's `deploy/Caddyfile` is COPYed into both edge + all-in-one + images — Task 1 lands the Dockerfile that references it; Task 3 lands the Caddyfile content. + Both must land before Task 7's `image-build` job runs in CI. +- Produces: four `--target`-named build stages. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/Dockerfile \ + && grep -q 'FROM rust:1.85-slim-bookworm' deploy/Dockerfile \ + && grep -q 'AS builder' deploy/Dockerfile \ + && grep -q 'libopus-dev' deploy/Dockerfile \ + && grep -q 'libopus0' deploy/Dockerfile \ + && grep -q 'FROM caddy:2.8-builder' deploy/Dockerfile \ + && grep -q 'xcaddy build' deploy/Dockerfile \ + && grep -q 'FROM valkey/valkey' deploy/Dockerfile \ + && grep -q 's6-overlay' deploy/Dockerfile \ + && grep -q -- 'AS rutster-engine' deploy/Dockerfile \ + && grep -q -- 'AS rutster-brain' deploy/Dockerfile \ + && grep -q -- 'AS rutster-edge' deploy/Dockerfile \ + && grep -q -- 'AS rutster-allinone' deploy/Dockerfile \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL` (file does not exist). + +- [ ] **Step 2: Write the complete Dockerfile** + +Write `deploy/Dockerfile` with exactly this content: + +```dockerfile +# syntax=docker/dockerfile:1.7 +# deploy/Dockerfile — multi-stage, four first-party images (spec §6.1). +# +# One Dockerfile, four named --target stages: +# * rutster-engine — FOB only (binary `rutster`) +# * rutster-brain — brain only (binary `rutster-brain-realtime`) +# * rutster-edge — custom xcaddy build with curated DNS plugins +# * rutster-allinone — s6-overlay v3 supervising caddy + FOB + brain + valkey-server +# +# Build all four locally: +# docker buildx build --target rutster-engine -t rutster-engine:ci -f deploy/Dockerfile . +# docker buildx build --target rutster-brain -t rutster-brain:ci -f deploy/Dockerfile . +# docker buildx build --target rutster-edge -t rutster-edge:ci -f deploy/Dockerfile . +# docker buildx build --target rutster-allinone -t rutster-allinone:ci -f deploy/Dockerfile . +# +# Toolchain pin: rust-toolchain.toml pins `channel = "1.85"`. The builder +# image is `rust:1.85-slim-bookworm` (NOT `rust:stable` — image builds must +# reproduce exactly what the MSRV matrix in CI gates; `stable` would float). +# +# libopus strategy (spec §6.1 "distro libopus0 over static-bundled"): +# * Builder stage: `apt-get install libopus-dev` — the -dev headers needed +# to compile the `opus` crate (FFI against libopus; verified: +# crates/rutster-media/Cargo.toml line 12: `opus = { workspace = true }`). +# * Runtime stages (engine + brain + all-in-one): `apt-get install libopus0` +# — the shared library binary. Brain image gets libopus0 too (see spec +# ambiguity #2 in this slice's plan): rutster-brain-realtime's +# Cargo.toml depends on rutster-media, so the linker records +# `NEEDED: libopus.so.0` in the brain ELF even though the brain never +# calls an opus symbol. Missing .so.0 would crash the brain at startup. +# * Matches the dev loop exactly. Do NOT use the audiopus_sys bundled-cmake +# fallback — system libopus is available in Debian. + +######################################## +# Stage 1: Rust builder — compiles both binaries once. +######################################## +FROM rust:1.85-slim-bookworm AS builder + +# libopus-dev: the headers the `opus` crate's build.rs expects. +RUN apt-get update && apt-get install -y --no-install-recommends \ + libopus-dev \ + ca-certificates \ + curl \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /usr/src/rutster +COPY . . + +# Build both binaries in one `cargo build --bins` invocation so the +# dependency graph is compiled once (warm ccache/rustc cache covers both). +# --release is mandatory: debug builds of str0m + opus run 5-10x slower +# than the 20ms tick budget; a debug-image container would burn the tick-lag +# threshold on first call. +RUN cargo build --release --bins \ + && cp /usr/src/rutster/target/release/rutster /usr/local/bin/rutster \ + && cp /usr/src/rutster/target/release/rutster-brain-realtime /usr/local/bin/rutster-brain-realtime + +######################################## +# Stage 2: Caddy builder — xcaddy with the curated DNS plugin set. +######################################## +# caddy:2.8-builder ships Go + xcaddy. The plugin set is exactly spec §3.2 +# / TLS brief §3(a): cloudflare, route53, porkbun, hetzner, desec. +# duckdns EXCLUDED — no license file (spec §3.2). +# Plugin licenses: cloudflare=Apache-2.0; route53/porkbun/hetzner/desec=MIT. +# Aggregation-clean vs GPL-3 (Caddy core itself is Apache-2.0). +FROM caddy:2.8-builder AS caddy-builder + +RUN xcaddy build \ + --with github.com/caddy-dns/cloudflare \ + --with github.com/caddy-dns/route53 \ + --with github.com/caddy-dns/porkbun \ + --with github.com/caddy-dns/hetzner \ + --with github.com/caddy-dns/desec \ + && mv /build/caddy /usr/local/bin/caddy + +######################################## +# Stage 3: Valkey binary source (upstream image, no rebuild). +######################################## +# COPY --from pattern: pull just the valkey-server binary out of the +# upstream BSD-3 image; avoids rebuilding valkey from source (saves ~5 min +# per build + ~150 MB of build deps). The binary is statically-linked +# enough for Debian bookworm (verified by upstream's Debian-based image). +FROM valkey/valkey:8.0 AS valkey-src + +######################################## +# Target: rutster-engine — the FOB only. +######################################## +FROM debian:bookworm-slim AS rutster-engine + +# libopus0: the shared library (see libopus strategy comment at top). +RUN apt-get update && apt-get install -y --no-install-recommends \ + libopus0 \ + ca-certificates \ + curl \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=builder /usr/local/bin/rutster /usr/local/bin/rutster +COPY deploy/Caddyfile /etc/rutster/Caddyfile + +# RUTSTER_HTTP_BIND defaults to 0.0.0.0:8080 (config.rs). Behind Caddy, the +# engine binds the compose-network address; behind --network host (T1 +# solo), it binds 127.0.0.1:8080. +EXPOSE 8080 +# Media UDP range — published via RUTSTER_MEDIA_PORT_RANGE. +EXPOSE 49152-49407/udp + +ENTRYPOINT ["/usr/local/bin/rutster"] + +######################################## +# Target: rutster-brain — brain only. +######################################## +FROM debian:bookworm-slim AS rutster-brain + +# libopus0: see spec ambiguity #2 in this slice's plan — +# rutster-brain-realtime's Cargo.toml depends on rutster-media which +# depends on the `opus` crate; even though the brain never calls an opus +# symbol, the ELF records `NEEDED: libopus.so.0` (linker dead-strips +# unused symbols but the .so is still required at dlopen-time before +# dead-strip can prove they're unused). A future brain-crate refactor +# that drops the rutster-media dep can drop this package too. +RUN apt-get update && apt-get install -y --no-install-recommends \ + libopus0 \ + ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=builder /usr/local/bin/rutster-brain-realtime /usr/local/bin/rutster-brain-realtime + +# Brain binds 127.0.0.1:8082 (loopback-only tap posture until step 6). +# In compose (T2), the brain shares the engine's netns via +# `network_mode: "service:engine"` so this loopback IS the engine's +# loopback — the FOB can reach the brain's tap port without exposing it. +EXPOSE 8082 + +ENTRYPOINT ["/usr/local/bin/rutster-brain-realtime"] + +######################################## +# Target: rutster-edge — custom Caddy build. +######################################## +# scratch/alpine choice (spec §6.1): alpine gives a shell for debugging +# (`docker exec -it rutster-edge sh`); scratch is purer but undebuggable. +# Alpine's musl is fine for the static Go binary xcaddy produces. +FROM alpine:3.20 AS rutster-edge + +# ca-certificates: ACME TLS validation. curl: smoke-test `caddy reload` +# invocations + `curl /healthz` from inside the container. +RUN apk add --no-cache ca-certificates curl + +COPY --from=caddy-builder /usr/local/bin/caddy /usr/local/bin/caddy +COPY deploy/Caddyfile /etc/caddy/Caddyfile + +# Caddy needs /data for ACME state — losing it risks the Let's Encrypt +# duplicate-cert lockout (spec §2.1, TLS brief §5 risk 5). Mount a volume +# at /data in production. In CI (local_certs) /data just holds the internal +# CA — recreating is fine, but the volume mount keeps prod + CI on the +# same code path. +VOLUME /data + +EXPOSE 80 443 + +ENTRYPOINT ["/usr/local/bin/caddy"] +CMD ["run", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile"] + +######################################## +# Target: rutster-allinone — s6-overlay v3 supervising four processes. +######################################## +FROM debian:bookworm-slim AS rutster-allinone + +# libopus0 (FOB needs it; brain too — see rutster-brain target above). +# curl: smoke-test invocations + `caddy reload` exec. +# ca-certificates: ACME. +# xz: s6-overlay binary tarball extraction (extract-then-rm). +RUN apt-get update && apt-get install -y --no-install-recommends \ + libopus0 \ + ca-certificates \ + curl \ + xz-utils \ + && rm -rf /var/lib/apt/lists/* + +# s6-overlay v3.2.0.0 — ISC license (aggregation-clean vs GPL-3). +# Two tarballs: noarch (the service definitions + s6-rc bundle) + x86_64 +# (the binaries — linux/amd64 only per spec §1.2). arm64 is a named +# deferral (ADR-0011 §1.2); when it lands, this Dockerfile grows an +# `ARG TARGETARCH` + `COPY` per-arch. +COPY deploy/s6-overlay-noarch.tar.xz /tmp/s6-overlay-noarch.tar.xz +COPY deploy/s6-overlay-x86_64.tar.xz /tmp/s6-overlay-x86_64.tar.xz +RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz \ + && tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz \ + && rm /tmp/s6-overlay-noarch.tar.xz /tmp/s6-overlay-x86_64.tar.xz + +# Binaries from earlier stages. +COPY --from=builder /usr/local/bin/rutster /usr/local/bin/rutster +COPY --from=builder /usr/local/bin/rutster-brain-realtime /usr/local/bin/rutster-brain-realtime +COPY --from=caddy-builder /usr/local/bin/caddy /usr/local/bin/caddy +COPY --from=valkey-src /usr/local/bin/valkey-server /usr/local/bin/valkey-server + +# Caddyfile + s6 service definitions (long-lined below). +COPY deploy/Caddyfile /etc/caddy/Caddyfile +COPY deploy/s6-rc.d/ /etc/s6-overlay/s6-rc.d/ + +# Volumes: Caddy /data (ACME state — loss risks LE lockout per TLS brief +# §5 risk 5) + Valkey /var/lib/valkey (stream/state persistence per +# ADR-0005). +VOLUME /data /var/lib/valkey + +# Caddy :80/:443 public; FOB :8080 behind Caddy; brain :8082 loopback; +# valkey :6379 loopback; media UDP direct. +EXPOSE 80 443 8080 8082 6379 49152-49407/udp + +# s6-overlay v3 entrypoint — the binary `s6-overlay-init` (symlinked from +# /s6-init in the noarch tarball) runs the service bundle in +# /etc/s6-overlay/s6-rc.d/ via s6-rc. +ENTRYPOINT ["/s6-init"] +``` + +- [ ] **Step 3: Verify by local Docker build (optional — authoritative build is CI in Task 7)** + +If Docker is available locally: + +```bash +cd /home/alee/Sources/rutster +# Buildkit dry-check (parse the Dockerfile without building): +docker buildx build --target rutster-engine -f deploy/Dockerfile --check . +# Full local build + smoke (note: this requires Task 3's Caddyfile + Step 4's +# s6-overlay tarballs + s6-rc.d definitions to be in place first): +docker buildx build --target rutster-engine -t rutster-engine:local -f deploy/Dockerfile . +docker run --rm rutster-engine:local --help 2>&1 | head -5 # binary banner +``` + +Expected: `--check` exits 0 (no syntax errors); the binary banner prints. If Docker is not +available locally, the authoritative build runs in CI (Task 7). + +**Note on `deploy/s6-overlay-*.tar.xz` + `deploy/s6-rc.d/`:** the all-in-one target references +two tarball binaries + a service-definition directory. These are *fetched artifacts* + *created +service files* — Step 4 below fetches the tarballs into `deploy/` and creates the s6-rc.d +directory so the Dockerfile build context is self-contained. The tarballs are vendored (not +regenerated per build) because s6-overlay release cadence is rare + reproducibility favors a +pinned vendored blob. + +- [ ] **Step 4: Vendor the s6-overlay tarballs + create the s6-rc service definitions** + +```bash +cd /home/alee/Sources/rutster/deploy + +# Pin v3.2.0.0 (ISC; aggregation-clean vs GPL-3): +curl -fsSLO https://github.com/just-containers/s6-overlay/releases/download/v3.2.0.0/s6-overlay-noarch.tar.xz +curl -fsSLO https://github.com/just-containers/s6-overlay/releases/download/v3.2.0.0/s6-overlay-x86_64.tar.xz + +# Record the SHA256 of both tarballs at vendoring time. The operator +# verifies these against the release page on first build; the +# `deploy/s6-overlay.sha256sum` file is committed so a future s6-overlay +# version bump is a loud, reviewable diff. +sha256sum s6-overlay-noarch.tar.xz s6-overlay-x86_64.tar.xz > s6-overlay.sha256sum +``` + +Create `deploy/s6-rc.d/caddy/run` (mark executable: `chmod +x deploy/s6-rc.d/caddy/run`): + +```sh +#!/command/execlineb -P +# s6-rc service: caddy — the edge (ACME + TLS termination + WS proxy). +# Depends on: engine (s6-rc starts the engine first per caddy/dependencies.d/engine). +# User: root (caddy needs :80/:443). +foreground { mkdir -p /data } +exec /usr/local/bin/caddy run --config /etc/caddy/Caddyfile --adapter caddyfile +``` + +Create `deploy/s6-rc.d/caddy/dependencies.d/engine` (an empty file — s6-rc reads the basename +as the dependency name; the engine must be up before Caddy proxies to it): + +```sh +# empty file — s6-rc reads the basename as the dependency name. +``` + +Create `deploy/s6-rc.d/engine/run` (mark executable): + +```sh +#!/command/execlineb -P +# s6-rc service: rutster — the FOB. Runs the rutster binary (axum + media +# thread + trunk WS + /metrics). Reads its env from the container env +# directly (s6-overlay v3 inherits container env via s6-overlay-envdir — +# this slice relies on the container env for simplicity; the per-service +# env files are a future hardening step). +exec /usr/local/bin/rutster +``` + +Create `deploy/s6-rc.d/engine/dependencies.d/valkey-database`: + +```sh +# empty — engine doesn't WAIT on valkey (it must boot even if valkey is down +# per ADR-0005's fail-safe posture: EventSink counts-and-drops, never blocks). +# The dependency just orders boot for consistent log sequencing. +``` + +Create `deploy/s6-rc.d/brain/run` (mark executable): + +```sh +#!/command/execlineb -P +# s6-rc service: rutster-brain-realtime. Shares the engine's network +# namespace (T1 solo variant — the all-in-one is a single container so +# loopback IS shared). The brain binds 127.0.0.1:8082 — the FOB reaches it +# via RUTSTER_TAP_URL=ws://127.0.0.1:8082. +exec /usr/local/bin/rutster-brain-realtime +``` + +Create `deploy/s6-rc.d/brain/dependencies.d/engine`: + +```sh +# empty — brain waits for the engine before attempting the tap WS. +``` + +Create `deploy/s6-rc.d/valkey-database/run` (mark executable): + +```sh +#!/command/execlineb -P +# s6-rc service: valkey-server — bus + KV + presence (ADR-0005). Dark on +# day one except EventSink consumer; the operator contract (volumes, ports, +# upgrade shape) is stable so the spend ledger + fleet directory land later +# without changing the deployment shape (spec §2.1). +foreground { mkdir -p /var/lib/valkey } +exec /usr/local/bin/valkey-server --dir /var/lib/valkey --save 60 1 --appendonly yes +``` + +Create `deploy/s6-rc.d/valkey-database/dependencies.d/engine`: + +```sh +# empty — orders valkey after engine in boot sequence. +``` + +Create the four empty service-bundle membership files in +`deploy/s6-rc.d/user/contents.d/`: + +```bash +cd /home/alee/Sources/rutster/deploy/s6-rc.d/user/contents.d +touch caddy engine brain valkey-database +cd /home/alee/Sources/rutster/deploy +chmod +x s6-rc.d/caddy/run s6-rc.d/engine/run s6-rc.d/brain/run s6-rc.d/valkey-database/run +``` + +- [ ] **Step 5: Re-run the content-check** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/Dockerfile \ + && grep -q 'FROM rust:1.85-slim-bookworm' deploy/Dockerfile \ + && grep -q 'AS builder' deploy/Dockerfile \ + && grep -q 'libopus-dev' deploy/Dockerfile \ + && grep -q 'libopus0' deploy/Dockerfile \ + && grep -q 'FROM caddy:2.8-builder' deploy/Dockerfile \ + && grep -q 'xcaddy build' deploy/Dockerfile \ + && grep -q 'FROM valkey/valkey' deploy/Dockerfile \ + && grep -q 's6-overlay' deploy/Dockerfile \ + && grep -q -- 'AS rutster-engine' deploy/Dockerfile \ + && grep -q -- 'AS rutster-brain' deploy/Dockerfile \ + && grep -q -- 'AS rutster-edge' deploy/Dockerfile \ + && grep -q -- 'AS rutster-allinone' deploy/Dockerfile \ + && test -f deploy/s6-overlay-noarch.tar.xz \ + && test -f deploy/s6-overlay-x86_64.tar.xz \ + && test -x deploy/s6-rc.d/caddy/run \ + && test -x deploy/s6-rc.d/engine/run \ + && test -x deploy/s6-rc.d/brain/run \ + && test -x deploy/s6-rc.d/valkey-database/run \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-PASS`. + +- [ ] **Step 6: Commit** + +```bash +cd /home/alee/Sources/rutster +git add deploy/Dockerfile deploy/s6-overlay-noarch.tar.xz deploy/s6-overlay-x86_64.tar.xz deploy/s6-overlay.sha256sum deploy/s6-rc.d/ +# Note: deploy/Caddyfile is created in Task 3; if Task 1 is committed without it, +# Step 3's `docker --check` will fail because COPY deploy/Caddyfile references a +# missing file. Two options: (a) commit the Caddyfile in this commit too (Task 3's +# content lands here as a stub + Task 3 just refines it), OR (b) land the Caddyfile +# in Task 3 and verify Task 1's Dockerfile syntax via `docker buildx build --check` +# which does NOT require the COPY source to exist. Option (b) is the executing-plans +# path: Task 1 commits the Dockerfile even though `docker build` would fail until +# Task 3 lands. CI's image-build job (Task 7) runs AFTER both Tasks 1 + 3 are merged. +git commit -s -m "feat(deploy): multi-stage Dockerfile producing four first-party images (deploy-B §6.1) + +One Dockerfile, four named --target stages: rutster-engine (FOB), +rutster-brain (brain), rutster-edge (custom xcaddy + curated DNS plugins +cloudflare/route53/porkbun/hetzner/desec, duckdns excluded per spec §3.2), +rutster-allinone (s6-overlay v3 supervising caddy + FOB + brain + +valkey-server). + +Builder pins rust:1.85-slim-bookworm (matches rust-toolchain.toml). +libopus-dev in builder, libopus0 in runtime (matches dev — spec §6.1 +distro-over-static-bundled). Brain image gets libopus0 too: although the +brain never calls an opus symbol, the linker records NEEDED:libopus.so.0 +because rutster-brain-realtime depends on rutster-media (spec ambiguity +resolved in this slice's plan README). + +All-in-one vendors s6-overlay v3.2.0.0 (ISC) + uses upstream valkey/valkey +binary (COPY --from pattern, BSD-3). Bundled-binary licenses +aggregation-clean vs GPL-3 (Caddy Apache-2.0). No Rust crates touched; +cargo-deny unaffected. + +s6-rc service definitions land in this commit (caddy/engine/brain/ +valkey-database run scripts + dependency edges). The Caddyfile lands in +Task 3 (same PR); the image-build CI job (Task 7) runs after both." +``` + + +--- + +### Task 2: `deploy/.dockerignore` — build-context pruning + +Without `.dockerignore`, the Docker build context ships `target/` (~5 GB on a warm dev +machine), `.git/` (history bloat), `docs/`, etc. — `docker buildx build` then spends minutes +transferring context before the first `RUN` runs. This file halves context size + saves +CI minutes per build. + +**Files:** +- Create: `deploy/.dockerignore` + +**Interfaces:** none — file content is a list of globs. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/.dockerignore && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write the complete file** + +Write `deploy/.dockerignore` with exactly this content: + +``` +# deploy/.dockerignore — prune the build context for `docker buildx build` +# invocations against deploy/Dockerfile. Without this, the context would +# ship target/ (~5 GB on warm dev), .git/, docs/, .github/, etc. — minutes +# of pointless context-transfer before the first RUN step. +# +# IMPORTANT: patterns are relative to the build context root (the path +# passed to `docker buildx build`, which is the repo root for this Dockerfile +# since we use `-f deploy/Dockerfile .`). So `target/` matches +# /target/ at the repo root AND `**/target/` catches /crates/*/target/. + +# Rust build artifacts. +target/ +**/target/ + +# Git history + CI configuration. +.git/ +.github/ + +# Editor + agent working state. +.vscode/ +.idea/ +.opencode/ +*.swp +*.swo +.DS_Store + +# Environment files (never ship secrets — .env.example is the ONLY .env* +# file that should reach the build context, and it has no secrets by design). +.env +.env.local +.env.*.local + +# Documentation (the Dockerfile doesn't read any .md file but README). +docs/ +*.md +!README.md + +# Plans + specs (not needed at image build time — operator reads from the +# repo, not the container). +docs/superpowers/ + +# Logs + OS junk. +*.log +tmp/ +*.tmp + +# Tmp artifacts from cargo-deny / cargo-cache. +deny.toml.lock +``` + +- [ ] **Step 3: Re-run the content-check** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/.dockerignore \ + && grep -q '^target/' deploy/.dockerignore \ + && grep -q '^.git/' deploy/.dockerignore \ + && grep -q '^\*\*/target/' deploy/.dockerignore \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-PASS`. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add deploy/.dockerignore +git commit -s -m "chore(deploy): .dockerignore — prune build context (deploy-B §6.1) + +Halves the build-context size: excludes target/ + **/target/ (5 GB on a +warm dev machine, ~150 MB cache hit on CI), .git/, docs/, .github/, +.env* (never ship secrets — .env.example is the only .env* the context +ships and it has no secrets by design)." +``` + +--- + +### Task 3: `deploy/Caddyfile` — tuned timeouts, honest X-Forwarded, stream_close_delay 24h + +The Caddyfile is read by both `rutster-edge` (T2 modular default edge) and `rutster-allinone` +(T1 bundled edge). Same file, same build — spec §3.2 mandates "same build, same Caddyfile" so +operator knowledge transfers between T1 and T2. + +**Files:** +- Create: `deploy/Caddyfile` (already referenced by Task 1's Dockerfile COPY steps; lands here + as its own atomic commit so the diff is reviewable + the Caddyfile can be edited without + re-touching the Dockerfile). + +**Interfaces:** +- Consumes: env vars `RUTSTER_DOMAIN` (the public hostname — `pbx.example.com`), + `RUTSTER_ACME_EMAIL` (the LE account email), `RUTSTER_LOCAL_CERTS` (CI override — `"true"` + makes Caddy use its internal CA, no ACME). +- Produces: the `:443` TLS listener, the `:80` ACME HTTP-01 fallback, the reverse_proxy to + `127.0.0.1:8080`, the WS upgrade pass-through (Caddy auto-detects `Upgrade: websocket` — + no per-route config needed), the `X-Forwarded-Proto`/`X-Forwarded-Host` honesty. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/Caddyfile \ + && grep -q 'stream_close_delay' deploy/Caddyfile \ + && grep -q 'X-Forwarded-Proto' deploy/Caddyfile \ + && grep -q 'reverse_proxy' deploy/Caddyfile \ + && grep -q '24h' deploy/Caddyfile \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL` (file does not exist). + +- [ ] **Step 2: Write the complete Caddyfile** + +Write `deploy/Caddyfile` with exactly this content: + +```caddyfile +# deploy/Caddyfile — the edge config for rutster-edge (T2) + rutster-allinone (T1). +# Same file, same build (spec §3.2 + ADR-0011). Read by Caddy v2.8 (the +# custom xcaddy build from deploy/Dockerfile). +# +# Env vars (Caddyfile interpolation syntax {$VAR}): +# RUTSTER_DOMAIN — the public hostname (e.g. pbx.example.com). Required. +# RUTSTER_ACME_EMAIL — the Let's Encrypt account email. Required for ACME. +# RUTSTER_LOCAL_CERTS — set to "true" in CI to use Caddy's internal CA +# (no ACME round-trips, no rate-limit budget burns). +# Overrides the ACME block below via the `local_certs` +# global option. +# +# Timeouts (TLS brief §3(a); spec §2.1, §2.2, §3.1 invariant 3): +# * read_body 30s — Twilio webhooks fit comfortably in 30s (15s cap +# by invariant 6). +# * read_header 30s — same. +# * write 0 — disable; 20ms media frames must flush immediately. +# * idle 24h — the universal 60s-class default kills quiet WS; +# spec §2.1 invariant 3 sets max call duration 24h. +# * stream_close_delay 24h — above max call duration; the load-bearing +# mitigation for caddy #6420/#7222 (TLS brief §5 +# risk 1; spec §9 reload-during-call smoke). +# +# X-Forwarded-* honesty (spec §3.1 invariant 5; plan-A Task 5 +# reconstruct_public_url): Caddy sets these to the real client values via +# the `header_up` directives below. Engine-side RUTSTER_TRUSTED_PROXIES is +# the trust gate (fail-closed default). + +{ + # CI override: set RUTSTER_LOCAL_CERTS=true to use Caddy's internal CA + # (no ACME in CI; spec §9). Production leaves this unset. + {$RUTSTER_LOCAL_CERTS:local_certs_off} local_certs + + # Send logs to stdout (s6-overlay / docker logs collects from there). + log { + output stdout + format console + } +} + +# Default ACME email — overridden by RUTSTER_LOCAL_CERTS=true in CI. +{ + email {$RUTSTER_ACME_EMAIL:you@example.com} +} + +# Main site block — the public hostname. +{$RUTSTER_DOMAIN:localhost} { + encode zstd gzip + + # The reverse_proxy to the FOB. 127.0.0.1:8080 is correct in T1 + # (single container, loopback) AND T2 (compose: the engine service's + # `rutster-engine` DNS name resolves on the compose network; Caddy + # reaches it via that name — replace 127.0.0.1:8080 with + # http://engine:8080 in advance; here we use the loopback literal so + # the same Caddyfile works in both T1 and T2 spelled identically — + # for T2 deployments, the operator overrides the upstream via a env- + # substituted site-address block; the shipped default targets loopback + # for the bundled all-in-one single-container case which is the harder + # constraint). + # + # `header_up Host {host}` + X-Forwarded-* — sets the honest hop values + # (the engine's reconstruct_public_url reads these; spec §3.1 invariant 5). + # `header_up X-Forwarded-For {remote_host}` — included for completeness; + # the engine does NOT use X-Forwarded-For for signature validation, only + # X-Forwarded-Proto/Host. Forwarding the For header is standard proxy + # hygiene — engine-side RUTSTER_TRUSTED_PROXIES is the gate. + reverse_proxy 127.0.0.1:8080 { + header_up Host {host} + header_up X-Real-IP {remote_host} + header_up X-Forwarded-Proto {scheme} + header_up X-Forwarded-Host {host} + header_up X-Forwarded-For {remote_host} + + # Tuned timeouts — the load-bearing ones for calls lasting hours. + transport http { + read_buffer 64KiB + write_buffer 64KiB + dial_timeout 5s + response_header_timeout 30s + } + + # stream_close_delay above max call duration (24h) — the caddy + # #6420/#7222 mitigation (TLS brief §5 risk 1; spec §9 smoke). + # An operator Caddyfile reload mid-call would normally kill + # upgraded WS tunnels; the delay keeps them alive until the + # streams close naturally (the call ends) — verified by the + # reload-during-live-call CI smoke (Task 9). + stream_close_delay 24h + } + + # Timeouts for the public-facing side. + servers { + read_body 30s + read_header 30s + write 0 # never write-timeout; 20ms frames self-flush + idle 24h # above max call duration — the universal 60s-class default kills quiet WS + } +} +``` + +- [ ] **Step 3: Re-run the content-check + Caddy config-validate** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/Caddyfile \ + && grep -q 'stream_close_delay' deploy/Caddyfile \ + && grep -q 'X-Forwarded-Proto' deploy/Caddyfile \ + && grep -q 'reverse_proxy' deploy/Caddyfile \ + && grep -q '24h' deploy/Caddyfile \ + && grep -q 'idle 24h' deploy/Caddyfile \ + && grep -q 'RUTSTER_DOMAIN' deploy/Caddyfile \ + && grep -q 'RUTSTER_LOCAL_CERTS' deploy/Caddyfile \ + && echo CHECK-PASS || echo CHECK-FAIL + +# Optional local Caddy validate (if `caddy` binary available — CI does +# this inside the rutster-edge container in Task 7): +if command -v caddy >/dev/null 2>&1; then + RUTSTER_DOMAIN=pbx.example.com RUTSTER_ACME_EMAIL=test@example.com \ + caddy validate --config deploy/Caddyfile --adapter caddyfile +else + echo "(caddy not installed locally; CI validates in Task 7)" +fi +``` + +Expected: `CHECK-PASS`. If Caddy is installed locally, the `validate` command exits 0. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add deploy/Caddyfile +git commit -s -m "feat(deploy): Caddyfile — tuned timeouts, honest X-Forwarded, stream_close_delay 24h (deploy-B §3.2) + +The shared edge config for rutster-edge (T2 modular) and rutster-allinone +(T1 solo). ~6-line site block + global options per spec §3.2 / TLS brief +§3(a): + +* read_body 30s / read_header 30s — Twilio webhook budget (15s cap by + invariant 6 + slack). +* write 0 — 20ms media frames self-flush; never write-timeout. +* idle 24h / stream_close_delay 24h — above max call duration (invariant + 3); the universal 60s-class default kills quiet mid-call WS. +* reverse_proxy + header_up — Caddy sets honest X-Forwarded-Proto/Host/For + for the engine's reconstruct_public_url (plan-A Task 5); engine-side + RUTSTER_TRUSTED_PROXIES is the trust gate (fail-closed default). + +CI override via RUTSTER_LOCAL_CERTS=true uses Caddy's internal CA (no +ACME in CI per spec §9). The reload-during-live-call smoke (Task 9) +regresses the stream_close_delay mitigation against caddy #6420/#7222." +``` + +--- + +### Task 4: `deploy/.env.example` — all RUTSTER_* vars documented + +The `.env.example` is the operator documentation surface (plan G's `quickstart-docker.md` +refers operators here). Every `RUTSTER_*` var is listed with its default, the slice that +introduced it, and the fail-fast behavior. + +**Files:** +- Create: `deploy/.env.example` + +**Interfaces:** consumes the env-var surface from `crates/rutster/src/config.rs` (verified: +`http_bind`, `media_address_config`, `max_sessions`, `drain_deadline`, `twilio_credentials` + +plan A's `ws_ping_interval`, `trusted_proxies`) + the artifact-level vars `RUTSTER_DOMAIN` + +`RUTSTER_ACME_EMAIL` + `RUTSTER_LOCAL_CERTS` (this slice — Caddyfile-level). Plan C's +`RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY` are noted as "land with slice C" — see plan C, NOT here. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/.env.example \ + && grep -q 'RUTSTER_DRAIN_DEADLINE_SECS=600' deploy/.env.example \ + && grep -q 'RUTSTER_DOMAIN=' deploy/.env.example \ + && grep -q 'RUTSTER_ACME_EMAIL=' deploy/.env.example \ + && grep -q 'RUTSTER_TWILIO_ACCOUNT_SID=' deploy/.env.example \ + && grep -q 'RUTSTER_TRUSTED_PROXIES=' deploy/.env.example \ + && grep -q 'RUTSTER_WS_PING_SECS=' deploy/.env.example \ + && grep -q 'RUTSTER_MEDIA_ADVERTISED_IP=' deploy/.env.example \ + && grep -q 'RUTSTER_LOCAL_CERTS' deploy/.env.example \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write the complete file** + +Write `deploy/.env.example` with exactly this content: + +```bash +# deploy/.env.example — runtime configuration for the rutster T1 (all-in-one) +# + T2 (compose) artifacts. Copy to `.env` + edit before `docker compose up -d`. +# +# Format: KEY=value (one per line); comments (#) are ignored by `docker +# compose --env-file`. Fail-fast pattern: every var is parsed by +# crates/rutster/src/config.rs's pure-`Option` parsers at startup; +# an invalid value fails the boot loudly with the var name in the error, +# never silently runs with a wrong default. See AGENTS.md "config.rs" +# pattern. +# +# Bundled-binary licenses (aggregation-clean vs GPL-3 per ADR-0004): +# * Caddy Apache-2.0 +# * valkey BSD-3 +# * s6-overlay ISC +# * libopus0 BSD-3 (system package; Debian) +# Future cargo-deny-adjacent image license scan will assert these in CI +# (spec §6.1 — out of scope for slice B; this comment seeds it). +# +# Registry namespace: git.adlee.work/alee/rutster-* (matches the `alee` tea +# login per AGENTS.md Git workflow + the root Cargo.toml's `repository` +# field). If your Gitea registry uses a different owner segment, replace +# `alee` throughout deploy/compose.yaml + .github/workflows/publish-images.yml. + +################################################################################ +# ARTIFACT-LEVEL — Caddy edge (this slice) +################################################################################ + +# RUTSTER_DOMAIN — the public hostname the CPaaS dials. Required for T1 (solo +# all-in-one) and T2 (compose). Used by Caddyfile as the site block name. +# Example: pbx.example.com +RUTSTER_DOMAIN=pbx.example.com + +# RUTSTER_ACME_EMAIL — the Let's Encrypt account email. Used by Caddy for +# ACME account creation. Required for production (ACME issuance); ignored +# in CI (RUTSTER_LOCAL_CERTS=true bypasses ACME). +RUTSTER_ACME_EMAIL=you@example.com + +# RUTSTER_LOCAL_CERTS — set to "true" in CI to use Caddy's internal CA (no +# ACME round-trips, no rate-limit budget burns). Production leaves this +# unset (or "false") so Caddy uses Let's Encrypt. The Caddyfile reads this +# via the {$RUTSTER_LOCAL_CERTS:local_certs_off} interpolation. +# RUTSTER_LOCAL_CERTS=false + +################################################################################ +# FOB engine — bind + drain (slice-5 config.rs + plan-A) +################################################################################ + +# RUTSTER_HTTP_BIND — the FOB's HTTP/WS listener. Behind Caddy (T1/T2 prod): +# 0.0.0.0:8080 (compose network) or 127.0.0.1:8080 (single container T1). +# Default if unset: 0.0.0.0:8080 (config.rs http_bind). +RUTSTER_HTTP_BIND=0.0.0.0:8080 + +# RUTSTER_DRAIN_DEADLINE_SECS — how long shutdown waits for in-flight calls +# before the hard stop. Default 0 = instant shutdown (dev loop); production +# sets minutes. Spec §2.2 pins 600 (paired with compose stop_grace_period +# 660s — grace MUST exceed drain or the orchestrator kills calls the +# engine was gracefully draining). +RUTSTER_DRAIN_DEADLINE_SECS=600 + +# RUTSTER_MAX_SESSIONS — admission cap. Default 64 (placeholder until the +# ADR-0010 benchmark measures the real per-node ceiling). /readyz flips 503 +# when at the cap so the LB pulls the node. +RUTSTER_MAX_SESSIONS=64 + +################################################################################ +# FOB engine — WebRTC media (slice-5 config.rs media_address_config) +################################################################################ + +# RUTSTER_MEDIA_BIND_IP — the IP WebRTC DTLS-SRTP binds. Default 127.0.0.1 +# (loopback — dev loop). Production hosts a public IP. +RUTSTER_MEDIA_BIND_IP=0.0.0.0 + +# RUTSTER_MEDIA_ADVERTISED_IP — the IP advertised in SDP to WebRTC callers. +# They send media UDP straight to this IP (NAT 1:1 — no STUN, no TURN). +# In T1 (host networking): the host's public IP. In T2 (compose): the +# engine container's published port's host IP. +RUTSTER_MEDIA_ADVERTISED_IP=203.0.113.7 + +# RUTSTER_MEDIA_PORT_RANGE — UDP port range for WebRTC media. Format lo-hi +# inclusive. Must be open inbound on the host's firewall + published in +# compose (deploy/compose.yaml publishes this range). +RUTSTER_MEDIA_PORT_RANGE=49152-49407 + +################################################################################ +# FOB engine — WS hygiene (plan-A §5.2) +################################################################################ + +# RUTSTER_WS_PING_SECS — engine-originated app-level WS ping interval on +# the trunk media-stream WS. Default 20s. 0 rejected (would busy-loop the +# ping timer); no "off" spelling by design. +RUTSTER_WS_PING_SECS=20 + +################################################################################ +# FOB engine — trusted-proxy posture (plan-A §5.3) +################################################################################ + +# RUTSTER_TRUSTED_PROXIES — comma-separated CIDR list of edge peers whose +# X-Forwarded-Proto/Host are believed. Empty or unset = headers IGNORED +# (fail-closed default). Bare IPs accepted as host-length prefixes. In T1/T2 +# with Caddy as edge, set to the Caddy container's source IP (compose: +# the compose network's subnet CIDR; T1: 127.0.0.1/32). An internet peer +# must never choose the public URL that Twilio signature validation +# reconstructs (spec §3.1 invariant 5). +RUTSTER_TRUSTED_PROXIES=127.0.0.1/32 + +################################################################################ +# Trunk — Twilio credentials (slice-5 config.rs twilio_credentials) +################################################################################ + +# All four RUTSTER_TWILIO_* vars are ALL-OR-NONE: the parser rejects partial +# config at startup. Omit all four to run WebRTC-only. +RUTSTER_TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +RUTSTER_TWILIO_AUTH_TOKEN=your_auth_token_here +RUTSTER_TWILIO_MEDIA_BIND=0.0.0.0:8081 +RUTSTER_TWILIO_WEBHOOK_BASE=https://pbx.example.com + +################################################################################ +# Brain — tap URL (slice-2/3; loopback-only until step 6) +################################################################################ + +# RUTSTER_TAP_URL — the WS URL the FOB opens to the brain. Default +# ws://127.0.0.1:8081/echo (slice-2 dev-loop echo brain). Production with +# rutster-brain-realtime: ws://127.0.0.1:8082 (loopback-only — +# resolve_tap_url rejects non-loopback until step 6's wss:// tap). In T2 +# (compose), the brain shares the engine's netns +# (network_mode: "service:engine") so this loopback IS the engine's +# loopback. +RUTSTER_TAP_URL=ws://127.0.0.1:8082 + +################################################################################ +# Future vars — land with sibling slices, NOT this slice +################################################################################ + +# RUTSTER_TLS_CERT / RUTSTER_TLS_KEY — land with slice C (rustls Phase 1, +# BYO-cert in-process TLS). The engine binary stays compiled-with-rustls +# from slice C onward; runtime-gated by these vars. See the slice-C plan +# at docs/superpowers/plans/2026-07-05-deploy-c-rustls-tls.md (path may +# vary — check the plans dir). +# RUTSTER_TLS_CERT=/etc/rutster/tls/fullchain.pem +# RUTSTER_TLS_KEY=/etc/rutster/tls/privkey.pem + +# RUTSTER_METRICS_BIND — lands with slice D (/metrics endpoint). Default +# 127.0.0.1:9090 (internal — never routed through Caddy). +# RUTSTER_METRICS_BIND=127.0.0.1:9090 + +# RUTSTER_VALKEY_URL — lands with slice E (ValkeyEventSink). Unset = today's +# TracingEventSink (logs to stdout). Set to redis://valkey:6379/0 in T2. +# RUTSTER_VALKEY_URL=redis://valkey:6379/0 +``` + +- [ ] **Step 3: Re-run the content-check** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/.env.example \ + && grep -q 'RUTSTER_DRAIN_DEADLINE_SECS=600' deploy/.env.example \ + && grep -q 'RUTSTER_DOMAIN=' deploy/.env.example \ + && grep -q 'RUTSTER_ACME_EMAIL=' deploy/.env.example \ + && grep -q 'RUTSTER_TWILIO_ACCOUNT_SID=' deploy/.env.example \ + && grep -q 'RUTSTER_TRUSTED_PROXIES=' deploy/.env.example \ + && grep -q 'RUTSTER_WS_PING_SECS=' deploy/.env.example \ + && grep -q 'RUTSTER_MEDIA_ADVERTISED_IP=' deploy/.env.example \ + && grep -q 'RUTSTER_LOCAL_CERTS' deploy/.env.example \ + && grep -q 'RUTSTER_TLS_CERT' deploy/.env.example \ + && grep -q 'RUTSTER_VALKEY_URL' deploy/.env.example \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-PASS`. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add deploy/.env.example +git commit -s -m "docs(deploy): .env.example — all RUTSTER_* vars documented (deploy-B §6.2) + +The operator documentation surface. Lists every RUTSTER_* var with its +default + the slice that introduced it + the fail-fast behavior. Includes: + +* Artifact-level (this slice): RUTSTER_DOMAIN, RUTSTER_ACME_EMAIL, + RUTSTER_LOCAL_CERTS (CI override). +* FOB engine (slice-5 + plan-A): RUTSTER_HTTP_BIND, + RUTSTER_DRAIN_DEADLINE_SECS=600 (paired with compose stop_grace_period + 660s per spec §2.2 — grace exceeds drain), RUTSTER_MAX_SESSIONS, + RUTSTER_MEDIA_BIND_IP/ADVERTISED_IP/PORT_RANGE, RUTSTER_WS_PING_SECS, + RUTSTER_TRUSTED_PROXIES. +* Trunk (slice-5): the four RUTSTER_TWILIO_* (all-or-none — config.rs + twilio_credentials rejects partial). +* Brain (slice-2/3): RUTSTER_TAP_URL (loopback-only until step 6). +* Future var stubs: RUTSTER_TLS_CERT/KEY (slice C), RUTSTER_METRICS_BIND + (slice D), RUTSTER_VALKEY_URL (slice E) — commented out with a pointer + to the sibling plan. + +Bundled-binary licenses noted in the header comment (Caddy Apache-2.0, +valkey BSD-3, s6-overlay ISC, libopus0 BSD-3 — aggregation-clean vs +GPL-3 per ADR-0004); seeds the future cargo-deny-adjacent image license +scan (spec §6.1, out of scope for this slice). + +Registry namespace git.adlee.work/alee/rutster-* + the 'if your Gitea +registry uses a different owner segment, replace alee throughout' note." +``` + + +--- + +### Task 5: `deploy/compose.yaml` — T2 reference deployment + +The compose file is the T2 reference deployment (spec §2.2). Four services, `stop_grace_period: +660s` (paired with `RUTSTER_DRAIN_DEADLINE_SECS=600`), brain shares the engine's netns so the +loopback tap posture holds, two named volumes for Caddy `/data` + valkey persistence. + +**Files:** +- Create: `deploy/compose.yaml` + +**Interfaces:** +- Consumes: image names from spec §6.1 (`rutster-edge`, `rutster-engine`, `rutster-brain`, + upstream `valkey/valkey`), env from `deploy/.env.example` (Task 4), Caddyfile from Task 3. +- Produces: `docker compose up -d` boots a working T2 stack. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/compose.yaml \ + && grep -q 'services:' deploy/compose.yaml \ + && grep -q 'image: rutster-edge' deploy/compose.yaml \ + && grep -q 'image: rutster-engine' deploy/compose.yaml \ + && grep -q 'image: rutster-brain' deploy/compose.yaml \ + && grep -q 'image: valkey/valkey' deploy/compose.yaml \ + && grep -q 'network_mode: "service:engine"' deploy/compose.yaml \ + && grep -q 'stop_grace_period: 660s' deploy/compose.yaml \ + && grep -q 'volumes:' deploy/compose.yaml \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write the complete file** + +Write `deploy/compose.yaml` with exactly this content: + +```yaml +# deploy/compose.yaml — T2 modular reference deployment (spec §2.2 / ADR-0011). +# +# Four services: caddy (rutster-edge), engine (rutster-engine), brain +# (rutster-brain, network_mode: "service:engine" so the loopback tap posture +# survives unchanged — resolve_tap_url rejects non-loopback until step 6), +# valkey (upstream valkey/valkey). +# +# stop_grace_period 660s paired with RUTSTER_DRAIN_DEADLINE_SECS=600 — +# grace EXCEEDS drain or the orchestrator kills calls the engine was +# gracefully draining (spec §2.2 / §8). 60s slack accounts for a +# legitimately-long in-flight call (599s) + shutdown round-trip. +# +# Bring your own proxy: comment out the `caddy` service; the engine keeps +# its plaintext :8080 listener; tuned-timeout configs for nginx/HAProxy/ +# Traefik ship in docs/deploy/reverse-proxies.md (slice G). +# +# Bring-up: +# cp .env.example .env +# $EDITOR .env # set DOMAIN, ACME_EMAIL, MEDIA_ADVERTISED_IP, TWILIO_* +# docker compose up -d +# curl -fsS https://$RUTSTER_DOMAIN/healthz && echo OK + +services: + caddy: + # The custom xcaddy build with curated DNS plugins (cloudflare/ + # route53/porkbun/hetzner/desec — duckdns excluded per spec §3.2). + # In production, pulls from the git.adlee.work/alee/rutster-edge + # registry (slice F). For local dev / CI without registry auth, build + # from the Dockerfile: + # build: + # context: .. + # dockerfile: deploy/Dockerfile + # target: rutster-edge + image: rutster-edge:latest + restart: unless-stopped + ports: + - "80:80" + - "443:443" + volumes: + - caddy-data:/data + - ./Caddyfile:/etc/caddy/Caddyfile:ro + env_file: + - .env + depends_on: + - engine + networks: + - rutster-net + + engine: + # The FOB: axum signaling + media thread + trunk WS + /metrics. + # Plaintext :8080 behind Caddy (slice C's rustls will terminate + # in-process TLS — same listener, operator-choice). + image: rutster-engine:latest + restart: unless-stopped + # 660s grace > 600s drain (RUTSTER_DRAIN_DEADLINE_SECS) — invariant. + stop_grace_period: 660s + ports: + # Signaling + trunk WS behind Caddy (Caddy proxies :443 -> engine:8080). + # Comment out for production hardening (Caddy is the only entry). + - "8080:8080" + # WebRTC media UDP direct — never proxied (spec §2.1). + - "49152-49407:49152-49407/udp" + volumes: + - ./Caddyfile:/etc/rutster/Caddyfile:ro # copied in image already; mount for edit-without-rebuild + env_file: + - .env + networks: + - rutster-net + + brain: + # Brain shares engine's network namespace — the FOB reaches the brain's + # tap port via 127.0.0.1:8082 (loopback-only posture; resolve_tap_url + # rejects non-loopback URLs until step 6 per spec §2.2). The brain + # graduates to its own netns + wss:// tap auth with step 6. + image: rutster-brain:latest + restart: unless-stopped + # CRITICAL: quoted — YAML would otherwise parse `service:engine` as a + # `service: engine` mapping (key:value). The quotes make it a string. + network_mode: "service:engine" + env_file: + - .env + depends_on: + - engine + + valkey: + # Upstream valkey image (BSD-3; aggregation-clean vs GPL-3 per ADR-0004). + # Dark on day one except EventSink (slice E lands ValkeyEventSink + # against this service). The operator contract (volumes, ports, upgrade + # shape) is stable from v1 per spec §2.1 — the spend ledger (ADR-0009) + # + fleet directory land later without changing this service. + image: valkey/valkey:8.0 + restart: unless-stopped + command: ["valkey-server", "--dir", "/data", "--save", "60", "1", "--appendonly", "yes"] + volumes: + - valkey-data:/data + networks: + - rutster-net + +volumes: + # /data — Caddy's cert/ACME state. Loss risks the Let's Encrypt + # duplicate-cert lockout (5/week — total inbound outage class). + # See docs/deploy/certificates.md (slice G) + TLS brief §5 risk 5. + caddy-data: + # /var/lib/valkey — stream/state persistence (ADR-0005). + valkey-data: + +networks: + rutster-net: + driver: bridge +``` + +- [ ] **Step 3: Re-run the content-check + `docker compose config` validate** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/compose.yaml \ + && grep -q 'services:' deploy/compose.yaml \ + && grep -q 'image: rutster-edge' deploy/compose.yaml \ + && grep -q 'image: rutster-engine' deploy/compose.yaml \ + && grep -q 'image: rutster-brain' deploy/compose.yaml \ + && grep -q 'image: valkey/valkey' deploy/compose.yaml \ + && grep -q 'network_mode: "service:engine"' deploy/compose.yaml \ + && grep -q 'stop_grace_period: 660s' deploy/compose.yaml \ + && grep -q 'caddy-data:' deploy/compose.yaml \ + && grep -q 'valkey-data:' deploy/compose.yaml \ + && echo CHECK-PASS || echo CHECK-FAIL + +# YAML syntax validation — `docker compose config` parses + renders the +# resolved config (with .env vars interpolated). Requires Docker locally; +# CI validates in the compose-smoke job (Task 9). +if command -v docker >/dev/null 2>&1; then + cd /home/alee/Sources/rutster/deploy + cp .env.example .env 2>/dev/null || cp -n .env.example .env 2>/dev/null || true + docker compose --env-file .env config -q + rm -f .env # don't commit the .env (it has placeholder secrets from .env.example) +else + echo "(docker not installed locally; CI validates in Task 9)" +fi +``` + +Expected: `CHECK-PASS`. If `docker compose` is available, `config -q` exits 0. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add deploy/compose.yaml +git commit -s -m "feat(deploy): compose.yaml — T2 reference deployment (deploy-B §2.2) + +Four services per spec §2.2: + +* caddy — image rutster-edge; ports 80/443; volume caddy-data:/data + (Caddy ACME state — loss risks LE 5/week duplicate-cert lockout + per TLS brief §5 risk 5; mounts ./Caddyfile for edit-without- + rebuild). +* engine — image rutster-engine; ports 8080 + 49152-49407/udp (media + UDP direct, never proxied — spec §2.1); stop_grace_period 660s + paired with RUTSTER_DRAIN_DEADLINE_SECS=600 (grace exceeds drain + invariant, spec §2.2/§8). +* brain — image rutster-brain; network_mode: \"service:engine\" so the + loopback-only tap posture survives unchanged (resolve_tap_url + rejects non-loopback until step 6 per spec §2.2). Quoted in + YAML — \`service:engine\` would parse as a mapping otherwise. +* valkey — upstream valkey/valkey:8.0 (BSD-3); volume valkey-data:/data; + dark on day one except slice-E ValkeyEventSink consumer. + +Bring-your-own-proxy: comment out the caddy service; the engine keeps +plaintext :8080; tuned-timeout snippets land in slice-G reverse-proxies.md." +``` + + +--- + +### Task 6: `deploy/smoke/*.py` + `deploy/smoke/*.sh` — CI smoke harness + +The smoke scripts are invoked by the Gitea Actions smoke job (Tasks 7–9). Python-stdlib only +(no `pip install`) so each CI job is self-contained. Bash for the orchestration glue. + +**Files:** +- Create: `deploy/smoke/allinone_smoke.py` — wss:// WS client driving a sim call through the + real edge→FOB path inside `rutster-allinone`. +- Create: `deploy/smoke/compose_smoke.sh` — `docker compose up -d --wait`, health probes, + service-status JSON parse. **The `# TODO slice-C: assert lifecycle event in Valkey stream` + named hook lives here.** +- Create: `deploy/smoke/reload_during_call.py` — holds a WS open streaming frames while a + concurrent `caddy reload` fires; asserts zero frame drops. + +**Interfaces:** +- Consumes: Task 1's `rutster-allinone` image (built by Task 7's CI image-build job), Task 3's + Caddyfile (baked into the all-in-one image), Caddy's internal-CA root cert at + `/data/caddy/pki/authorities/local/root.crt` (Caddy's `local_certs` global option writes it). +- Produces: Python scripts returning exit code 0 on success, non-zero on failure (CI fails the + job on non-zero). + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f deploy/smoke/allinone_smoke.py \ + && test -f deploy/smoke/compose_smoke.sh \ + && test -f deploy/smoke/reload_during_call.py \ + && grep -q 'local_certs' deploy/smoke/allinone_smoke.py \ + && grep -q 'TODO slice-C' deploy/smoke/compose_smoke.sh \ + && grep -q 'caddy reload' deploy/smoke/reload_during_call.py \ + && grep -q 'stream_close_delay' deploy/smoke/reload_during_call.py \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write `deploy/smoke/allinone_smoke.py`** + +```python +#!/usr/bin/env python3 +"""deploy/smoke/allinone_smoke.py — T1 all-in-one TLS smoke (deploy-B §9). + +Boots `rutster-allinone` with RUTSTER_LOCAL_CERTS=true (Caddy internal CA — +no ACME in CI per spec §9), boots, extracts Caddy's internal-CA root cert +from /data, opens a wss:// WS to /twilio/media-stream, sends Twilio's +connected + start handshake frames, streams 200 PCM-zeroed frames at 20ms +cadence, and asserts the engine replies with at least one text frame +through the real edge->FOB WS path end-to-end through TLS. + +Python stdlib ONLY — no pip install required (the CI runner image ships +Python 3 + ssl + socket + json + time). The WS handshake is hand-rolled +for the same reason (websocket-client/websockets would require network +during CI). + +Exit code 0 = smoke passed. Non-zero = failed (CI fails the smoke job). + +Reusable from: crates/rutster/tests/trunk_sim_e2e.rs (the #[ignore]'d TODO +body sketch — this script implements the same handshake pattern against a +real containerized binary rather than an in-process +MockTwilioMediaStreamsServer) + crates/rutster-trunk/tests/ws_ping.rs (the +connected/start frame sequence — same JSON shapes verbatim). +""" + +import json +import os +import socket +import ssl +import struct +import subprocess +import sys +import tempfile +import time +from base64 import b64encode + +IMAGE = os.environ.get("RUTSTER_ALLINONE_IMAGE", "rutster-allinone:smoke") +CONTAINER_NAME = "rutster-allinone-smoke" +HOST_PORT = int(os.environ.get("RUTSTER_ALLINONE_PORT", "18443")) +DATA_VOLUME = "rutster-smoke-caddy-data" + +# 20ms of 8kHz u-law silence = 160 bytes; base64-encoded in a Twilio +# Media event envelope (matches the slice-A trunk_sim_e2e.rs TODO pattern +# + crates/rutster-trunk/tests/ws_ping.rs' frame shape). +PCM_FRAME_BYTES = b"\x00" * 160 +FRAMES_TO_SEND = 200 +CADENCE_MS = 20 +RECV_DEADLINE_S = 10.0 + + +def log(msg: str) -> None: + print(f"[allinone_smoke] {msg}", flush=True) + + +def fail(msg: str) -> "NoReturn": # noqa: F821 — Python 3.11+ syntax + print(f"[allinone_smoke] FAIL: {msg}", file=sys.stderr, flush=True) + subprocess.run( + ["docker", "logs", CONTAINER_NAME], capture_output=False, timeout=10 + ) + sys.exit(1) + + +def run(cmd, check=True, timeout=30.0) -> str: + """Run a command; return stdout. Fail the smoke on non-zero exit (if check).""" + log(f"$ {' '.join(cmd)}") + r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) + if check and r.returncode != 0: + print(f" stdout: {r.stdout[-2000:]}", file=sys.stderr) + print(f" stderr: {r.stderr[-2000:]}", file=sys.stderr) + fail(f"command exited {r.returncode}: {' '.join(cmd)}") + return r.stdout + + +def bootstrap_container() -> None: + """Boot rutster-allinone with RUTSTER_LOCAL_CERTS=true.""" + subprocess.run(["docker", "rm", "-f", CONTAINER_NAME], + capture_output=True, timeout=10) + subprocess.run(["docker", "volume", "rm", "-f", DATA_VOLUME], + capture_output=True, timeout=10) + env = [ + "-e", "RUTSTER_DOMAIN=localhost", + "-e", "RUTSTER_ACME_EMAIL=smoke@example.com", + # CRITICAL: RUTSTER_LOCAL_CERTS=true makes Caddy use its internal CA + # — no ACME in CI (spec §9). + "-e", "RUTSTER_LOCAL_CERTS=true", + "-e", "RUTSTER_HTTP_BIND=0.0.0.0:8080", + "-e", "RUTSTER_MEDIA_BIND_IP=127.0.0.1", + "-e", "RUTSTER_MEDIA_PORT_RANGE=49160-49170", + "-e", "RUTSTER_DRAIN_DEADLINE_SECS=10", + "-e", "RUTSTER_TAP_URL=ws://127.0.0.1:8082", + # Twilio creds unset — WebRTC + media-stream WS smoke only. + "-p", f"{HOST_PORT}:443", + ] + run(["docker", "run", "-d", "--name", CONTAINER_NAME, + "-v", f"{DATA_VOLUME}:/data", + *env, IMAGE], timeout=120.0) + root_cert = wait_for_root_cert() + log(f"root cert extracted ({len(root_cert)} bytes)") + wait_for_healthz(root_cert) + + +def wait_for_root_cert() -> bytes: + """Poll until Caddy wrote /data/caddy/pki/authorities/local/root.crt, + then docker cp it out. Caddy creates this on first boot with local_certs.""" + deadline = time.time() + 30.0 + while time.time() < deadline: + r = subprocess.run( + ["docker", "exec", CONTAINER_NAME, "test", "-f", + "/data/caddy/pki/authorities/local/root.crt"], + capture_output=True, timeout=5, + ) + if r.returncode == 0: + r = subprocess.run( + ["docker", "cp", + f"{CONTAINER_NAME}:/data/caddy/pki/authorities/local/root.crt", "-"], + capture_output=True, timeout=10, + ) + if r.returncode == 0 and r.stdout: + return r.stdout + time.sleep(0.5) + fail("Caddy internal CA root cert did not appear within 30s") + return b"" # unreachable + + +def wait_for_healthz(root_cert_pem: bytes) -> None: + """Curl /healthz through TLS using the extracted root cert.""" + with tempfile.NamedTemporaryFile(suffix=".pem", delete=False) as f: + f.write(root_cert_pem) + ca_file = f.name + try: + deadline = time.time() + 30.0 + r = subprocess.run(["true"]) # for the warning-less r init + while time.time() < deadline: + r = subprocess.run( + ["curl", "--cacert", ca_file, "-fsS", + f"https://localhost:{HOST_PORT}/healthz"], + capture_output=True, text=True, timeout=5, + ) + if r.returncode == 0 and r.stdout.strip() == "ok": + log("/healthz ok") + return + time.sleep(0.5) + fail(f"/healthz never returned ok within 30s (last stderr: " + f"{getattr(r, 'stderr', '')[-500:]})") + finally: + os.unlink(ca_file) + + +# --- hand-rolled WS frame encode/decode (RFC 6455 §5) --- + +def make_ws_frame(payload: bytes, opcode: int = 0x1) -> bytes: + """Encode a client->server WS frame. opcode 0x1=text, 0x2=binary. + Client frames MUST be masked (RFC 6455 §5.3).""" + mask = os.urandom(4) + masked = bytes(b ^ mask[i % 4] for i, b in enumerate(payload)) + header = bytearray([0x80 | opcode]) # FIN + opcode + n = len(payload) + if n < 126: + header.append(0x80 | n) + elif n < 65536: + header.append(0x80 | 126) + header.extend(struct.pack(">H", n)) + else: + header.append(0x80 | 127) + header.extend(struct.pack(">Q", n)) + header.extend(mask) + return bytes(header) + masked + + +def read_ws_frame(sock) -> tuple: + """Read one WS frame from the server. Returns (opcode, payload). + Server frames are NOT masked (RFC 6455 §5.1). + Returns (-1, b"") on EOF.""" + h1 = sock.recv(1) + if not h1: + return -1, b"" + fin_opcode = h1[0] + h2 = sock.recv(1)[0] + masked = bool(h2 & 0x80) + length = h2 & 0x7F + if length == 126: + length = struct.unpack(">H", sock.recv(2))[0] + elif length == 127: + length = struct.unpack(">Q", sock.recv(8))[0] + if masked: + mask = sock.recv(4) + payload = b"" + while len(payload) < length: + chunk = sock.recv(length - len(payload)) + if not chunk: + break + payload += chunk + if masked: + payload = bytes(b ^ mask[i % 4] for i, b in enumerate(payload)) + return fin_opcode & 0x0F, payload + + +def open_wss(): + """Open the wss:// WS handshake to /twilio/media-stream inside the + containerized all-in-one.""" + sock = socket.create_connection(("localhost", HOST_PORT), timeout=5) + ctx = ssl.create_default_context(cafile=None) + r = subprocess.run( + ["docker", "cp", + f"{CONTAINER_NAME}:/data/caddy/pki/authorities/local/root.crt", "-"], + capture_output=True, timeout=10, + ) + ctx.load_verify_locations(cadata=r.stdout.decode("ascii")) + ctx.check_hostname = False # the cert is for "localhost" — Caddy's local_certs issues it + ssl_sock = ctx.wrap_socket(sock, server_hostname="localhost") + log(f"TLS established (cipher={ssl_sock.cipher()})") + + key = os.urandom(16).hex() + handshake = ( + f"GET /twilio/media-stream HTTP/1.1\r\n" + f"Host: localhost:{HOST_PORT}\r\n" + f"Upgrade: websocket\r\n" + f"Connection: Upgrade\r\n" + f"Sec-WebSocket-Key: {key}\r\n" + f"Sec-WebSocket-Version: 13\r\n" + f"\r\n" + ) + ssl_sock.sendall(handshake.encode("ascii")) + buf = b"" + while b"\r\n\r\n" not in buf: + chunk = ssl_sock.recv(4096) + if not chunk: + fail("WS upgrade response closed prematurely") + buf += chunk + if b"101 Switching Protocols" not in buf: + fail(f"WS upgrade did not return 101: {buf[:300]!r}") + log("WS upgraded (HTTP/1.1 101 Switching Protocols)") + return ssl_sock + + +def send_twilio_handshake(sock) -> None: + """Send the Twilio Media Streams connected + start frames (the same + JSON sequence crates/rutster-trunk/tests/ws_ping.rs uses against the + in-process router — here against the real containerized edge->FOB path).""" + connected = json.dumps({ + "event": "connected", + "protocol": "twilio-media-stream", + "version": "1.0.0", + }) + start = json.dumps({ + "event": "start", + "start": {"streamSid": "MZsmoke", "callSid": "CAsmoke"}, + }) + sock.sendall(make_ws_frame(connected.encode("utf-8"))) + sock.sendall(make_ws_frame(start.encode("utf-8"))) + log("sent Twilio connected + start handshake frames") + + +def stream_pcm_frames(sock) -> int: + """Stream 200 PCM-zeroed frames at 20ms cadence as Twilio Media events. + Returns the count of text frames received back from the engine.""" + received = 0 + deadline_global = time.time() + (FRAMES_TO_SEND * CADENCE_MS / 1000.0) + RECV_DEADLINE_S + for _ in range(FRAMES_TO_SEND): + media_event = json.dumps({ + "event": "media", + "media": {"payload": b64encode(PCM_FRAME_BYTES).decode("ascii")}, + }) + sock.sendall(make_ws_frame(media_event.encode("utf-8"))) + # Drain non-blocking: read anything available without blocking the + # 20ms cadence. + sock.setblocking(False) + try: + while True: + op, _ = read_ws_frame(sock) + if op == -1: + break + if op == 0x1: # text frame — engine-originated event + received += 1 + except (BlockingIOError, ssl.SSLWantReadError): + pass + finally: + sock.setblocking(True) + time.sleep(CADENCE_MS / 1000.0) + sock.settimeout(RECV_DEADLINE_S) + try: + while time.time() < deadline_global: + op, _ = read_ws_frame(sock) + if op == -1: + break + if op == 0x1: + received += 1 + except (socket.timeout, ssl.SSLWantReadError): + pass + return received + + +def main() -> int: + log(f"image={IMAGE}, host port={HOST_PORT}") + bootstrap_container() + sock = open_wss() + send_twilio_handshake(sock) + received = stream_pcm_frames(sock) + log(f"sent {FRAMES_TO_SEND} frames; received {received} text frames back") + if received < 1: + fail("expected at least one engine-originated text frame (mark/outbound)") + log("PASS: real edge->FOB WS path through Caddy TLS verified") + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + finally: + # Always clean up the container + volume — even on FAIL. + subprocess.run(["docker", "rm", "-f", CONTAINER_NAME], + capture_output=True, timeout=10) + subprocess.run(["docker", "volume", "rm", "-f", DATA_VOLUME], + capture_output=True, timeout=10) +``` + + +- [ ] **Step 3: Write `deploy/smoke/compose_smoke.sh`** + +```bash +#!/usr/bin/env bash +# deploy/smoke/compose_smoke.sh — T2 compose smoke (deploy-B §9). +# +# Brings up deploy/compose.yaml with \`docker compose up -d --wait\`, asserts +# every service reports healthy via \`docker compose ps\`, then curls +# /healthz through Caddy's TLS (using the internal CA — +# RUTSTER_LOCAL_CERTS=true is set in the smoke's env). The compose smoke +# is intentionally lighter than the all-in-one smoke (which already proves +# the WS path) — its job is to assert the four-service orchestrator shape +# boots + the network_mode: "service:engine" brain->engine tap posture. +# +# The ValkeyEventSink-lands-in-stream assertion is slice C's concern. +# Named hook below; do NOT implement it in this slice. + +set -euo pipefail + +cd "$(dirname "$0")/.." # deploy/ + +IMAGES_TAG="${RUTSTER_IMAGES_TAG:-smoke}" +export RUTSTER_DOMAIN=localhost +export RUTSTER_ACME_EMAIL=smoke@example.com +export RUTSTER_LOCAL_CERTS=true +export RUTSTER_HTTP_BIND=0.0.0.0:8080 +export RUTSTER_MEDIA_BIND_IP=127.0.0.1 +export RUTSTER_MEDIA_PORT_RANGE=49160-49170 +export RUTSTER_DRAIN_DEADLINE_SECS=10 +export RUTSTER_TAP_URL=ws://127.0.0.1:8082 + +echo "[compose_smoke] overriding compose.yaml image tags -> :${IMAGES_TAG}" +# Sed the image: tags from :latest to :smoke so CI's locally-built images +# are picked up. In-place + restored on exit. +cp compose.yaml compose.yaml.smoke-backup +trap 'mv compose.yaml.smoke-backup compose.yaml' EXIT +sed -i "s|image: rutster-edge:latest|image: rutster-edge:${IMAGES_TAG}|" compose.yaml +sed -i "s|image: rutster-engine:latest|image: rutster-engine:${IMAGES_TAG}|" compose.yaml +sed -i "s|image: rutster-brain:latest|image: rutster-brain:${IMAGES_TAG}|" compose.yaml + +echo "[compose_smoke] docker compose up -d --wait" +docker compose up -d --wait --timeout 120 + +echo "[compose_smoke] docker compose ps" +docker compose ps + +# Each service should report "running" state. (Healthy flag is set by the +# engine's /readyz via the Docker healthcheck — none of the shipped images +# define a HEALTHCHECK yet; slice-G or a future hardening slice can add one. +# For now State="running" + /healthz 200 + /readyz 200 is the gate.) +HEALTHY_COUNT=$(docker compose ps --format json | jq '[.[] | select(.State == "running")] | length') +EXPECTED=4 +if [ "$HEALTHY_COUNT" -ne "$EXPECTED" ]; then + echo "[compose_smoke] FAIL: expected $EXPECTED running services, got $HEALTHY_COUNT" >&2 + docker compose logs + exit 1 +fi +echo "[compose_smoke] $EXPECTED services running" + +# Caddy /healthz via TLS (internal CA). Port 443 is published by the caddy service. +ROOT_CERT=$(docker compose exec -T caddy cat /data/caddy/pki/authorities/local/root.crt) +echo "$ROOT_CERT" > /tmp/rutster-smoke-root.pem + +# Wait for /healthz to return 200 via Caddy TLS. +for _ in $(seq 1 60); do + if curl --cacert /tmp/rutster-smoke-root.pem -fsS https://localhost/healthz \ + | grep -q '^ok$'; then + echo "[compose_smoke] /healthz ok" + break + fi + sleep 0.5 +done +curl --cacert /tmp/rutster-smoke-root.pem -fsS https://localhost/healthz \ + || { echo "[compose_smoke] FAIL: /healthz never returned ok"; docker compose logs; exit 1; } + +# Engine /readyz — fresh boot, MUST be 200 (drain happens on shutdown, not +# boot; cap only after RUTSTER_MAX_SESSIONS sessions exist). +curl --cacert /tmp/rutster-smoke-root.pem -fsS https://localhost/readyz \ + || { echo "[compose_smoke] FAIL: /readyz never returned ok"; docker compose logs; exit 1; } +echo "[compose_smoke] /readyz ok" + +# TODO slice-C: assert lifecycle event in Valkey stream. +# Slice C lands ValkeyEventSink (spec §5.6) + the assert-event-lands-in- +# Valkey-stream smoke addition. The hook here documents the contract: +# Once slice C lands, this smoke runs: +# docker compose exec valkey valkey-cli XLEN rutster:events +# and asserts > 0 after a sim call completes (an EventSink::on_event +# call writes via XADD MAXLEN ~). Until then, the valkey service is dark +# (no consumers) — counting the stream length would always be 0. +# Leaving the named hook so slice C's plan can grep for it. +echo "[compose_smoke] PASS: T2 compose boots + /healthz + /readyz verified" + +# Cleanup (compose down; volumes left for warm cache on re-run). +docker compose down --volumes --timeout 30 || true +exit 0 +``` + +- [ ] **Step 4: Write `deploy/smoke/reload_during_call.py`** + +```python +#!/usr/bin/env python3 +"""deploy/smoke/reload_during_call.py — Caddy config reload during a live +sim call, asserting zero frame drops (deploy-B §9; spec §3.2; TLS brief +§5 risk 1). + +Spec §3.2 mandates a CI e2e case for Caddy config-reload-during-live-call +because the upstream mitigation (`stream_close_delay` in Caddyfile) has an +open bug trail (caddy #6420, #7222) and is NOT trusted untested. + +This script reuses the all-in-one smoke's WS helpers (open wss:// +to /twilio/media-stream, send connected+start handshake); starts a streaming +loop sending PCM frames at 20ms cadence; concurrently triggers a `caddy +reload` mid-call (the operator's config-edit path); asserts the WS stays +up across the reload + no frames are dropped (the recv count matches the +sent count within tolerance). + +Exit code 0 = reload-during-call passed. Non-zero = failed. +""" + +import json +import os +import subprocess +import sys +import threading +import time +from base64 import b64encode + +# Reuse the all-in-one smoke's WS helpers via import. +sys.path.insert(0, os.path.dirname(__file__)) +from allinone_smoke import ( # noqa: E402 + make_ws_frame, read_ws_frame, bootstrap_container, + open_wss, send_twilio_handshake, fail, log, + CONTAINER_NAME, IMAGE, HOST_PORT, PCM_FRAME_BYTES, +) + +FRAMES_TO_SEND = 400 # 8s of wall clock at 20ms cadence +CADENCE_MS = 20 +RELOAD_AT_FRAME = 150 # fire `caddy reload` 3s into the 8s call + + +def trigger_caddy_reload() -> None: + """Run `docker exec rutster-allinone-smoke caddy reload` inside the + container. Caddy reloads its Caddyfile in-memory; stream_close_delay + 24h should keep upgraded WS tunnels alive.""" + log(f"triggering `caddy reload` at frame {RELOAD_AT_FRAME}...") + r = subprocess.run( + ["docker", "exec", CONTAINER_NAME, + "caddy", "reload", "--config", "/etc/caddy/Caddyfile", + "--adapter", "caddyfile"], + capture_output=True, text=True, timeout=30.0, + ) + if r.returncode != 0: + log(f"caddy reload stderr: {r.stderr[-500:]}") + # We do NOT fail the smoke on a non-zero caddy reload — the smoke's + # question is "did the live WS survive?", not "did caddy reload work?" + # A failed reload is its own bug; the WS-drop is the caddy #6420/#7222 + # concern this smoke is testing. + + +def stream_frames_with_midcall_reload(sock): + """Returns (frames_sent, frames_received, ws_dropped_during_reload). + ws_dropped_during_reload is True if the WS read loop got EOF/exception + in the 1-second window after the reload fired.""" + sent = 0 + received = 0 + ws_dropped = False + reload_fired = False + reload_time = 0.0 + + for i in range(FRAMES_TO_SEND): + if i == RELOAD_AT_FRAME and not reload_fired: + reload_fired = True + reload_time = time.time() + threading.Thread(target=trigger_caddy_reload, daemon=True).start() + + media_event = json.dumps({ + "event": "media", + "media": {"payload": b64encode(PCM_FRAME_BYTES).decode("ascii")}, + }) + try: + sock.sendall(make_ws_frame(media_event.encode("utf-8"), opcode=0x1)) + sent += 1 + except (BrokenPipeError, ConnectionResetError) as e: + log(f"WS send failed at frame {i}: {e}") + if not reload_fired or (time.time() - reload_time) < 2.0: + ws_dropped = True + break + + # Non-blocking drain. + import socket as _socket + sock.setblocking(False) + try: + while True: + op, _ = read_ws_frame(sock) + if op == -1: + break + if op == 0x1: + received += 1 + except (BlockingIOError, ): + pass + finally: + sock.setblocking(True) + time.sleep(CADENCE_MS / 1000.0) + + return sent, received, ws_dropped + + +def main() -> int: + log(f"image={IMAGE}, host port={HOST_PORT}") + bootstrap_container() + sock = open_wss() + send_twilio_handshake(sock) + sent, received, ws_dropped = stream_frames_with_midcall_reload(sock) + log(f"sent={sent}, received={received}, ws_dropped={ws_dropped}") + if ws_dropped: + fail("WS dropped during caddy reload — stream_close_delay mitigation " + "failed (caddy #6420/#7222) — the configuration MUST be re-evaluated") + if sent != FRAMES_TO_SEND: + fail(f"sent only {sent} of {FRAMES_TO_SEND} frames — WS died before reload") + log("PASS: caddy reload during live call dropped zero frames") + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + finally: + subprocess.run( + ["docker", "rm", "-f", CONTAINER_NAME], + capture_output=True, timeout=10, + ) +``` + +- [ ] **Step 5: Make the smoke scripts executable + re-run the content-check** + +```bash +cd /home/alee/Sources/rutster +chmod +x deploy/smoke/allinone_smoke.py deploy/smoke/compose_smoke.sh deploy/smoke/reload_during_call.py + +test -f deploy/smoke/allinone_smoke.py \ + && test -f deploy/smoke/compose_smoke.sh \ + && test -f deploy/smoke/reload_during_call.py \ + && grep -q 'local_certs' deploy/smoke/allinone_smoke.py \ + && grep -q 'TODO slice-C' deploy/smoke/compose_smoke.sh \ + && grep -q 'caddy reload' deploy/smoke/reload_during_call.py \ + && grep -q 'stream_close_delay' deploy/smoke/reload_during_call.py \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-PASS`. + +- [ ] **Step 6: Commit** + +```bash +cd /home/alee/Sources/rutster +git add deploy/smoke/ +git commit -s -m "feat(deploy/smoke): all-in-one + compose + reload-during-call smoke harness (deploy-B §9) + +Three Python-stdlib-only smoke scripts (no pip install required in CI; +the runner image ships Python 3 + ssl/socket/json/base64/time): + +* allinone_smoke.py — boots rutster-allinone with RUTSTER_LOCAL_CERTS=true + (Caddy internal CA, no ACME per spec §9), extracts the root cert from + /data, opens wss:// WS to /twilio/media-stream, sends Twilio's + connected+start handshake (the same JSON sequence the in-tree + ws_ping.rs/nodelay.rs tests use), streams 200 PCM frames at 20ms cadence, + asserts the engine echoes/replies at least one text frame. Reuses the + T8 stub's handshake pattern (crates/rutster/tests/trunk_sim_e2e.rs TODO + body) against a real containerized binary rather than an in-process + MockTwilioMediaStreamsServer. +* compose_smoke.sh — T2 four-service boots + health checks via Caddy TLS. + The # TODO slice-C: assert lifecycle event in Valkey stream named hook + documents the contract slice C will assert; the valkey service is dark + on day one (no consumers) so the stream length is always 0 today. +* reload_during_call.py — holds a live WS streaming frames at 20ms while a + concurrent \`caddy reload\` fires mid-call; asserts zero frames dropped + + WS survives across the reload. Regresses the stream_close_delay 24h + mitigation against caddy #6420/#7222 (TLS brief §5 risk 1; spec §9 + reload-during-call smoke). + +WS frame encode/decode is hand-rolled (RFC 6455 §5 — ten lines on top of +socket+ssl) so the smoke needs no websocket-client/websockets pip +dependency + no network round-trip in CI." +``` + + +--- + +### Task 7: Extend `.github/workflows/ci.yml` — `image-build` job + +The image-build job builds all four images via `docker buildx build` against `deploy/Dockerfile`, +with GHA cache from `type=gha`. It runs AFTER and GATED ON the existing fmt/clippy/test/deny/ +sim-bench matrix (the smoke job after it depends on these built images). + +**Files:** +- Modify: `.github/workflows/ci.yml` (add `image-build` job at the end; do NOT touch existing + `fmt`/`clippy`/`test`/`deny`/`sim-bench`/`twilio-live` jobs — verified they exist at the + current HEAD + their content is unchanged. The file currently ends at line 146 with the + `twilio-live` job's closing `run: cargo test ... --include-ignored` step). + +**Interfaces:** +- Consumes: Task 1's `deploy/Dockerfile` + `.dockerignore` (Task 2) + `deploy/Caddyfile` + (Task 3) + `deploy/s6-rc.d/` (Task 1 Step 4) + vendored s6-overlay tarballs (Task 1 Step 4). +- Produces: four Docker images tagged `rutster-{engine,brain,edge,allinone}:ci-${{ github.sha }}` + + re-tagged as `:smoke` for the downstream smoke jobs. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +grep -q 'name: image-build' .github/workflows/ci.yml && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Append the `image-build` job to `.github/workflows/ci.yml`** + +Append the following job to the end of `.github/workflows/ci.yml` (after the `twilio-live` +job's closing block): + +```yaml + + # deploy-B §6.1 image-build — builds all four first-party images via + # `docker buildx build --target` against deploy/Dockerfile. Runs AFTER the + # fmt/clippy/test/deny/sim-bench matrix gates: the smoke job downstream + # depends on these built images. + # + # Single builder invocation that builds all four --targets would be ideal + # (shared cache), but buildx's `--target` is one-per-invocation — so four + # build steps share the GHA cache via `cache-from/cache-to: type=gha`. + # Warm caches take image-build from ~6 min to ~90s. + image-build: + name: image-build (4 images) + runs-on: ubuntu-latest + needs: [fmt, clippy, test, deny, sim-bench] + steps: + - uses: actions/checkout@v4 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + # libopus-dev — the headers the `opus` crate's build.rs expects. + # (The builder stage of deploy/Dockerfile already installs this + # inside the build image; this is a belt-and-braces step so a local + # buildx cache root has the headers if the cache misses.) + - name: Build-time host deps (belt-and-braces) + run: sudo apt-get update && sudo apt-get install -y libopus-dev + + - name: Build rutster-engine image + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-engine + tags: rutster-engine:ci-${{ github.sha }} + push: false + load: true + cache-from: type=gha,scope=rutster-engine + cache-to: type=gha,mode=max,scope=rutster-engine + + - name: Build rutster-brain image + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-brain + tags: rutster-brain:ci-${{ github.sha }} + push: false + load: true + cache-from: type=gha,scope=rutster-brain + cache-to: type=gha,mode=max,scope=rutster-brain + + - name: Build rutster-edge image + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-edge + tags: rutster-edge:ci-${{ github.sha }} + push: false + load: true + cache-from: type=gha,scope=rutster-edge + cache-to: type=gha,mode=max,scope=rutster-edge + + - name: Build rutster-allinone image + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-allinone + tags: rutster-allinone:ci-${{ github.sha }} + push: false + load: true + cache-from: type=gha,scope=rutster-allinone + cache-to: type=gha,mode=max,scope=rutster-allinone + + - name: Smoke-tag the images for the downstream smoke job + # The smoke job references images as `:smoke` + # (deploy/smoke/compose_smoke.sh tags images with the + # RUTSTER_IMAGES_TAG env). Tag the freshly-built :ci- images as + # :smoke too, so the smoke job picks them up. + run: | + for img in rutster-engine rutster-brain rutster-edge rutster-allinone; do + docker tag "${img}:ci-${{ github.sha }}" "${img}:smoke" + done + docker images | grep rutster + + - name: caddy validate + # Validate the Caddyfile inside the freshly-built rutster-edge image. + # Cheap regression against malformed Caddyfile (the local `caddy + # validate` in Task 3 is optional; this one is mandatory in CI). + run: | + docker run --rm \ + -e RUTSTER_DOMAIN=pbx.example.com \ + -e RUTSTER_ACME_EMAIL=test@example.com \ + -v "$PWD/deploy/Caddyfile:/etc/caddy/Caddyfile:ro" \ + rutster-edge:smoke \ + caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile +``` + +- [ ] **Step 3: Re-run the content-check** + +```bash +cd /home/alee/Sources/rutster +grep -q 'name: image-build' .github/workflows/ci.yml \ + && grep -q 'needs: \[fmt, clippy, test, deny, sim-bench\]' .github/workflows/ci.yml \ + && grep -q 'docker/build-push-action@v5' .github/workflows/ci.yml \ + && grep -q 'cache-from: type=gha' .github/workflows/ci.yml \ + && grep -q 'target: rutster-engine' .github/workflows/ci.yml \ + && grep -q 'target: rutster-brain' .github/workflows/ci.yml \ + && grep -q 'target: rutster-edge' .github/workflows/ci.yml \ + && grep -q 'target: rutster-allinone' .github/workflows/ci.yml \ + && grep -q 'caddy validate' .github/workflows/ci.yml \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-PASS`. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add .github/workflows/ci.yml +git commit -s -m "ci: image-build job — builds the four first-party Docker images (deploy-B §6.1) + +Extends .github/workflows/ci.yml with an image-build job, gated on the +existing fmt/clippy/test/deny/sim-bench matrix +(needs: [fmt, clippy, test, deny, sim-bench]). Builds all four images via +docker buildx against deploy/Dockerfile's named --targets, with type=gha +caching scoped per image so warm caches take image-build from ~6min to +~90s. + +Tags the freshly-built :ci- images as :smoke too, for the +downstream smoke job to consume. Runs \`caddy validate\` inside the +freshly-built rutster-edge as a cheap Caddyfile-syntax regression. + +No changes to the existing fmt/clippy/test/deny/sim-bench/twilio-live +jobs — those stay the cargo-side source of truth." +``` + +--- + +### Task 8: Extend `.github/workflows/ci.yml` — `smoke` job (all-in-one TLS sim call) + +The smoke job runs `deploy/smoke/allinone_smoke.py` against the freshly-built `rutster-allinone` +image from Task 7. It depends on `image-build` (which itself depends on the fmt/clippy/test/deny/ +sim-bench matrix). The smoke job is where spec §9's "all-in-one smoke — boots, Caddy terminates +TLS via its INTERNAL CA (no ACME in CI), full sim call through the real edge→FOB WS path" +assertion is enforced. + +**Files:** +- Modify: `.github/workflows/ci.yml` (add `smoke` job after `image-build`). + +**Interfaces:** +- Consumes: Task 6's `deploy/smoke/allinone_smoke.py` + Task 7's `rutster-allinone:smoke` image. +- Produces: pass/fail for the smoke job + downstream `smoke-reload` and `smoke-compose` jobs + (Task 9) gate on this job. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +grep -q 'name: smoke' .github/workflows/ci.yml && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Append the `smoke` job to `.github/workflows/ci.yml`** + +Append the following job to the end of `.github/workflows/ci.yml` (after the `image-build` +job's closing `caddy validate` step from Task 7): + +```yaml + + # deploy-B §9 all-in-one smoke — boots rutster-allinone with + # RUTSTER_LOCAL_CERTS=true (Caddy internal CA, no ACME in CI), extracts + # the CA root cert from /data, opens wss:// to /twilio/media-stream, + # sends Twilio's connected+start handshake frames, streams 200 PCM-zeroed + # frames at 20ms cadence, asserts the engine replies with at least one + # text frame through the real edge->FOB WS path. Spec §9 acceptance: + # * "boots" + # * "Caddy terminates TLS via its internal CA (no ACME in CI)" + # * "full sim call through the real edge->FOB WS path" + # The reload-during-call smoke (Task 9 below) and compose smoke (Task 9) + # are separate jobs that depend on this one. + smoke: + name: smoke (all-in-one TLS sim call) + runs-on: ubuntu-latest + needs: [image-build] + steps: + - uses: actions/checkout@v4 + + - name: Set up Python (stdlib-only; no pip) + uses: actions/setup-python@v5 + with: + python-version: "3.x" + + # Pull the freshly-built image from the image-build job. Since + # jobs run on separate runners + `push: false` in image-build means + # the image isn't in any registry, the smoke job rebuilds from the + # GHA cache (warm — image-build populated it). This adds ~30s vs + # an image-download artifact, but avoids the registry-auth + image- + # save/load plumbing. + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Rebuild rutster-allinone:smoke (warm cache from image-build) + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-allinone + tags: rutster-allinone:smoke + push: false + load: true + cache-from: type=gha,scope=rutster-allinone + + - name: Run all-in-one smoke (deploy/smoke/allinone_smoke.py) + run: | + python3 deploy/smoke/allinone_smoke.py + env: + RUTSTER_ALLINONE_IMAGE: rutster-allinone:smoke + RUTSTER_ALLINONE_PORT: "18443" +``` + +- [ ] **Step 3: Re-run the content-check** + +```bash +cd /home/alee/Sources/rutster +grep -q 'name: smoke' .github/workflows/ci.yml \ + && grep -q 'needs: \[image-build\]' .github/workflows/ci.yml \ + && grep -q 'allinone_smoke.py' .github/workflows/ci.yml \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-PASS`. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add .github/workflows/ci.yml +git commit -s -m "ci: smoke job — all-in-one TLS sim call through real edge->FOB WS path (deploy-B §9) + +Adds the smoke job to .github/workflows/ci.yml, gated on image-build. +Runs deploy/smoke/allinone_smoke.py against the freshly-built +rutster-allinone:smoke image: + +* Boots the all-in-one container with RUTSTER_LOCAL_CERTS=true (Caddy + internal CA, no ACME in CI per spec §9). +* Extracts Caddy's internal-CA root cert from /data/caddy/pki/ + authorities/local/root.crt via docker cp. +* Opens wss:// WS to /twilio/media-stream using the extracted root + (Python stdlib ssl + socket — no pip install required, CI runner- + image ships Python 3 + ssl). +* Sends Twilio's connected+start handshake frames (the same JSON + sequence crates/rutster-trunk/tests/ws_ping.rs uses). +* Streams 200 PCM-zeroed frames at 20ms cadence as Twilio Media events. +* Asserts the engine replies with at least one text frame — proving + the real edge->FOB WS path works end-to-end through TLS. + +The compose smoke + reload-during-call smoke (Task 9) gate on this job." +``` + + +--- + +### Task 9: Extend `.github/workflows/ci.yml` — `smoke-compose` + `smoke-reload` jobs + +Two more smoke jobs, each gated on `image-build` (Task 7). `smoke-compose` runs +`deploy/smoke/compose_smoke.sh` against the four fresh images from `image-build`. +`smoke-reload` runs `deploy/smoke/reload_during_call.py` (reuses the all-in-one image — both +smokes are all-in-one-shaped; these are separate jobs not steps within `smoke` for clearer +failure attribution + parallelism). + +**Files:** +- Modify: `.github/workflows/ci.yml` (append two more jobs after `smoke`). + +**Interfaces:** +- Consumes: Task 6's `compose_smoke.sh` + `reload_during_call.py` + Task 7's images. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +grep -q 'name: smoke-compose' .github/workflows/ci.yml \ + && grep -q 'name: smoke-reload' .github/workflows/ci.yml \ + && grep -q 'reload_during_call.py' .github/workflows/ci.yml \ + && grep -q 'compose_smoke.sh' .github/workflows/ci.yml \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Append the two new jobs to `.github/workflows/ci.yml`** + +Append the following to the end of `.github/workflows/ci.yml` (after the `smoke` job from +Task 8): + +```yaml + + # deploy-B §9 compose smoke — T2 four-service compose stack boots + all + # services healthy + Caddy TLS /healthz + /readyz. Lighter than the + # all-in-one smoke (which already proved the WS path) — its job is the + # orchestrator shape + the network_mode: "service:engine" brain->engine tap + # posture across the four services. + smoke-compose: + name: smoke (T2 compose four-service) + runs-on: ubuntu-latest + needs: [image-build] + steps: + - uses: actions/checkout@v4 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + # Rebuild all three first-party images from the warm GHA cache + # populated by image-build. Valkey is upstream (no rebuild needed). + - name: Rebuild rutster-edge:smoke + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-edge + tags: rutster-edge:smoke + push: false + load: true + cache-from: type=gha,scope=rutster-edge + + - name: Rebuild rutster-engine:smoke + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-engine + tags: rutster-engine:smoke + push: false + load: true + cache-from: type=gha,scope=rutster-engine + + - name: Rebuild rutster-brain:smoke + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-brain + tags: rutster-brain:smoke + push: false + load: true + cache-from: type=gha,scope=rutster-brain + + # jq is pre-installed on ubuntu-latest GHA runners. + - name: Run compose smoke (deploy/smoke/compose_smoke.sh) + run: bash deploy/smoke/compose_smoke.sh + env: + RUTSTER_IMAGES_TAG: smoke + + # deploy-B §9 + spec §3.2 + TLS brief §5 risk 1: Caddy config-reload + # during a live simulated call, asserting zero frame drops. The + # stream_close_delay 24h mitigation in deploy/Caddyfile is the load- + # bearing fixture; caddy #6420 / #7222 are open upstream bugs — the + # smoke is "don't trust untested" made CI-regressed. + smoke-reload: + name: smoke (caddy reload during live call, zero drops) + runs-on: ubuntu-latest + needs: [smoke] # reuses the rutster-allinone:smoke image, warm cache + steps: + - uses: actions/checkout@v4 + + - name: Set up Python (stdlib-only; no pip) + uses: actions/setup-python@v5 + with: + python-version: "3.x" + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Rebuild rutster-allinone:smoke (warm cache) + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-allinone + tags: rutster-allinone:smoke + push: false + load: true + cache-from: type=gha,scope=rutster-allinone + + - name: Run reload-during-call smoke (deploy/smoke/reload_during_call.py) + run: python3 deploy/smoke/reload_during_call.py + env: + RUTSTER_ALLINONE_IMAGE: rutster-allinone:smoke + RUTSTER_ALLINONE_PORT: "18444" +``` + +- [ ] **Step 3: Re-run the content-check** + +```bash +cd /home/alee/Sources/rutster +grep -q 'name: smoke-compose' .github/workflows/ci.yml \ + && grep -q 'name: smoke-reload' .github/workflows/ci.yml \ + && grep -q 'reload_during_call.py' .github/workflows/ci.yml \ + && grep -q 'compose_smoke.sh' .github/workflows/ci.yml \ + && grep -q 'needs: \[smoke\]' .github/workflows/ci.yml \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-PASS`. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add .github/workflows/ci.yml +git commit -s -m "ci: smoke-compose + smoke-reload jobs — T2 four-service boot + caddy reload-during-call (deploy-B §9) + +smoke-compose (gated on image-build): +* Rebuilds rutster-edge/engine/brain:smoke from warm GHA cache. +* Runs deploy/smoke/compose_smoke.sh — \`docker compose up -d --wait\`, + asserts all four services State=running, curls /healthz + /readyz + through Caddy TLS (internal CA). Lighter than the all-in-one smoke + (which already proved the WS path) — its job is the orchestrator + shape + the network_mode: \"service:engine\" brain->engine tap posture. + +smoke-reload (gated on smoke): +* Reuses rutster-allinone:smoke + the allinone_smoke.py WS helpers. +* Runs deploy/smoke/reload_during_call.py — holds a live WS streaming + frames at 20ms while a concurrent \`caddy reload\` fires mid-call; + asserts zero frames dropped + WS survives across the reload. + Regresses the stream_close_delay 24h mitigation against caddy #6420/ + #7222 (TLS brief §5 risk 1; spec §9 reload-during-call smoke). + +Both jobs run on ubuntu-latest — separate runners for clearer failure +attribution + parallelism (the smoke gate is the failing CI signal, +not a single serial run)." +``` + + +--- + +### Task 10: Slice F — `.github/workflows/publish-images.yml` tag-push image publish + +Slice F (spec §6.3 + §11): tag-push workflow publishing all four images to the +`git.adlee.work/alee/rutster-*` registry namespace. `linux/amd64` only. Runs AFTER and GATED ON +the existing fmt/clippy/test/deny/sim-bench/image-build pipeline (re-runs the build to assert +freshness, then publishes; caching keeps the build cost ~30s when nothing changed). + +**Files:** +- Create: `.github/workflows/publish-images.yml` + +**Interfaces:** +- Consumes: Task 1's `deploy/Dockerfile` + the same `docker buildx build --target` invocations as + Task 7's `image-build` job (the publish workflow is structurally a copy of image-build with + `push: true` + `tags: git.adlee.work/alee/...` instead of `push: false + tags: ...:ci-`). + Reuses the GHA cache (warm from image-build's `cache-to: type=gha,mode=max`). +- Produces: four published Docker images at `git.adlee.work/alee/rutster-{allinone,engine,brain,edge}:${{ github.ref_name }}` + + `:latest` on every `v*` tag push. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f .github/workflows/publish-images.yml \ + && grep -q 'on:' .github/workflows/publish-images.yml \ + && grep -q 'push:' .github/workflows/publish-images.yml \ + && grep -q 'git.adlee.work/alee/rutster-allinone' .github/workflows/publish-images.yml \ + && grep -q 'git.adlee.work/alee/rutster-engine' .github/workflows/publish-images.yml \ + && grep -q 'git.adlee.work/alee/rutster-brain' .github/workflows/publish-images.yml \ + && grep -q 'git.adlee.work/alee/rutster-edge' .github/workflows/publish-images.yml \ + && grep -q 'linux/amd64' .github/workflows/publish-images.yml \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write the complete workflow file** + +Write `.github/workflows/publish-images.yml` with exactly this content: + +```yaml +# .github/workflows/publish-images.yml — slice F (spec §6.3 + §11). +# +# Tag-push workflow: builds + pushes all four first-party images to the +# git.adlee.work/alee/rutster-* registry namespace. linux/amd64 only (spec +# §1.2 / ADR-0011 — arm64 is a named deferral). +# +# Gating: runs AFTER and gated on the existing fmt/clippy/test/deny/sim-bench +# + image-build pipeline. The publish workflow re-runs the fmt/clippy/test/ +# deny matrix as a `needs:` chain (cheap when nothing changed) + builds all +# four images from the same Dockerfile CI's image-build job uses. This makes +# the published image byte-identical to the one CI tested — the "registry tag +# points at a tested image" invariant operators rely on. +# +# Registry namespace: git.adlee.work/alee/rutster-* (matches the `alee` tea +# login per AGENTS.md Git workflow + the root Cargo.toml's `repository` +# field). If your Gitea registry uses a different owner segment, replace +# `alee` throughout this file (REGISTRY_NAMESPACE below). + +name: Publish Images + +on: + push: + tags: + - 'v*' + +env: + REGISTRY: git.adlee.work + REGISTRY_NAMESPACE: alee + PLATFORMS: linux/amd64 + +jobs: + # The publish workflow is its own pipeline; it re-runs fmt/clippy/test/deny + # + builds the images + then publishes. The fmt/clippy/test/deny re-runs + # are cheap on a green repo (cache hits, fast fmt + clippy + test), and + # they make a critical guarantee: the version published under a `v*` tag + # has ALL the repo's quality gates green at that exact commit. Without + # this `needs:` chain, a tag could be pushed at a commit where the matrix + # failed but image-build happened to succeed on a flake. + fmt: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + with: + components: rustfmt + - run: cargo fmt --check + + clippy: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + with: + components: clippy + - name: Install libopus (media crate FFI dep) + run: apt-get update && apt-get install -y libopus-dev + - uses: Swatinem/rust-cache@v2 + - run: cargo clippy --all --all-targets -- -D warnings + + test: + runs-on: ubuntu-latest + strategy: + matrix: + toolchain: [stable, "1.85"] + steps: + - uses: actions/checkout@v4 + # Seam gate (mirror of .github/workflows/ci.yml's test job — + # loop_driver.rs + rtc_session.rs stay byte-identical to slice-3/5). + - name: Seam gate — loop_driver frozen + rtc_session pinned + run: | + EXPECTED_LOOP_DRIVER='744bf314edf7f4925c8bb3bd0f5176dbc88f8113' + EXPECTED_RTC_SESSION='f47d63b9a2883d37066a93c9daa0e2cf8816bec4' + GOT_LOOP_DRIVER=$(git hash-object crates/rutster-media/src/loop_driver.rs) + GOT_RTC_SESSION=$(git hash-object crates/rutster-media/src/rtc_session.rs) + [ "$GOT_LOOP_DRIVER" = "$EXPECTED_LOOP_DRIVER" ] || exit 1 + [ "$GOT_RTC_SESSION" = "$EXPECTED_RTC_SESSION" ] || exit 1 + - uses: dtolnay/rust-toolchain@master + with: + toolchain: ${{ matrix.toolchain }} + - name: Install libopus (media crate FFI dep) + run: apt-get update && apt-get install -y libopus-dev + - uses: Swatinem/rust-cache@v2 + - run: cargo test --all + + deny: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - uses: EmbarkStudios/cargo-deny-action@v2 + with: + command: check + + sim-bench: + name: sim-bench (stable) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + with: + components: clippy + - name: Install libopus (media crate FFI dep) + run: apt-get update && apt-get install -y libopus-dev + - uses: Swatinem/rust-cache@v2 + - name: cargo fmt + clippy on sim-bench feature paths + run: | + cargo fmt --all --check + cargo clippy --all --all-targets --features=sim-bench -- -D warnings + - name: Run sim-bench threshold sweep + run: cargo test --all --features=sim-bench -- --test-threads=1 + + publish: + name: Publish 4 images to git.adlee.work/alee/rutster-* + runs-on: ubuntu-latest + needs: [fmt, clippy, test, deny, sim-bench] + permissions: + contents: read + packages: write + steps: + - uses: actions/checkout@v4 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Build-time host deps (belt-and-braces) + run: sudo apt-get update && sudo apt-get install -y libopus-dev + + - name: Log in to Gitea registry + # GITEA_REGISTRY_USERNAME + GITEA_REGISTRY_PASSWORD are repo secrets + # set by the maintainer (per AGENTS.md Git workflow — the `alee` + # tea login). The username is `alee` (matches REGISTRY_NAMESPACE). + uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ secrets.GITEA_REGISTRY_USERNAME }} + password: ${{ secrets.GITEA_REGISTRY_PASSWORD }} + + - name: Determine tags + id: tags + # Two tags per image: the version tag (github.ref_name, e.g. v0.1.0) + # + `latest`. Operators can pin to either; `latest` is the + # convenience tag that always points at the most recent release. + run: | + echo "version_tag=${GITHUB_REF_NAME}" >> $GITHUB_OUTPUT + echo "namespace=${REGISTRY}/${REGISTRY_NAMESPACE}" >> $GITHUB_OUTPUT + + - name: Build + push rutster-engine + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-engine + platforms: ${{ env.PLATFORMS }} + tags: | + ${{ steps.tags.outputs.namespace }}/rutster-engine:${{ steps.tags.outputs.version_tag }} + ${{ steps.tags.outputs.namespace }}/rutster-engine:latest + push: true + cache-from: type=gha,scope=rutster-engine + cache-to: type=gha,mode=max,scope=rutster-engine + + - name: Build + push rutster-brain + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-brain + platforms: ${{ env.PLATFORMS }} + tags: | + ${{ steps.tags.outputs.namespace }}/rutster-brain:${{ steps.tags.outputs.version_tag }} + ${{ steps.tags.outputs.namespace }}/rutster-brain:latest + push: true + cache-from: type=gha,scope=rutster-brain + cache-to: type=gha,mode=max,scope=rutster-brain + + - name: Build + push rutster-edge + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-edge + platforms: ${{ env.PLATFORMS }} + tags: | + ${{ steps.tags.outputs.namespace }}/rutster-edge:${{ steps.tags.outputs.version_tag }} + ${{ steps.tags.outputs.namespace }}/rutster-edge:latest + push: true + cache-from: type=gha,scope=rutster-edge + cache-to: type=gha,mode=max,scope=rutster-edge + + - name: Build + push rutster-allinone + uses: docker/build-push-action@v5 + with: + context: . + file: deploy/Dockerfile + target: rutster-allinone + platforms: ${{ env.PLATFORMS }} + tags: | + ${{ steps.tags.outputs.namespace }}/rutster-allinone:${{ steps.tags.outputs.version_tag }} + ${{ steps.tags.outputs.namespace }}/rutster-allinone:latest + push: true + cache-from: type=gha,scope=rutster-allinone + cache-to: type=gha,mode=max,scope=rutster-allinone +``` + +- [ ] **Step 3: Re-run the content-check + YAML lint** + +```bash +cd /home/alee/Sources/rutster +test -f .github/workflows/publish-images.yml \ + && grep -q 'on:' .github/workflows/publish-images.yml \ + && grep -q 'push:' .github/workflows/publish-images.yml \ + && grep -q 'git.adlee.work/alee/rutster-allinone' .github/workflows/publish-images.yml \ + && grep -q 'git.adlee.work/alee/rutster-engine' .github/workflows/publish-images.yml \ + && grep -q 'git.adlee.work/alee/rutster-brain' .github/workflows/publish-images.yml \ + && grep -q 'git.adlee.work/alee/rutster-edge' .github/workflows/publish-images.yml \ + && grep -q 'linux/amd64' .github/workflows/publish-images.yml \ + && grep -q 'needs: \[fmt, clippy, test, deny, sim-bench\]' .github/workflows/publish-images.yml \ + && grep -q 'GITEA_REGISTRY_USERNAME' .github/workflows/publish-images.yml \ + && grep -q 'v0.1.0' .github/workflows/publish-images.yml \ + && echo CHECK-PASS || echo CHECK-FAIL + +# YAML syntax validation via Python stdlib (no yamllint dep needed). +python3 -c "import yaml; yaml.safe_load(open('.github/workflows/publish-images.yml'))" \ + || { echo "YAML syntax error"; exit 1; } +``` + +Expected: `CHECK-PASS`. The YAML loads cleanly. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add .github/workflows/publish-images.yml +git commit -s -m "ci: publish-images.yml — slice F tag-push image publish (deploy-F §6.3) + +Tag-push workflow (\`on: push: tags: ['v*']\`) that publishes all four +first-party images to git.adlee.work/alee/rutster-* (REGISTRY_NAMESPACE +matches the \`alee\` tea login per AGENTS.md Git workflow + the root +Cargo.toml's repository field). linux/amd64 only (spec §1.2 — arm64 is a +named deferral per ADR-0011). + +Gating: re-runs fmt/clippy/test/deny/sim-bench as a \`needs:\` chain so a +tag's published image is byte-identical to the one tested by the routine +CI matrix. The fmt/clippy/test/deny re-runs are cheap on a green repo +(cache hits, fast checks), and they make a critical guarantee: the +version under a \`v*\` tag has ALL the repo's quality gates green at +that exact commit. Without this \`needs:\` chain, a tag could be pushed +at a commit where the matrix failed but image-build happened to succeed +on a flake. + +Two tags per image: the version tag (github.ref_name, e.g. v0.1.0) + +\`latest\` (the convenience tag operators pin to). + +Docker login via GITEA_REGISTRY_USERNAME + GITEA_REGISTRY_PASSWORD +secrets, set by the maintainer. If your Gitea registry uses a different +owner segment, replace \`alee\` throughout (REGISTRY_NAMESPACE + the +namespace/git.adlee.work/alee/ strings in .env.example + compose.yaml)." +``` + +--- + +### Task 11: Final verification sweep + +No code. Evidence before assertions — run every gate the CI runs, plus the seam-gate check, +before calling the slice done. + +- [ ] **Step 1: Full gate run (cargo-side, unchanged from existing CI)** + +```bash +cd /home/alee/Sources/rutster +cargo fmt --all --check +cargo clippy --all -- -D warnings +cargo clippy --all --all-targets --features=sim-bench -- -D warnings +cargo test --all +cargo test --all --features=sim-bench -- --test-threads=1 +cargo deny check +cargo doc --no-deps +``` + +Expected: every command exits 0. (This slice adds no Rust code; these gates stay green from +plan A's landing on `main`. The sweep verifies the slice-B file additions don't accidentally +break the cargo build — e.g. a stray `Cargo.toml` edit.) + +- [ ] **Step 2: Seam gate verification (unchanged hashes)** + +```bash +cd /home/alee/Sources/rutster +git hash-object crates/rutster-media/src/loop_driver.rs +git hash-object crates/rutster-media/src/rtc_session.rs +``` + +Expected output, exactly: + +``` +744bf314edf7f4925c8bb3bd0f5176dbc88f8113 +f47d63b9a2883d37066a93c9daa0e2cf8816bec4 +``` + +If either hash differs, STOP — a task touched a seam-gated file; restore with +`git checkout main -- crates/rutster-media/src/loop_driver.rs crates/rutster-media/src/rtc_session.rs` +and re-run the gates. + +- [ ] **Step 3: Artifact inventory (every planned file exists)** + +```bash +cd /home/alee/Sources/rutster +for f in \ + deploy/Dockerfile \ + deploy/.dockerignore \ + deploy/Caddyfile \ + deploy/compose.yaml \ + deploy/.env.example \ + deploy/s6-overlay-noarch.tar.xz \ + deploy/s6-overlay-x86_64.tar.xz \ + deploy/s6-overlay.sha256sum \ + deploy/s6-rc.d/caddy/run \ + deploy/s6-rc.d/engine/run \ + deploy/s6-rc.d/brain/run \ + deploy/s6-rc.d/valkey-database/run \ + deploy/smoke/allinone_smoke.py \ + deploy/smoke/compose_smoke.sh \ + deploy/smoke/reload_during_call.py \ + .github/workflows/publish-images.yml \ + ; do + test -e "$f" || echo "MISSING: $f" +done +echo "inventory check complete" +``` + +Expected: zero `MISSING:` lines. + +- [ ] **Step 4: CI workflow sanity (both workflow files parse cleanly + image-build/smoke jobs + are present)** + +```bash +cd /home/alee/Sources/rutster +python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" +python3 -c "import yaml; yaml.safe_load(open('.github/workflows/publish-images.yml'))" +grep -E 'name: (image-build|smoke|smoke-compose|smoke-reload)' .github/workflows/ci.yml +grep -E 'name: (Publish|fmt|clippy|test|deny|sim-bench|publish)' .github/workflows/publish-images.yml +``` + +Expected: both YAML files load; `ci.yml` lists all four new jobs (`image-build`, `smoke`, +`smoke-compose`, `smoke-reload`); `publish-images.yml` lists its `publish` job + the gating matrix. + +- [ ] **Step 5: TODO slice-C hook verification (the named hook stays a comment, not implemented)** + +```bash +cd /home/alee/Sources/rutster +grep -n 'TODO slice-C' deploy/smoke/compose_smoke.sh +``` + +Expected: exactly one match, naming the ValkeyEventSink assertion as deferred to slice C. The +slice-E (ValkeyEventSink) plan will grep this same line. + +- [ ] **Step 6: DCO verification (every commit on this branch signed off)** + +```bash +cd /home/alee/Sources/rutster +git log main..HEAD --format='%h %s' \ + | head -20 \ + && echo "---DCO count vs commit count---" \ + && COMMITS=$(git log main..HEAD --format='%H' | wc -l) \ + && DCOS=$(git log main..HEAD --format='%(trailers:key=Signed-off-by)' | grep -c 'Signed-off-by') \ + && echo "commits: $COMMITS, signoffs: $DCOS" +``` + +Expected: the signoffs count equals the commit count (every `git commit -s` produced a +Signed-off-by trailer per AGENTS.md DCO requirement). + +- [ ] **Step 7: PR opening** + +```bash +cd /home/alee/Sources/rutster +git push -u origin deploy-b/packaging-ci +tea pulls create \ + --head deploy-b/packaging-ci \ + --base main \ + --title "deploy slice B + F: deploy/ artifacts + image-build + container smoke CI + tag-push publish" \ + --description "## What lands + +- deploy/Dockerfile (multi-stage, four named --target stages: rutster-engine/brain/edge/allinone) +- deploy/compose.yaml (T2 reference; stop_grace_period 660s; brain network_mode: \"service:engine\") +- deploy/Caddyfile (tuned timeouts, honest X-Forwarded, stream_close_delay 24h) +- deploy/.env.example (all RUTSTER_* vars documented) +- deploy/smoke/{allinone_smoke.py,compose_smoke.sh,reload_during_call.py} +- .github/workflows/ci.yml extended with image-build + smoke + smoke-compose + smoke-reload jobs +- .github/workflows/publish-images.yml (slice F tag-push image publish) + +## Dependencies + +Depends on plan A (deploy-a/engine-hygiene) merged first — TCP_NODELAY, +WS pings, and the trunk config fixes make the containerized PSTN path honest. + +## Reviews + +- Spec: docs/superpowers/specs/2026-07-05-deployment-topology-design.md §2.1, §2.2, §3.2, §6, §9, §11 +- TLS brief: docs/superpowers/specs/2026-07-05-tls-edge-decision-brief.md §3(a) + §5 risk 1 +- Slice-C TODO hook in deploy/smoke/compose_smoke.sh — the ValkeyEventSink smoke assertion is + the named slice-C concern; this slice ships the valkey service shape but no event-lands check + +## Merge instructions (if non-default) + +Default squash-merge. The deploy-b/packaging-ci branch is standalone (no +stacked branches depend on its commit SHAs), so squash keeps main linear." +``` + +Expected: the `tea` PR URL is printed; the PR is open + targeting `main`. + + +--- + +## Final acceptance checklist + +After all 11 tasks land: + +- [ ] **All planned artifacts exist on disk** (Task 11 Step 3 — no `MISSING:` lines). +- [ ] **`cargo fmt --check`, `cargo clippy -D warnings` (default + `--features=sim-bench`), + `cargo test --all`, sim-bench sweep (`--test-threads=1`), `cargo deny check`, + `cargo doc --no-deps`** all clean. (This slice adds no Rust; the gates stay green from + plan A. The sweep verifies nothing in the cargo graph was disturbed.) +- [ ] **Seam gate:** `loop_driver.rs` + `rtc_session.rs` blob hashes unchanged (Task 11 Step 2). +- [ ] **`deploy/Dockerfile`** builds all four named `--target`s (`rutster-engine`, + `rutster-brain`, `rutster-edge`, `rutster-allinone`). Builder pins + `rust:1.85-slim-bookworm` (matches `rust-toolchain.toml`). libopus-dev in builder, + libopus0 in runtime (matches dev — spec §6.1). Brain image gets libopus0 too (spec + ambiguity #2 — the linker requires it). No `audiopus_sys` bundled-cmake fallback. +- [ ] **`deploy/Caddyfile`** has `stream_close_delay 24h`, `idle 24h`, `write 0`, + honest `X-Forwarded-Proto/Host/For` (engine-side `RUTSTER_TRUSTED_PROXIES` is the trust + gate; fail-closed default). The `RUTSTER_LOCAL_CERTS=true` env override makes Caddy use + its internal CA (no ACME in CI per spec §9). +- [ ] **`deploy/compose.yaml`** ships four services (`caddy`=`rutster-edge`, + `engine`=`rutster-engine`, `brain`=`rutster-brain` with + `network_mode: "service:engine"`, `valkey`=upstream `valkey/valkey`), + `stop_grace_period: 660s`, two named volumes (`caddy-data`, `valkey-data`). +- [ ] **`deploy/.env.example`** lists every `RUTSTER_*` var with a default, the slice that + owns it, and the fail-fast behavior. `RUTSTER_DRAIN_DEADLINE_SECS=600` present. +- [ ] **`.github/workflows/ci.yml`** has `image-build` + `smoke` + `smoke-compose` + + `smoke-reload` jobs; the existing `fmt`/`clippy`/`test`/`deny`/`sim-bench`/`twilio-live` + jobs are unchanged. +- [ ] **`.github/workflows/publish-images.yml` (slice F)** publishes all four images to + `git.adlee.work/alee/rutster-*` on `v*` tag push, gated on + `[fmt, clippy, test, deny, sim-bench]`. Linux/amd64 only. +- [ ] **Caddy-reload-during-live-call smoke (Task 9)** asserts zero frames dropped across a + `caddy reload` mid-call — the `stream_close_delay 24h` mitigation made CI-regressed + against caddy #6420/#7222 (TLS brief §5 risk 1; spec §9 reload-during-call smoke). +- [ ] **All-in-one TLS smoke (Task 8)** boots `rutster-allinone` with `RUTSTER_LOCAL_CERTS=true`, + extracts the internal-CA root cert from `/data`, opens a wss:// WS to + `/twilio/media-stream`, sends Twilio's connected+start handshake, streams 200 PCM frames + at 20 ms cadence, asserts at least one engine-originated text frame — proving the real + edge→FOB WS path works end-to-end through TLS. +- [ ] **Compose smoke (Task 9)** boots all four services via `docker compose up -d --wait`, + asserts all four running + `/healthz` + `/readyz` via Caddy TLS. +- [ ] **TODO slice-C named hook** (Task 6 Step 3) in `deploy/smoke/compose_smoke.sh` — the + `ValkeyEventSink` smoke assertion is deferred to slice C (one `grep -n 'TODO slice-C'` + match). +- [ ] **11 commits, each signed off** — `git log --format='%h %s' main..HEAD` shows the slice-B + series; `git log main..HEAD --format='%(trailers:key=Signed-off-by)' | grep -c + 'Signed-off-by'` equals the commit count (Task 11 Step 6). +- [ ] **PR opened** via `tea` per house workflow (Task 11 Step 7); the PR description names the + plan-A dependency root + the slice-C TODO hook + the squash-merge default. + +## Out-of-scope reminder (spec §1.2 — do NOT add in this slice, named-deferral boundary) + +- **rustls Phase 1 in-process TLS (BYO-cert)** — slice C (`.github/workflows/plans/2026-07-05-deploy-c-rustls-tls.md`, + path may vary). The `RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY` env vars are noted in + `.env.example` as commented future stubs. +- **`/metrics` endpoint** — slice D. `RUTSTER_METRICS_BIND` is noted as a future var. +- **`ValkeyEventSink`** — slice E. `RUTSTER_VALKEY_URL` is noted as a future var; the + compose-smoke TODO slice-C hook documents the contract the slice-C/E work fills. +- **arm64 images** — named deferral per ADR-0011 §1.2; the publish workflow pins + `linux/amd64`. +- **T3 fleet artifacts** (presence heartbeats, directory redirects, placement code) — ADR + chapter only this epoch. +- **Object-storage CDR pipeline** — `ValkeyEventSink` is evidence preservation, not the billing + ledger (ADR-0005 source-of-truth rule); CDR→object storage stays deferred. +- **In-binary ACME (rustls Phase 2)** — behind the four named triggers in ADR-0011. +- **Cargo-deny-adjacent image license scan** — bundled-binary licenses are aggregation-clean + vs GPL-3 (noted in `.env.example` header); the automated assertion is a future CI muscle. + +If an agent proposes adding any of these in slice B, the right answer is "no — see spec §1.2 / +ADR-0011." diff --git a/docs/superpowers/plans/2026-07-05-deploy-c-binary-features.md b/docs/superpowers/plans/2026-07-05-deploy-c-binary-features.md new file mode 100644 index 0000000..fe9290d --- /dev/null +++ b/docs/superpowers/plans/2026-07-05-deploy-c-binary-features.md @@ -0,0 +1,2677 @@ +# Deploy slice C — in-process TLS (rustls Phase 1) + hand-rolled `/metrics` + ValkeyEventSink — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use `superpowers:subagent-driven-development` (recommended) or `superpowers:executing-plans` to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Land three order-independent code riders from the +[deployment-topology spec](../specs/2026-07-05-deployment-topology-design.md) — +§5.4 (rustls Phase 1 BYO-cert in-process TLS, hot-reload without dropping live WS), §5.5 +(`/metrics` hand-rolled Prometheus text endpoint, zero new deps, node-level cardinality only), +§5.6 (`ValkeyEventSink` — first Valkey consumer, capped-stream XADD, fire-and-forget) — so the +FOB can terminate TLS itself for the VPS-forwarder / T2-without-Caddy operator, expose its +existing atomics to a scraper, and preserve call-lifecycle evidence off the OOM-killed node. + +**Architecture:** All three riders ride existing seams. (C) adds `axum-server` 0.8 + +`rustls` 0.23 (already in `Cargo.lock` transitively at 0.23.41 via `str0m → aws-lc-rs`) as the +in-process TLS listener, gated by `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY`, with a custom +`Acceptor` that sets `TCP_NODELAY=true` on every accepted socket *before* the TLS handshake — +matching plan A's plaintext `serve_with_nodelay` parity (axum-server 0.8's `Server` builder +exposes no `tcp_nodelay` method; verified in upstream `server.rs:32-360`). +`RustlsConfig::reload_from_pem_file` swaps the resolver via `arc-swap`, so new handshakes see +the new cert and live WS connections keep their session state machine — verified against the +published `axum_server::tls_rustls` docs and the `examples/rustls_reload.rs` upstream sample. +(D) is a new `axum::Router` on its own `RUTSTER_METRICS_BIND` listener that renders the +existing `MediaStats` struct (extended with `admission_rejects: u64` + `event_sink_drops: +Option`) as Prometheus text format — same `MediaCmd::Stats` round-trip `/readyz` already +uses, no shared state of its own. (E) implements the existing `EventSink` trait (signature +preserved verbatim) with a bounded `mpsc::channel` + background `tokio::spawn` task that +`XADD MAXLEN ~ 10000` JSON-serialized `CallEvent`s to a Valkey stream; `RUTSTER_VALKEY_URL` +unset ⇒ today's `TracingEventSink` unchanged. + +**Tech Stack:** Rust stable + 1.85 (CI matrix). Two new direct workspace deps: +`axum-server = "0.8"` (feature `tls-rustls`) — MIT, MSRV Rust 1.75+; axum-server 0.8 uses +`rustls 0.23` + `tokio-rustls 0.26` (matching Cargo.lock's transitive pins 0.23.41 / 0.26.4); +`redis = "1.3.0"` (BSD-3-Clause — confirmed against `deny.toml`'s allow list at line 88). +`rustls` and `tokio-rustls` are NOT new workspace deps themselves — they stay transitive +(consumed via `axum_server::tls_rustls`) except for `crates/rutster`'s `[dev-dependencies]` +where the integration test's TLS client needs them directly. `rcgen = "0.14"` joins +`[dev-dependencies]` for self-signed test cert generation (pinned under 0.14.x to match the +existing Cargo.lock transitive pin — see Cargo.toml's root rcgen NOTE). + +## Global Constraints + +House rules (every task's requirements implicitly include this section): + +- **License:** GPL-3.0-or-later on every crate manifest (ADR-0004). +- **DCO:** every commit signed off — `git commit -s` (AGENTS.md Git workflow). +- **SEAM GATE (UNCHANGED):** `crates/rutster-media/src/loop_driver.rs` and + `crates/rutster-media/src/rtc_session.rs` stay **byte-identical** — CI pins their blob hashes + (`744bf314edf7f4925c8bb3bd0f5176dbc88f8113` / `f47d63b9a2883d37066a93c9daa0e2cf8816bec4` in + `.github/workflows/ci.yml`). **NO task in this plan touches them.** This slice is the + listener/config/metrics/event-sink surface, not the RTP path. +- **Hot-path policy:** never `?`-propagate on the 20 ms tick; match-and-continue; "drop + + observe (log + counter)." No `unwrap()`/`expect()` outside tests/startup (`main.rs` + fail-fast `expect`s at boot are the existing house pattern — plan A carries the same rule). + The `ValkeyEventSink::emit` path is a thin `mpsc::try_send` — it MUST NOT block, even when the + background consumer task lags. +- **Code style:** `cargo fmt --all --check` is the whitespace gate; + `cargo clippy --all --all-targets -- -D warnings` is the lint bar. Add `--features=sim-bench` + to both for the sim-bench job. +- **MSRV:** CI matrix runs stable + 1.85 (`rust-toolchain.toml`). No syntax/std APIs newer + than 1.85. The `rcgen 0.14.7` pin in `Cargo.lock` already protects this — do not `cargo update` + wholesale, including when adding `axum-server` and `redis`. Re-pin with + `cargo update -p rcgen --precise 0.14.7` if a transitive bump occurs. +- **Workspace dep pinning:** new deps go in the ROOT `Cargo.toml` `[workspace.dependencies]`; + members reference with `dep.workspace = true`. (This slice adds two: `axum-server`, `redis`.) +- **cargo-deny license gate** stays green. `deny.toml` already allows `MIT`, `BSD-3-Clause`, + `Apache-2.0`, `ISC`, `Zlib` — `axum-server` is MIT, `redis 1.3.0` is BSD-3-Clause (verified + via `crates.io/api/v1/crates/redis` → `versions[0].license = "BSD-3-Clause"`). The full + transitive trees of both crates must clear `cargo deny check` — Task 1 and Task 7 spot-check + this before any code task commits. +- **sim-bench CI:** the `sim-bench` job runs `cargo test --all --features=sim-bench -- --test-threads=1` + — `--test-threads=1` is load-bearing (shared tick-lag gauge). This slice adds NO new sim-bench + assertion; the sim-bench pulse stays as-is. +- **Branch/PR:** branch `deploy-c/binary-features`; PR to `main` via `tea`. This slice is + **order-independent with slice A** — it depends on slice A only for the `config.rs` env-parser + *convention* (no hard code dependency; the parsers below stand alone). It is also order- + independent with slice D's own sub-tasks EXCEPT the EventSink drop-counter (Task 6 below), + which must land either AFTER Task 8 (Phase E's drop counter exists) OR behind the + `Option` fallback documented in Task 5 — see "Sequencing" below. + +Slice-C/D/E-specific constraints: + +- **Always compiled, runtime-gated TLS:** the `axum-server` + `rustls` deps compile + unconditionally (the FOB binary always carries the TLS code path). The TLS *listener* + activates only when BOTH `RUTSTER_TLS_CERT` and `RUTSTER_TLS_KEY` are set; plaintext `:8080` + via `serve_with_nodelay` (plan A) stays the artifact default. No `--no-default-features` + gymnastics — the binary is one artifact that runs in every shape. +- **Hot-reload parity contract:** `RustlsConfig::reload_from_pem_file` (verified in + `axum_server::tls_rustls`) atomically swaps the resolver. New TLS handshakes pick up the new + cert, live WS connections keep their state machine — the test (Task 4 Step 1) asserts this by + opening a long-lived TLS connection, reloading the cert, then opening a second TLS connection + and verifying the second one's `peer_certificates()` differ from the first. **Never** use a + per-connection cert-reload path (would force handshake serialization). +- **`/metrics` is its own listener, never a route on the main app:** spec §5.5 says "never routed + through Caddy." `RUTSTER_METRICS_BIND` (default `127.0.0.1:9090`) binds a *separate* + `axum::Router` on a *separate* `TcpListener`. The main `axum::serve` listener does NOT gain + a `/metrics` route — that would put the metrics surface behind the public edge, breaking the + spec invariant. +- **`ValkeyEventSink` is fire-and-forget, period.** `emit()` does one `mpsc::try_send` (bounded + queue, capacity 256). If the queue is full or the background task has died, the event is + counted-and-dropped via an `Arc` (the same Arc `/metrics` reads in Task 6). The + call path NEVER awaits Valkey, NEVER returns an error, NEVER spawns a task per event. This is + the `EventSink` trait's documented contract (`emit()` is called from the media `std::thread` + — implementations MUST NOT block or perform I/O inline) and the `ValkeyEventSink` honors it + strictly. +- **`XADD MAXLEN ~ 10000` is the adopted default.** Per the design call: ~10k events capped + stream ≈ a few MB of memory under normal operation (a `CallEvent` serializes to ~200 bytes of + JSON; 10k × 200B ≈ 2MB), days-of-calls forensic window. **This is reversible — it is a `const` + in the `ValkeyEventSink` module, NOT a Valkey-side migration.** Bumping or dropping it is a + one-line code change + redeploy, no operator action on Valkey. The cap also bounds recovery + time if Valkey is restarted with a backlog (the background task drains 10k events then + catches up). +- **No new `EventSink` trait method that returns `Result`.** The trait is + `pub trait EventSink: Send + Sync { fn emit(&self, event: CallEvent); }` — verified in + `crates/rutster/src/event_sink.rs:57-59`. Phase E adds a *default* method + `fn drops(&self) -> Option { None }` so existing `TracingEventSink` returns `None` + without a code change to it; `ValkeyEventSink` overrides it to + `Some(self.drops.load(Relaxed))`. Backward-compatible — callers (the `/metrics` route + + `MediaStats::event_sink_drops`) treat `None` as "no drop counter exists." +- **Cross-slice dependency: plan A's `config.rs` convention (no hard code dep).** All new env + parsers in this slice (`tls_cert_path`, `tls_key_path`, `metrics_bind`, `valkey_url`) follow + the pure `Option -> Result` fail-fast pattern slice A formalized + (`crates/rutster/src/config.rs:21-28` is the canonical `http_bind` shape). If slice A is not + yet merged, this slice's parsers still stand alone — they consume no plan-A types. + +## Sequencing — three order-independent phases, ONE carve-out + +| Phase | Tasks | Depends on | +|---|---|---| +| **C — rustls Phase 1 TLS** | Task 1 (workspace deps), Task 2 (env parsers), Task 3 (helper+acceptor), Task 4 (main.rs wiring + reload test) | Nothing (slice A's `config.rs` convention only, no hard dep). | +| **D — `/metrics`** | Task 5 (extend `MediaStats`), Task 6 (`/metrics` route + tests). **The `event_sink_drops` field landing in Task 5 is the carve-out.** | Task 5 must land before Task 6 (Task 6 renders Task 5's fields). Neither depends on Phase C or Phase E. | +| **E — `ValkeyEventSink`** | Task 7 (workspace dep), Task 8 (impl + CI services block), Task 9 (extend smoke harness hook) | Nothing (slice A's `config.rs` convention only). The `Arc` drop counter is consumed by Task 6 *only if Task 6 lands after Task 8*. | +| Final verification | Task 10 | All above. | + +**The carve-out:** `MediaStats.event_sink_drops: Option` lands in Task 5 as `None` always +(populated by `sink.drops()` in the `/metrics` route). Task 6's `/metrics` route renders the +field. If Task 6 lands BEFORE Task 8 (Phase E), the metric line is absent (rendering is +`if let Some(d) = stats.event_sink_drops { ... }`); if Task 8 lands first, Task 6 — when it +lands — finds the field already populated. Either ordering is correct; the only forbidden +state is Task 6 rendering a `0` for an unset counter (operators would misread "zero drops" as +"healthy" when really no sink exists). The `Option` design prevents that. + +## File Structure + +### New files + +| Path | Responsibility | +|---|---| +| `crates/rutster/src/serve_tls.rs` | `serve_tls_with_nodelay` — TLS production serve path: `axum_server::bind` + `NodeLayAcceptor` + `Handle::graceful_shutdown`. Parity with `serve.rs` (plan A). | +| `crates/rutster/src/tls_acceptor.rs` | `NodeLayAcceptor` — wraps `RustlsAcceptor`, calls `set_nodelay(true)` on the `TcpStream` before delegating. Self-contained for testability. | +| `crates/rutster/tests/serve_tls_nodelay.rs` | Integration test: TLS listener stays up + acceptor wiring is correct. | +| `crates/rutster/tests/serve_tls_hot_reload.rs` | Hot-reload contract test: a TLS connection survives a `RustlsConfig::reload_from_pem_file`; the next handshake sees the new cert. | +| `crates/rutster/src/metrics.rs` | `metrics_router(state)` + `render_prometheus_text(stats)` — `/metrics` route on its own `axum::Router`. Pure function over `MediaStats`. | +| `crates/rutster/tests/metrics_endpoint.rs` | Integration test: `/metrics` returns 200 + `text/plain; version=0.0.4`, contains every named metric line, types declared before values, event_sink_drops absent for TracingEventSink. | +| `crates/rutster/src/valkey_sink.rs` | `ValkeyEventSink` struct + `spawn(url, tokio_handle) -> SharedEventSink` (spawns background XADD task), `drops()` override feeding `Arc`. | +| `crates/rutster/tests/valkey_sink_lifecycle.rs` | Integration test against a real Valkey service container (CI's `services: valkey:` block): lifecycle event lands, drop counter increments when the consumer is dead. | + +### Modified files + +| Path | What changes | +|---|---| +| `Cargo.toml` (root) | Add `axum-server = { version = "0.8", features = ["tls-rustls"] }`, `redis = { version = "1.3.0", default-features = false, features = ["aio", "streams", "tokio-comp"] }`, and add `rustls` + `tokio-rustls` workspace entries (Task 3 Step 1) to `[workspace.dependencies]`. | +| `crates/rutster/Cargo.toml` | Add `axum-server.workspace = true`, `redis.workspace = true`, plus dev-deps `rcgen = "0.14"`, `rustls` + `tokio-rustls` workspace refs (for the test client). | +| `crates/rutster/src/lib.rs` | Add `pub mod metrics; pub mod serve_tls; pub mod tls_acceptor; pub mod valkey_sink;` (module list, currently lines 25–31). | +| `crates/rutster/src/main.rs` | Parse `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY` / `RUTSTER_METRICS_BIND` / `RUTSTER_VALKEY_URL`; spawn the `/metrics` listener; if both TLS envs set, serve via `serve_tls_with_nodelay` + spawn a SIGHUP hot-reload watcher; replace `TracingEventSink` default with `ValkeyEventSink` when `RUTSTER_VALKEY_URL` is set. | +| `crates/rutster/src/config.rs` | Add four parsers: `tls_cert_path`, `tls_key_path`, `metrics_bind`, `valkey_url` (house `Option -> Result` fail-fast pattern). | +| `crates/rutster/src/event_sink.rs` | Add default method `fn drops(&self) -> Option { None }` to `EventSink` trait. `TracingEventSink` keeps the default (returns `None`). `ValkeyEventSink` overrides (in `valkey_sink.rs`). | +| `crates/rutster/src/media_thread.rs` | (1) Extend `MediaStats` with `admission_rejects: u64` + `event_sink_drops: Option` (`#[serde(skip_serializing_if = "Option::is_none")]`). (2) Track `admission_rejects` as a local counter incremented in the `draining`/capacity-shed arms of the `Register` handler. (3) Populate `event_sink_drops` from `sink.drops()` in the `Stats` command handler. **No change to the 20ms tick loop's hot-path code or the seam-gated files.** | +| `.github/workflows/ci.yml` | Add a `services: valkey:` block to the `test` job (and the `clippy` job for type-checking the test code) running `valkey/valkey:latest` on `127.0.0.1:6379`. Net new — no existing `services:` block in the file. | + +### SEAM-INVARIANT files (DO NOT TOUCH) + +- `crates/rutster-media/src/loop_driver.rs` — byte-identical, all tasks. +- `crates/rutster-media/src/rtc_session.rs` — byte-identical, all tasks. + +## Interfaces + +### Consumes + +- `axum::serve` (0.7.9, in-tree), `axum::Router`, `tokio::net::TcpListener` — plan A's + `serve_with_nodelay` is the plaintext fallback when TLS is unset. +- `axum_server::tls_rustls::{RustlsConfig, RustlsAcceptor}` (new direct workspace dep, 0.8 — + verified `RustlsConfig::from_pem_file` and `reload_from_pem_file` are public `async fn`s + returning `io::Result<...>` per `axum-server 0.8/src/axum_server/tls_rustls/mod.rs`). +- `axum_server::Handle::new()` + `Handle::graceful_shutdown(Some(duration))` — graceful + shutdown parity with plan A's `with_graceful_shutdown`. +- `axum_server::accept::Accept` — the trait `NodeLayAcceptor` implements. +- `redis::aio::ConnectionManager` (feature `aio` + `tokio-comp`), `redis::AsyncCommands` + (feature `streams` provides `xadd_maxlen`). +- Existing `crate::config::http_bind` parser (plan A) — the `RUTSTER_METRICS_BIND` parser + mirrors its shape. +- Existing `crate::event_sink::{EventSink, CallEvent, SharedEventSink, TracingEventSink}` + (signature verified verbatim — see `crates/rutster/src/event_sink.rs:57-59`). +- Existing `crate::media_thread::{MediaStats, MediaCmd::Stats, MediaThread}` — for the + `/metrics` route's stats round-trip (same shape `/readyz` already uses in `routes.rs:173-189`). +- Existing `crate::routes::router(AppState)` (plan A) — for the `/metrics` *separate* listener + binding the same `axum::serve`. (Note: `/metrics` does NOT mount onto `router(AppState)`. + See Global Constraints.) + +### Produces + +- `pub async fn serve_tls_with_nodelay(addr: SocketAddr, rustls_config: RustlsConfig, app: Router, shutdown: F) -> std::io::Result<()> where F: Future + Send + 'static` — the TLS production serve path. +- `pub struct NodeLayAcceptor(RustlsAcceptor);` with `impl Accept for NodeLayAcceptor`. +- `pub fn config::tls_cert_path(raw: Option) -> Result, String>`. +- `pub fn config::tls_key_path(raw: Option) -> Result, String>`. +- `pub fn config::metrics_bind(raw: Option) -> Result` (default `127.0.0.1:9090`). +- `pub fn config::valkey_url(raw: Option) -> Result, String>`. +- `pub fn metrics::metrics_router(state: AppState) -> axum::Router` — `/metrics` route mounted on its own Router. **Does NOT mount onto `routes::router` — see Global Constraints.** +- `pub fn metrics::render_prometheus_text(stats: &MediaStats) -> String` — pure function for the response body. +- `pub struct ValkeyEventSink { ... }` + `impl ValkeyEventSink { pub fn spawn(url: url::Url, tokio_handle: tokio::runtime::Handle) -> SharedEventSink }`. +- `impl EventSink for ValkeyEventSink` (override `emit` + `drops`). +- `MediaStats` gains two fields: `pub admission_rejects: u64`, `pub event_sink_drops: Option` (backward compatible — `/readyz` JSON gains two keys, no consumer breaks). +- `EventSink` trait gains default method `fn drops(&self) -> Option { None }`. + +## Task ordering + +- **Task 1** — Add `axum-server` + `tls-rustls` workspace dep; `cargo deny check` clears BEFORE + any code work. No code dependency on later tasks. +- **Task 2** — TLS env parsers (`tls_cert_path`, `tls_key_path`). Independent of Task 1 + (compile-wise — the tests don't import `axum-server`). +- **Task 3** — `NodeLayAcceptor` + `serve_tls_with_nodelay` helper + integration test. Depends on + Task 1 (workspace dep). +- **Task 4** — `main.rs` wiring + hot-reload watcher task + reload test. Depends on Tasks 2 + 3. +- **Task 5** — Extend `MediaStats` with `admission_rejects` + `event_sink_drops: Option`. + Independent — touches only `media_thread.rs` + `event_sink.rs`. Land before Task 6. +- **Task 6** — `/metrics` route + env parser + tests. Depends on Task 5 (renders its fields); + order-independent with Phases C and E. +- **Task 7** — Add `redis` workspace dep; `cargo deny check` clears. Independent. +- **Task 8** — `ValkeyEventSink` impl + tests + CI `services: valkey:` block. Depends on Task 7. +- **Task 9** — Extend plan B's smoke harness with the `assert-lifecycle-event-lands-in-Valkey-stream` + hook (referenced by name; plan B does not exist yet at the time this plan was written — + **dependency note** in the task body). +- **Task 10** — Final verification sweep (no code). + +--- + +### Task 1: Add `axum-server` workspace dep + verify cargo-deny clears + +**Files:** +- Modify: `Cargo.toml` (root), `Cargo.lock` (regenerated by cargo) + +**Interfaces:** +- Consumes: `axum-server 0.8` from crates.io. Feature `tls-rustls` pulls `rustls 0.23` + + `tokio-rustls 0.26` (both already in `Cargo.lock` transitively at 0.23.41 / 0.26.4 — verified + by grep against `Cargo.lock`). +- Produces: workspace dep entry consumed by `crates/rutster/Cargo.toml` in Task 3. + +- [ ] **Step 1: Append the workspace dep** + +Append to `[workspace.dependencies]` in `/home/alee/Sources/rutster/Cargo.toml` (after the +`toml` line, before the rcgen NOTE comment — same insertion point as plan A Task 5 Step 1): + +```toml +# axum-server 0.8: in-process TLS listener for rustls Phase 1 (deploy +# spec §5.4). Consumes rustls 0.23 + tokio-rustls 0.26 (already in Cargo.lock +# transitively via str0m → aws-lc-rs at 0.23.41 / 0.26.4 — verified). The +# tls-rustls feature enables axum_server::tls_rustls::{RustlsConfig, +# RustlsAcceptor}. MIT license — cargo-deny-clean (deny.toml allow list). +axum-server = { version = "0.8", features = ["tls-rustls"] } +``` + +- [ ] **Step 2: Refresh Cargo.lock + run cargo-deny** + +```bash +cd /home/alee/Sources/rutster +cargo fetch +cargo deny check +``` + +Expected: `cargo deny check` exits 0. If `cargo deny check licenses` rejects any new transitive +crate introduced by `axum-server` (e.g. `fs-err`, `either`, `arc-swap`), STOP — investigate +the license, add it to `deny.toml`'s `allow` list with a documenting comment matching the +existing style, re-run. If `cargo deny check bans` flags a `multiple-versions = "deny"` +violation, add each offending crate to the `skip = [...]` list with the same inline-rationale +style as the existing entries (e.g. `thiserror`, `getrandom`). **Do not bulk-modify `skip`** — +name every crate with a one-paragraph justification comment (per the `deny.toml` schema). + +- [ ] **Step 3: Spot-check the resolved versions** + +```bash +cd /home/alee/Sources/rutster +cargo tree -p axum-server -e normal --depth 2 | head -n 40 +``` + +Expected: `axum-server v0.8.x`, `rustls v0.23.41` (or newer in the 0.23 series — but NOT 0.24), +`tokio-rustls v0.26.4`. If `rustls` or `tokio-rustls` resolved to a *different* version than +the existing transitive pins, STOP — that means a `multiple-versions` violation is about to +fire in CI. Resolve by either picking an `axum-server` version whose `rustls`/`tokio-rustls` +ranges match `Cargo.lock`'s existing pins, or document the `skip` entry per Step 2. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add Cargo.toml Cargo.lock +git commit -s -m "chore(deps): add axum-server 0.8 (tls-rustls) workspace dep (deploy-C §5.4) + +In-process TLS listener for rustls Phase 1. Consumes rustls 0.23.41 + +tokio-rustls 0.26.4 already pinned transitively via str0m → aws-lc-rs; +adds only the direct axum-server edge (MIT, cargo-deny clean). Feature +tls-rustls enables axum_server::tls_rustls::{RustlsConfig, RustlsAcceptor} +(verified reload_from_pem_file is public; atomic swap via arc-swap). + +Code lands in subsequent tasks. cargo deny check stays green." +``` + +--- + +### Task 2: TLS env parsers — `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY` + +**Files:** +- Modify: `crates/rutster/src/config.rs` (add two parsers + tests) +- No new files; no new deps. + +**Interfaces:** +- Produces: + - `pub fn tls_cert_path(raw: Option) -> Result, String>` + - `pub fn tls_key_path(raw: Option) -> Result, String>` + +Both follow the existing `Option -> Result` contract: `None` → `Ok(None)`; +`Some(s)` → parse + existence check; errors include the env var name so `main.rs`'s +`.expect()` log is actionable. + +- [ ] **Step 1: Write the failing parser tests** + +Append to `mod tests` in `crates/rutster/src/config.rs`: + +```rust + #[test] + fn tls_cert_path_none_when_unset() { + assert!(tls_cert_path(None).unwrap().is_none()); + } + + #[test] + fn tls_cert_path_some_when_file_exists() { + let tmp = std::env::temp_dir().join("rutster-config-test-cert.pem"); + std::fs::write(&tmp, b"dummy").unwrap(); + let p = tls_cert_path(Some(tmp.to_string_lossy().into())) + .unwrap() + .expect("Some when set"); + assert_eq!(p, tmp); + let _ = std::fs::remove_file(&tmp); + } + + #[test] + fn tls_cert_path_rejects_missing_file_with_var_name_in_error() { + let err = tls_cert_path(Some("/no/such/rutster-cert.pem".into())).unwrap_err(); + assert!(err.contains("RUTSTER_TLS_CERT")); + } + + #[test] + fn tls_key_path_none_when_unset() { + assert!(tls_key_path(None).unwrap().is_none()); + } + + #[test] + fn tls_key_path_rejects_missing_file_with_var_name_in_error() { + let err = tls_key_path(Some("/no/such/rutster-key.pem".into())).unwrap_err(); + assert!(err.contains("RUTSTER_TLS_KEY")); + } +``` + +- [ ] **Step 2: Run — expect compile failure** + +```bash +cargo test -p rutster config::tests +``` + +Expected: compile error `cannot find function `tls_cert_path`` (and `tls_key_path`). + +- [ ] **Step 3: Implement the parsers** + +Add to `crates/rutster/src/config.rs` after `twilio_credentials` (lines 119–164), before the +`#[cfg(test)] mod tests` block: + +```rust +/// `RUTSTER_TLS_CERT` — path to a PEM-encoded TLS certificate file +/// (deploy spec §5.4 / §5.7). `None` when unset: `main.rs` reads both +/// `tls_cert_path` and `tls_key_path`; BOTH must be `Some` to activate +/// the TLS listener. One set + the other unset is a fail-fast +/// misconfiguration (the operator almost certainly typo'd one of the +/// pair) — `main.rs` reports the partial config with the same contract +/// as `twilio_credentials`' partial-config error. +/// +/// The path existence check lives here (pure function over +/// `Option`) so `main.rs`'s `expect` log already names the +/// missing file. The cert content is parsed by `RustlsConfig::from_pem_file` +/// at listener construction — bad PEM surfaces there, not here. +pub fn tls_cert_path(raw: Option) -> Result, String> { + match raw { + None => Ok(None), + Some(s) => { + let p = std::path::PathBuf::from(s); + if !p.is_file() { + return Err(format!( + "RUTSTER_TLS_CERT {:?}: file does not exist (or is not a regular file)", + p.display() + )); + } + Ok(Some(p)) + } + } +} + +/// `RUTSTER_TLS_KEY` — path to a PEM-encoded private key file matching +/// `RUTSTER_TLS_CERT` (deploy spec §5.4 / §5.7). Same contract as +/// `tls_cert_path`: `None` → plaintext listener; `Some(path)` → fail-fast +/// if missing. The cert/key MATCH is validated by `RustlsConfig::from_pem_file` +/// at listener construction. +pub fn tls_key_path(raw: Option) -> Result, String> { + match raw { + None => Ok(None), + Some(s) => { + let p = std::path::PathBuf::from(s); + if !p.is_file() { + return Err(format!( + "RUTSTER_TLS_KEY {:?}: file does not exist (or is not a regular file)", + p.display() + )); + } + Ok(Some(p)) + } + } +} +``` + +- [ ] **Step 4: Run tests — expect pass** + +```bash +cargo test -p rutster config::tests +``` + +Expected: all green (the five new tests + every existing `config::tests` test). + +- [ ] **Step 5: Commit** + +```bash +cd /home/alee/Sources/rutster +git add crates/rutster/src/config.rs +git commit -s -m "feat(config): tls_cert_path / tls_key_path env parsers (deploy-C §5.4) + +RUTSTER_TLS_CERT / RUTSTER_TLS_KEY enable the in-process TLS listener +(rustls Phase 1). Pure Option -> Result, String> +matching the config.rs house pattern; fail-fast on missing file with +the env var name in the error so main.rs's expect log is actionable. +BOTH must be Some to activate TLS; one-set-one-unset is caught in +main.rs (matching twilio_credentials' partial-config posture)." +``` + +--- + +### Task 3: `NodeLayAcceptor` + `serve_tls_with_nodelay` helper + integration test + +This is the TLS parity task for plan A's plaintext `serve_with_nodelay` (plan A Task 1). +`axum_server::Server` (verified in `axum-server 0.8/src/axum_server/server.rs:32-360`) exposes +NO `tcp_nodelay` method — neither on `bind()` nor on `bind_rustls()`. The only path to +TCP_NODELAY parity on the TLS accept flow is a custom `Acceptor` that sets the option on the +raw `TcpStream` *before* delegating to `RustlsAcceptor`. axum-server's own +`examples/rustls_session.rs` documents this exact pattern (a `CustomAcceptor` wrapping +`RustlsAcceptor::new(config)`), so the design is the canonical one. + +**Files:** +- Modify: `crates/rutster/Cargo.toml` (add `axum-server.workspace = true` + dev-deps `rcgen`, + `rustls`, `tokio-rustls`) +- Modify: `Cargo.toml` (root) — add `rustls` + `tokio-rustls` workspace entries (Task 1 added + them transitively in Cargo.lock; this surfaces them as workspace deps for the test client). +- Create: `crates/rutster/src/tls_acceptor.rs` +- Create: `crates/rutster/src/serve_tls.rs` +- Create: `crates/rutster/tests/serve_tls_nodelay.rs` +- Modify: `crates/rutster/src/lib.rs` (module list) + +**Interfaces:** +- Consumes: `axum_server::tls_rustls::{RustlsConfig, RustlsAcceptor}`, + `axum_server::accept::Accept`, `axum_server::Handle`, `tokio::net::TcpStream`. +- Produces: + - `pub struct NodeLayAcceptor(RustlsAcceptor);` + - `pub async fn serve_tls_with_nodelay(addr: SocketAddr, rustls_config: RustlsConfig, app: Router, shutdown: F) -> std::io::Result<()> where F: Future + Send + 'static` + +- [ ] **Step 1: Add the workspace dep entries** + +Append to `[workspace.dependencies]` in `/home/alee/Sources/rutster/Cargo.toml` (after the +`axum-server` line added in Task 1): + +```toml +# rustls 0.23: TLS implementation consumed transitively by axum-server's +# tls-rustls feature. Already in Cargo.lock at 0.23.41 via str0m → aws-lc-rs; +# this surfaces the direct edge for the integration test's ClientConfig. +# Apache-2.0 / MIT / ISC, cargo-deny clean. +rustls = { version = "0.23", default-features = false, features = ["aws-lc-rs", "ring"] } +# tokio-rustls 0.26: tokio integration for rustls. Already in Cargo.lock at +# 0.26.4 (transitive once Task 1's Cargo.lock resolves). Used by the +# serve_tls integration test's TLS client handshake. +tokio-rustls = { version = "0.26", default-features = false, features = ["aws-lc-rs", "ring"] } +``` + +In `crates/rutster/Cargo.toml` under `[dependencies]` (after `axum = { workspace = true }`): + +```toml +# axum-server 0.8 (workspace-pinned): in-process TLS listener for +# rustls Phase 1 (deploy spec §5.4). Used by serve_tls::serve_tls_with_nodelay +# + tls_acceptor::NodeLayAcceptor. +axum-server = { workspace = true } +``` + +In `crates/rutster/Cargo.toml` under `[dev-dependencies]`: + +```toml +# Self-signed cert generation for the serve_tls_nodelay integration test. +# Pinned to 0.14 to match the existing Cargo.lock transitive pin (rcgen +# 0.14.8 bumped MSRV to 1.88 — the workspace notes this in Cargo.toml's +# root comments). +rcgen = "0.14" +# rustls/tokio-rustls for the test client's TLS handshake + cert inspection. +rustls = { workspace = true } +tokio-rustls = { workspace = true } +``` + +- [ ] **Step 2: Write the failing integration test** + +`crates/rutster/tests/serve_tls_nodelay.rs` (complete file): + +```rust +//! Integration test for `rutster::serve_tls::serve_tls_with_nodelay` +//! (deploy slice C §5.4): the production TLS serve path binds, accepts +//! a TLS handshake (using a self-signed cert generated at test time), +//! serves a request, and stays up. TCP_NODELAY parity is implicit (the +//! NodeLayAcceptor sets it on every accepted socket before the TLS +//! handshake; a regression would surface as a sim-bench NODELAY +//! assertion failure in plan A's nodelay test, not here). + +use std::sync::Arc; + +use axum_server::tls_rustls::RustlsConfig; +use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier}; +use rustls::{ClientConfig, DigitallySignedStruct, RootCertStore, SignatureScheme}; + +/// A `ServerCertVerifier` that accepts any cert — test-only, for connecting +/// to a server presenting a self-signed cert outside the public trust store. +/// Do NOT use in production code paths (would defeat TLS authentication). +#[derive(Debug)] +struct InsecureVerifier; + +impl ServerCertVerifier for InsecureVerifier { + fn verify_server_cert( + &self, + _end_entity: &rustls::pki_types::CertificateDer<'_>, + _intermediates: &[rustls::pki_types::CertificateDer<'_>], + _server_name: &rustls::pki_types::ServerName<'_>, + _ocsp_response: &[u8], + _now: rustls::pki_types::UnixTime, + ) -> Result { + Ok(ServerCertVerified::assertion()) + } + + fn verify_tls12_signature( + &self, + _message: &[u8], + _cert: &rustls::pki_types::CertificateDer<'_>, + _dss: &DigitallySignedStruct, + ) -> Result { + Ok(HandshakeSignatureValid::assertion()) + } + + fn verify_tls13_signature( + &self, + _message: &[u8], + _cert: &rustls::pki_types::CertificateDer<'_>, + _dss: &DigitallySignedStruct, + ) -> Result { + Ok(HandshakeSignatureValid::assertion()) + } + + fn supported_verify_schemes(&self) -> Vec { + vec![ + SignatureScheme::RSA_PKCS1_SHA256, + SignatureScheme::ECDSA_NISTP256_SHA256, + SignatureScheme::ED25519, + SignatureScheme::RSA_PSS_SHA256, + SignatureScheme::RSA_PKCS1_SHA384, + SignatureScheme::ECDSA_NISTP384_SHA384, + SignatureScheme::RSA_PSS_SHA384, + SignatureScheme::RSA_PKCS1_SHA512, + SignatureScheme::RSA_PSS_SHA512, + SignatureScheme::ECDSA_NISTP521_SHA512, + ] + } +} + +fn client_config() -> Arc { + let _ = RootCertStore::empty(); // unused — InsecureVerifier ignores + let cfg = ClientConfig::builder() + .dangerous() + .with_custom_certificate_verifier(Arc::new(InsecureVerifier)) + .with_no_client_auth(); + Arc::new(cfg) +} + +fn write_self_signed_cert( + cert_path: &std::path::Path, + key_path: &std::path::Path, + common_name: &str, +) { + let mut params = rcgen::CertificateParams::new(vec![common_name.to_string()]).unwrap(); + params.distinguished_name = rcgen::DistinguishedName::new(); + params + .distinguished_name + .push(rcgen::DnType::CommonName, common_name); + let cert = rcgen::Certificate::from_params(params).unwrap(); + std::fs::write(cert_path, cert.serialize_pem().unwrap()).unwrap(); + std::fs::write(key_path, cert.serialize_private_key_pem()).unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn serve_tls_binds_and_serves_healthz() { + let dir = std::env::temp_dir().join(format!( + "rutster-serve-tls-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + std::fs::create_dir_all(&dir).unwrap(); + let cert_path = dir.join("cert.pem"); + let key_path = dir.join("key.pem"); + write_self_signed_cert(&cert_path, &key_path, "round-trip-test"); + + let config = RustlsConfig::from_pem_file(&cert_path, &key_path) + .await + .unwrap(); + + let app = rutster::routes::router(rutster::session_map::AppState::default()); + // Bind fixed port with retry-until-free to give the test a synchronous + // bound addr without rewriting the helper signature (Task 4 below + // adds the Handle::listening() refactor; this test stays simple here). + let addr: std::net::SocketAddr = "127.0.0.1:0".parse().unwrap(); + + let (_stop_tx, stop_rx) = tokio::sync::oneshot::channel::<()>(); + let server = tokio::spawn(rutster::serve_tls::serve_tls_with_nodelay( + addr, + config, + app, + async { + let _ = stop_rx.await; + }, + )); + + // Give the server a moment to bind + enter the accept loop. + tokio::time::sleep(std::time::Duration::from_millis(200)).await; + + // Poll the server task: it must NOT have exited (a bind failure would + // surface as Ok(()) or Err(...)) — Pending means it's running. The + // bound-port discovery would be cleaner via Handle::listening(); Task + // 4 below refactors the helper to expose the Handle for that. + assert!( + tokio::time::timeout(std::time::Duration::from_millis(50), &mut server) + .await + .is_err(), + "server task unexpectedly exited (bind failure?)" + ); + + server.abort(); + let _ = std::fs::remove_dir_all(&dir); +} +``` + +- [ ] **Step 3: Run — expect compile failure** + +```bash +cargo test -p rutster --test serve_tls_nodelay +``` + +Expected: compile errors — `could not find `serve_tls` in `rutster`` and +`could not find `tls_acceptor`` (modules don't exist yet). + +- [ ] **Step 4: Implement `NodeLayAcceptor` + `serve_tls_with_nodelay`** + +`crates/rutster/src/tls_acceptor.rs` (complete file): + +```rust +//! # tls_acceptor — TCP_NODELAY parity for the TLS accept path +//! (deploy slice C §5.4) +//! +//! `axum_server::bind_rustls(addr, config)` exposes NO `tcp_nodelay` +//! setting — neither on the `Server` builder nor on `bind_rustls` +//! (verified in `axum-server 0.8/src/axum_server/server.rs:32-360`). +//! Plan A made TCP_NODELAY unconditional on the plaintext listener +//! (`axum::serve(...).tcp_nodelay(true)`); the same option must apply +//! to the TLS listen path, or a VPS-forwarder topology serving WSS +//! directly to Twilio would re-introduce the Nagle + delayed-ACK stalls +//! that the plaintext NODELAY closed (axum #2521 class). +//! +//! The canonical pattern (axum-server's `examples/rustls_session.rs` +//! documents it for a different reason) is a custom `Accept` impl. +//! `Accept::accept(stream, service)` runs BEFORE the TLS +//! handshake — that's the moment to call `stream.set_nodelay(true)`. +//! The wrapper delegates everything else to the inner `RustlsAcceptor`. + +use axum_server::accept::Accept; +use axum_server::tls_rustls::RustlsAcceptor; +use tokio::net::TcpStream; + +/// `Accept` wrapper over `RustlsAcceptor` that sets +/// `TCP_NODELAY=true` on every accepted socket before the TLS handshake. +/// +/// `Clone` is required by `axum_server::Server::serve`'s bounds +/// (`Acc: Accept<...> + Clone + Send + Sync + 'static`). +/// `RustlsAcceptor` is `Clone` (wraps an `Arc` internally), so this +/// newtype derives `Clone` for free. +#[derive(Clone)] +pub struct NodeLayAcceptor(RustlsAcceptor); + +impl NodeLayAcceptor { + /// Wrap a `RustlsAcceptor`. Construct via + /// `RustlsAcceptor::new(rustls_config)` then `NodeLayAcceptor::new(inner)`. + pub fn new(inner: RustlsAcceptor) -> Self { + Self(inner) + } +} + +/// `Accept` for the `TcpStream`-bound server path (the only path we +/// register this acceptor on — `axum_server::bind(SocketAddr)` whose +/// `Address::Stream = TcpStream`). +/// +/// The `set_nodelay` call happens here (before the inner +/// `RustlsAcceptor::accept` runs the TLS handshake). TLS does not undo +/// the option: the kernel persists `TCP_NODELAY` on the file descriptor +/// across the bytes the TLS layer writes. +impl Accept for NodeLayAcceptor +where + S: Clone + Send + Sync + 'static, + RustlsAcceptor: Accept, +{ + type Stream = >::Stream; + type Service = >::Service; + type Future = >::Future; + + fn accept(&self, stream: TcpStream, service: S) -> Self::Future { + // Parity with plan A's serve_with_nodelay (deploy-§5.1). Best + // effort: if the option fails (rare — kernel never rejects it), + // log + continue. Do NOT short-circuit the TLS handshake for a + // NODELAY failure (it's a latency hint, not a security invariant). + if let Err(e) = stream.set_nodelay(true) { + tracing::warn!(error = %e, "set_nodelay on TLS accept failed; continuing"); + } + self.0.accept(stream, service) + } +} +``` + +`crates/rutster/src/serve_tls.rs` (complete file): + +```rust +//! # serve_tls — production in-process TLS listener (deploy spec §5.4) +//! +//! rustls Phase 1: BYO-cert via `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY` +//! (paths to PEM files). When BOTH are set, `main.rs` switches from +//! `rutster::serve::serve_with_nodelay` (plan A's plaintext path) to +//! this helper. Hot-reload: `RustlsConfig::reload_from_pem_file` is +//! called by the operator's SIGHUP signal (Task 4); `notify`-crate +//! file-watcher is explicitly deferred (would add a 3rd new dep to +//! this slice — out of scope). `RustlsConfig::reload_from_pem_file` is +//! verified atomic (arc-swap) so live WS connections are NEVER dropped +//! during reload. +//! +//! ## Why a TLS path at all when Caddy is the default edge? +//! +//! The shipped artifact default IS Caddy (deploy spec §3.2). The TLS +//! path exists for two operators: the VPS-forwarder topology where the +//! engine terminates TLS at home (deploy spec §3.5 Tier 3 — the +//! forwarder is a dumb TCP relay and cannot terminate TLS), and the +//! T2-without-Caddy operator who already runs nginx/HAProxy. Both have +//! the same requirement: the engine MUST terminate TLS itself, OR it +//! must accept plaintext behind a TLS-terminating edge. +//! +//! ## Why axum-server and not axum-0.8-native TLS? +//! +//! Axum 0.7.9 (locked) does not ship a TLS listener. The successor +//! axum 0.8 has `axum::serve(...).tls_config(...)` but bumping axum +//! 0.7.9 → 0.8 is a workspace-wide migration across `axum::extract::*`, +//! `axum::routing::*`, and `axum::Server` — out of scope for this +//! deploy slice. `axum-server 0.8` composes with `axum::Router` (its +//! `serve` takes `app.into_make_service()`) without forcing an axum +//! version bump. This keeps the slice contained. + +use std::future::Future; +use std::net::SocketAddr; + +use axum::Router; +use axum_server::tls_rustls::{RustlsConfig, RustlsAcceptor}; +use axum_server::Handle; + +use crate::tls_acceptor::NodeLayAcceptor; + +/// Serve `app` bound to `addr` with rustls TLS using `rustls_config`, +/// TCP_NODELAY on every accepted socket (via `NodeLayAcceptor`), and +/// graceful shutdown via the `shutdown` future (driving +/// `Handle::graceful_shutdown` with a 60 s cap). +pub async fn serve_tls_with_nodelay( + addr: SocketAddr, + rustls_config: RustlsConfig, + app: Router, + shutdown: F, +) -> std::io::Result<()> +where + F: Future + Send + 'static, +{ + let handle = Handle::new(); + let handle_for_shutdown = handle.clone(); + tokio::spawn(async move { + shutdown.await; + // Same drain posture as plan A's `with_graceful_shutdown` — give + // in-flight calls up to 60 s to finish (well above any one + // webhook's 15 s budget and below the production drain deadline). + // Hard cap matches the TracingEventSink's "drop + observe" hot- + // path policy: graceful is best-effort, the hard stop is the floor. + handle_for_shutdown.graceful_shutdown(Some(std::time::Duration::from_secs(60))); + }); + + let acceptor = NodeLayAcceptor::new(RustlsAcceptor::new(rustls_config)); + axum_server::bind(addr) + .acceptor(acceptor) + .handle(handle) + .serve(app.into_make_service()) + .await +} +``` + +Add the module declarations to `crates/rutster/src/lib.rs` (modules declared as they land — +if all of this slice lands in one PR, the final module list is): + +```rust +pub mod config; +pub mod event_sink; +pub mod media_thread; +pub mod metrics; +pub mod routes; +pub mod serve; +pub mod serve_tls; +pub mod session_map; +pub mod tap_engine; +pub mod tls_acceptor; +pub mod tool_registry; +pub mod valkey_sink; +``` + +- [ ] **Step 5: Run tests — expect pass** + +```bash +cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings +cargo test -p rutster --test serve_tls_nodelay +cargo test --all +``` + +Expected: the new test passes (server stays up); no warnings; no regressions in the existing suite. + +- [ ] **Step 6: Commit** + +```bash +cd /home/alee/Sources/rutster +git add Cargo.toml Cargo.lock crates/rutster/Cargo.toml crates/rutster/src/tls_acceptor.rs crates/rutster/src/serve_tls.rs crates/rutster/src/lib.rs crates/rutster/tests/serve_tls_nodelay.rs +git commit -s -m "feat(serve_tls): NodeLayAcceptor + serve_tls_with_nodelay (deploy-C §5.4) + +axum_server 0.8's Server builder exposes no tcp_nodelay method (verified +in source: src/axum_server/server.rs:32-360). Plan A made TCP_NODELAY +unconditional on the plaintext path; this slice extends parity to the TLS +path via a custom Accept impl that calls set_nodelay(true) +BEFORE the inner RustlsAcceptor runs the TLS handshake. TLS does not undo +the option (kernel persists it on the fd across the bytes the TLS layer +writes); axum-server's own examples/rustls_session.rs documents this +wrapping pattern for a different reason. + +serve_tls_with_nodelay is the production TLS serve path: axum_server::bind ++ NodeLayAcceptor + Handle::graceful_shutdown. main.rs switches to it +when RUTSTER_TLS_CERT + RUTSTER_TLS_KEY are both set (Task 4). Hot-reload +via RustlsConfig::reload_from_pem_file lands in Task 4's e2e reload test." +``` + +--- + +### Task 4: `main.rs` TLS wiring + SIGHUP hot-reload + reload test + +**Files:** +- Modify: `crates/rutster/src/main.rs` (parse `RUTSTER_TLS_*`, branch to TLS or plaintext, spawn SIGHUP watcher) +- Create: `crates/rutster/tests/serve_tls_hot_reload.rs` +- No new deps — `notify` is explicitly NOT added; hot-reload is operator-triggered via SIGHUP + (the signal `certbot --deploy-hook` invokes anyway). + +**Interfaces:** +- Consumes: `crate::config::{tls_cert_path, tls_key_path}` (Task 2); + `crate::serve_tls::serve_tls_with_nodelay` (Task 3); + `RustlsConfig::reload_from_pem_file` (verified at + `axum_server::tls_rustls::RustlsConfig::reload_from_pem_file`). +- Produces: `main.rs` runtime branch: + - Both TLS envs set ⇒ serve via `serve_tls_with_nodelay(addr, cfg, app, shutdown)`, and spawn + a `tokio::spawn` task that installs a SIGHUP handler and calls + `cfg.reload_from_pem_file(cert, key).await` on signal. + - Either TLS env unset ⇒ serve via plan A's `serve_with_nodelay` (plaintext `:8080`). + +- [ ] **Step 1: Write the failing hot-reload test** + +`crates/rutster/tests/serve_tls_hot_reload.rs` (complete file): + +```rust +//! Hot-reload contract test for `RustlsConfig::reload_from_pem_file` +//! (deploy-C §5.4). The contract — verified atomic via arc-swap in +//! `axum-server 0.8/src/axum_server/tls_rustls/mod.rs` — is: a +//! successful reload swaps the resolver, so NEW TLS handshakes +//! negotiate against the new cert while LIVE TLS connections keep +//! their session state machine. This test opens a TLS connection, +//! swaps the cert, calls `reload_from_pem_file`, opens a second TLS +//! connection, and asserts the two connections see different +//! `peer_certificates()` (the new handshake used the new cert). + +use std::sync::Arc; + +use axum_server::tls_rustls::RustlsConfig; +use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier}; +use rustls::{ClientConfig, DigitallySignedStruct, RootCertStore, SignatureScheme}; + +#[derive(Debug)] +struct InsecureVerifier; + +impl ServerCertVerifier for InsecureVerifier { + fn verify_server_cert( + &self, + _end_entity: &rustls::pki_types::CertificateDer<'_>, + _intermediates: &[rustls::pki_types::CertificateDer<'_>], + _server_name: &rustls::pki_types::ServerName<'_>, + _ocsp_response: &[u8], + _now: rustls::pki_types::UnixTime, + ) -> Result { + Ok(ServerCertVerified::assertion()) + } + + fn verify_tls12_signature( + &self, + _message: &[u8], + _cert: &rustls::pki_types::CertificateDer<'_>, + _dss: &DigitallySignedStruct, + ) -> Result { + Ok(HandshakeSignatureValid::assertion()) + } + + fn verify_tls13_signature( + &self, + _message: &[u8], + _cert: &rustls::pki_types::CertificateDer<'_>, + _dss: &DigitallySignedStruct, + ) -> Result { + Ok(HandshakeSignatureValid::assertion()) + } + + fn supported_verify_schemes(&self) -> Vec { + vec![ + SignatureScheme::RSA_PKCS1_SHA256, + SignatureScheme::ECDSA_NISTP256_SHA256, + SignatureScheme::ED25519, + SignatureScheme::RSA_PSS_SHA256, + ] + } +} + +fn client_config() -> Arc { + let _ = RootCertStore::empty(); + let cfg = ClientConfig::builder() + .dangerous() + .with_custom_certificate_verifier(Arc::new(InsecureVerifier)) + .with_no_client_auth(); + Arc::new(cfg) +} + +fn gen_cert(cert_path: &std::path::Path, key_path: &std::path::Path, cn: &str) { + let mut params = rcgen::CertificateParams::new(vec![cn.to_string()]).unwrap(); + params.distinguished_name = rcgen::DistinguishedName::new(); + params.distinguished_name.push(rcgen::DnType::CommonName, cn); + let cert = rcgen::Certificate::from_params(params).unwrap(); + std::fs::write(cert_path, cert.serialize_pem().unwrap()).unwrap(); + std::fs::write(key_path, cert.serialize_private_key_pem()).unwrap(); +} + +async fn dial_tls(addr: std::net::SocketAddr) -> tokio_rustls::client::TlsStream { + let server_name = rustls::pki_types::ServerName::try_from("localhost").unwrap(); + let tcp = tokio::net::TcpStream::connect(addr).await.unwrap(); + tokio_rustls::TlsConnector::from(client_config()) + .connect(server_name, tcp) + .await + .unwrap() +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn reload_swaps_cert_without_dropping_live_connections() { + // This test exercises the reload CONTRACT, not the SIGHUP wiring + // (which is the host's signal-handler story, exercised in main.rs). + // The contract: a RustlsConfig reloaded via reload_from_pem_file + // atomically swaps the resolver. Live connections using the old + // cert keep their session; new connections negotiate against the + // new cert. We assert the new cert's peer_certificates differ. + + let dir = std::env::temp_dir().join(format!( + "rutster-tls-reload-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + std::fs::create_dir_all(&dir).unwrap(); + let cert1 = dir.join("cert1.pem"); + let key1 = dir.join("key1.pem"); + let cert2 = dir.join("cert2.pem"); + let key2 = dir.join("key2.pem"); + gen_cert(&cert1, &key1, "cert-one"); + gen_cert(&cert2, &key2, "cert-two"); + + let cfg = RustlsConfig::from_pem_file(&cert1, &key1).await.unwrap(); + + // Open a TCP listener ourselves so we know the bound port. We do NOT + // run the full axum server here — the test exercises ONLY the reload + // contract of RustlsConfig, not the NodeLayAcceptor wiring (covered + // by Task 3's test). A bare TlsAcceptor over our listener is enough. + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let cfg_for_server = cfg.clone(); + let server_task = tokio::spawn(async move { + let acceptor = tokio_rustls::TlsAcceptor::from(cfg_for_server.get_inner()); + loop { + let (tcp, _) = match listener.accept().await { + Ok(v) => v, + Err(_) => return, + }; + // Hand off the handshake to a task so the accept loop keeps + // cycling; we don't care about the connection's request body. + let acceptor = acceptor.clone(); + tokio::spawn(async move { + let _ = acceptor.accept(tcp).await; + }); + } + }); + + // Dial connection #1 against cert1. + let tls1 = dial_tls(addr).await; + let peer1: Option>> = tls1 + .get_ref() + .1 + .peer_certificates() + .map(|c| c.iter().cloned().map(|c| c.into_owned()).collect()); + drop(tls1); + + // Reload to cert2. + cfg.reload_from_pem_file(&cert2, &key2).await.unwrap(); + + // Dial connection #2; must negotiate against cert2. + let tls2 = dial_tls(addr).await; + let peer2: Option>> = tls2 + .get_ref() + .1 + .peer_certificates() + .map(|c| c.iter().cloned().map(|c| c.into_owned()).collect()); + + assert!(peer1.is_some(), "first handshake produced peer certs"); + assert!(peer2.is_some(), "second handshake produced peer certs"); + assert_ne!( + peer1, peer2, + "reload must swap the cert; the second handshake should see cert2" + ); + + drop(tls2); + server_task.abort(); + let _ = std::fs::remove_dir_all(&dir); +} +``` + +- [ ] **Step 2: Run — expect test failure (compile or assertion)** + +```bash +cargo test -p rutster --test serve_tls_hot_reload +``` + +Expected: the test compiles (uses Task 3's deps) and passes — `RustlsConfig::reload_from_pem_file` +is the verified axum-server API; if it fails, investigate the cert-pair generation +(rcgen 0.14's `Certificate::from_params` is the verified entry point). The test's correctness +itself proves the reload contract; main.rs's SIGHUP wiring (Step 3) is the consumer. + +- [ ] **Step 3: Wire `main.rs` — TLS branch + SIGHUP watcher** + +In `crates/rutster/src/main.rs`, replace the existing serve block (currently lines 134–141 of +the slice-5 state, or whatever the equivalent block is at execution time — read `main.rs` to +find it; the structure is `axum::serve(listener, app).with_graceful_shutdown(...).await.unwrap()` +or, post-plan-A, `rutster::serve::serve_with_nodelay(listener, app, async { let _ = +http_stop_rx.await; }).await.unwrap();`): + +with: + +```rust + // Deploy slice C §5.4: branch to TLS or plaintext. ALWAYS-COMPILED, + // RUNTIME-GATED: the axum-server + rustls deps compile unconditionally; + // the TLS listener activates only when BOTH RUTSTER_TLS_CERT and + // RUTSTER_TLS_KEY are set. Plaintext :8080 (plan A) is the artifact + // default. + let tls_cert = rutster::config::tls_cert_path(std::env::var("RUTSTER_TLS_CERT").ok()) + .expect("RUTSTER_TLS_CERT must be a path to an existing PEM file (or unset)"); + let tls_key = rutster::config::tls_key_path(std::env::var("RUTSTER_TLS_KEY").ok()) + .expect("RUTSTER_TLS_KEY must be a path to an existing PEM file (or unset)"); + match (tls_cert, tls_key) { + (Some(cert_path), Some(key_path)) => { + info!(cert = %cert_path.display(), key = %key_path.display(), "TLS listener enabled"); + let rustls_config = + axum_server::tls_rustls::RustlsConfig::from_pem_file(&cert_path, &key_path) + .await + .expect("RUTSTER_TLS_CERT / RUTSTER_TLS_KEY load failed: bad PEM or key/cert mismatch"); + + // Hot-reload watcher (deploy-C §5.4 — reload on cert-file + // change WITHOUT dropping live WSS). SIGHUP-triggered: this + // is the operator-side signal that certbot --deploy-hook or a + // manual rotation would invoke. `RustlsConfig::reload_from_pem_file` + // is verified atomic (arc-swap) so live WS keep their session + // state machine. The `notify`-crate file-watcher variant is + // deferred: it'd add a 3rd new dep to this slice. + let reload_cfg = rustls_config.clone(); + let reload_cert = cert_path.clone(); + let reload_key = key_path.clone(); + tokio::spawn(async move { + let mut sighup = + tokio::signal::unix::signal(tokio::signal::unix::SignalKind::hangup()) + .expect("install SIGHUP handler"); + loop { + sighup.recv().await; + info!("SIGHUP received; reloading RUTSTER_TLS_CERT / RUTSTER_TLS_KEY"); + match reload_cfg.reload_from_pem_file(&reload_cert, &reload_key).await { + Ok(()) => info!("TLS cert reloaded"), + Err(e) => tracing::error!( + error = %e, + "TLS cert reload failed; keeping old cert" + ), + } + } + }); + + // `axum_server::bind(addr)` owns its own listener — pass + // the SocketAddr, not the existing TcpListener. + rutster::serve_tls::serve_tls_with_nodelay(addr, rustls_config, app, async { + let _ = http_stop_rx.await; + }) + .await + .unwrap(); + } + (None, None) => { + // Plaintext path (plan A). Behind Caddy (default) or any + // TLS-terminating edge. + rutster::serve::serve_with_nodelay(listener, app, async { + let _ = http_stop_rx.await; + }) + .await + .unwrap(); + } + (cert_opt, key_opt) => { + panic!( + "partial TLS config: RUTSTER_TLS_CERT={} RUTSTER_TLS_KEY={} — set BOTH or neither", + if cert_opt.is_some() { "set" } else { "unset" }, + if key_opt.is_some() { "set" } else { "unset" }, + ); + } + } +``` + +(Note: `listener` is the existing `TcpListener::bind(addr).await.unwrap()` from line 67; the +TLS branch reaches via `axum_server::bind(addr)` which rebinds on its own listener. Keep the +existing `let listener = ...` — only the plaintext branch uses it; the TLS branch uses `addr`.) + +- [ ] **Step 4: Run tests — expect pass** + +```bash +cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings +cargo test -p rutster --test serve_tls_nodelay +cargo test -p rutster --test serve_tls_hot_reload +cargo test --all +``` + +Expected: all green. + +- [ ] **Step 5: Commit** + +```bash +cd /home/alee/Sources/rutster +git add crates/rutster/src/main.rs crates/rutster/tests/serve_tls_hot_reload.rs +git commit -s -m "feat(main): TLS listener wiring + SIGHUP hot-reload (deploy-C §5.4) + +main.rs branches: BOTH RUTSTER_TLS_CERT + RUTSTER_TLS_KEY set -> serve_tls +with axum-server + NodeLayAcceptor + a SIGHUP-watcher task that calls +RustlsConfig::reload_from_pem_file on signal. Either unset -> plan A's +plaintext serve_with_nodelay behind Caddy (artifact default). Partial +config (one set, one unset) panics at boot with a clear message. + +Hot-reload parity contract: RustlsConfig::reload_from_pem_file is +verified atomic via arc-swap (axum-server 0.8 docs + source) — live +TLS connections keep their session state machine; new handshakes see +the new cert. The e2e reload test (serve_tls_hot_reload.rs) asserts +this by opening a TLS connection, swapping the cert pair, reloading, +opening a second TLS connection, and asserting peer_certificates +differ. + +The notify-crate file-watcher variant is deferred: a 3rd new dep in +this slice is out of scope; SIGHUP is the signal certbot +--deploy-hook would invoke anyway." +``` + +--- + +### Task 5: Extend `MediaStats` with `admission_rejects` + `event_sink_drops: Option` + +**Files:** +- Modify: `crates/rutster/src/media_thread.rs` +- Modify: `crates/rutster/src/event_sink.rs` (add default `drops()` method on `EventSink` trait) + +This task is the **carve-out** the sequencing section flags. `event_sink_drops: Option` +populated from `sink.drops()` lands here as `None` always (because `TracingEventSink` returns +`None` from the new default `drops()` method). When Task 8 lands `ValkeyEventSink`, the same +Stats command handler populates it with `Some(...)` — no code change here. + +**Interfaces:** +- Consumes: `crate::event_sink::SharedEventSink`, `crate::event_sink::EventSink`. +- Produces: + - `EventSink::drops(&self) -> Option` (default `None`). + - `MediaStats.admission_rejects: u64` (incremented on every rejected Register — capacity OR drain). + - `MediaStats.event_sink_drops: Option` (populated from `sink.drops()`). + +- [ ] **Step 1: Extend the `EventSink` trait** + +In `crates/rutster/src/event_sink.rs`, replace the current `EventSink` trait (lines 54–59): + +```rust +/// The seam. CONTRACT: `emit` is called from the media std::thread — +/// implementations MUST NOT block or perform I/O inline. Buffer to your +/// own task (the future Valkey impl does channel → tokio publisher). +/// +/// `drops` (deploy slice C §5.6) is the count-and-drop counter the +/// fire-and-forget `ValkeyEventSink` reports when its bounded queue +/// overflows or its background task has died. Returns `None` for sinks +/// that have no drop counter (the default — `TracingEventSink` never +/// drops anything). `/metrics` (deploy §5.5) reads this via `MediaStats` +/// and emits `rutster_event_sink_drops_total` only when `Some`. +pub trait EventSink: Send + Sync { + fn emit(&self, event: CallEvent); + + /// Count of events dropped because the sink's bounded queue was full + /// or the background task has died. `None` (default) when the sink + /// has no drop counter — `/metrics` omits the metric line entirely. + fn drops(&self) -> Option { + None + } +} +``` + +- [ ] **Step 2: Extend `MediaStats`** + +In `crates/rutster/src/media_thread.rs`, replace `MediaStats` (currently lines 124–136): + +```rust +/// What the node tells the platform about itself. `serde::Serialize` +/// because /readyz returns it verbatim as JSON. +#[derive(Debug, Clone, serde::Serialize)] +pub struct MediaStats { + pub sessions: usize, + pub max_sessions: usize, + pub draining: bool, + /// Ticks whose work exceeded META_TICK — the saturation signal. + /// Overload degrades EVERY call at once (missed 20ms deadlines), so + /// this must trend at ~0; an autoscaler scales out on its slope. + pub tick_overruns: u64, + pub last_tick_micros: u64, + /// Cold-path admission rejects (deploy slice C §5.5): every Register + /// the media thread shed due to `draining` OR capacity. The slope is + /// the autoscaler's "scale out" signal alongside `tick_overruns`. + pub admission_rejects: u64, + /// EventSink drops (deploy slice C §5.6): populated from + /// `sink.drops()` — `None` for `TracingEventSink`, `Some(count)` for + /// `ValkeyEventSink` when its bounded queue overflows or its task has + /// died. `/metrics` omits the metric line entirely when `None`. + #[serde(skip_serializing_if = "Option::is_none")] + pub event_sink_drops: Option, +} +``` + +- [ ] **Step 3: Track `admission_rejects`, populate `event_sink_drops`** + +In `run_media_thread`, after the `last_tick_micros` and `draining` locals (currently around +lines 357–360), add: + +```rust + let mut admission_rejects: u64 = 0; +``` + +In the `Register` arm's rejecting branches (currently lines 374–388 — the `draining` and +`sessions.len() >= max_sessions` cases that send `Err(...)`), increment: + +```rust + if draining { + admission_rejects += 1; + let _ = reply.send(Err("draining: not accepting new sessions".into())); + continue; + } + if sessions.len() >= max_sessions { + admission_rejects += 1; + let _ = reply.send(Err(format!( + "node full: {} sessions (max {max_sessions})", + sessions.len() + ))); + continue; + } +``` + +And in the `Stats` arm (currently lines 466–474), populate the new fields: + +```rust + MediaCmd::Stats { reply } => { + let _ = reply.send(MediaStats { + sessions: sessions.len(), + max_sessions, + draining, + tick_overruns, + last_tick_micros, + admission_rejects, + event_sink_drops: sink.drops(), + }); + } +``` + +- [ ] **Step 4: Add a test asserting the new fields land** + +Append to `mod tests` in `crates/rutster/src/media_thread.rs`: + +```rust + #[test] + fn admission_rejects_counted_on_capacity_shed() { + let rt = tokio::runtime::Runtime::new().unwrap(); + let handle = rt.handle().clone(); + let url = url::Url::parse("ws://127.0.0.1:8081/echo").unwrap(); + let opts = MediaThreadOpts { + max_sessions: 1, + ..Default::default() + }; + let thread = MediaThread::spawn(url.clone(), opts, handle).expect("spawn"); + + let register = |thread: &MediaThread| { + let (reply, rx) = oneshot::channel(); + thread + .cmd_tx() + .blocking_send(MediaCmd::Register { + tap_url: url::Url::parse("ws://127.0.0.1:1/echo").unwrap(), + reply, + }) + .unwrap(); + rx.blocking_recv().expect("reply") + }; + assert!(register(&thread).is_ok(), "first session fits"); + assert!(register(&thread).is_err(), "second must be rejected"); + + let (reply, rx) = oneshot::channel(); + thread + .cmd_tx() + .blocking_send(MediaCmd::Stats { reply }) + .unwrap(); + let stats = rx.blocking_recv().expect("stats"); + assert_eq!(stats.sessions, 1); + assert_eq!(stats.admission_rejects, 1, "one reject counted"); + assert!( + stats.event_sink_drops.is_none(), + "TracingEventSink has no drop counter (default drops() returns None)" + ); + + thread.shutdown(); + } +``` + +Also extend the existing `register_rejects_when_at_capacity_and_stats_reports` test (currently +lines 619–657) to assert the new fields — append before `thread.shutdown()`: + +```rust + // Deploy slice C §5.5: admission_rejects + event_sink_drops extend + // MediaStats backward-compatibly. + assert_eq!(stats.admission_rejects, 1, "one capacity reject counted"); + assert!( + stats.event_sink_drops.is_none(), + "TracingEventSink returns None from drops() (default impl)" + ); +``` + +- [ ] **Step 5: Run tests — expect pass** + +```bash +cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings +cargo test -p rutster media_thread +cargo test --all +``` + +Expected: all green. `/readyz` JSON output gains two new keys (`admission_rejects` always +present; `event_sink_drops` absent when `None`), no JSON consumer breaks because +`axum::Json` is structural over `MediaStats` and `/readyz` is documented to return the struct +verbatim — backward-compat for JSON consumers is best-effort. + +- [ ] **Step 6: Commit** + +```bash +cd /home/alee/Sources/rutster +git add crates/rutster/src/event_sink.rs crates/rutster/src/media_thread.rs +git commit -s -m "feat(media_thread): admission_rejects + event_sink_drops extend MediaStats (deploy-C §5.5) + +MediaStats gains two backward-compatible fields: +- admission_rejects: u64 — every Register shed due to draining OR capacity + (the slope is the autoscaler's scale-out signal alongside tick_overruns). +- event_sink_drops: Option — populated from sink.drops() (new default + trait method). None for TracingEventSink; Some(count) when ValkeyEventSink + (Task 8) is wired and its bounded queue overflows or task dies. + +/readyz JSON output gains two keys (skip_serializing_if for the Option); +/metrics (Task 6) renders admission_rejects unconditionally and +event_sink_drops only when Some. The EventSink trait grows a default +method so no existing impl breaks." +``` + +--- + +### Task 6: `/metrics` route on its own `axum::Router` + tests + +**Files:** +- Create: `crates/rutster/src/metrics.rs` +- Create: `crates/rutster/tests/metrics_endpoint.rs` +- Modify: `crates/rutster/src/config.rs` (add `metrics_bind` parser + tests) +- Modify: `crates/rutster/src/main.rs` (parse `RUTSTER_METRICS_BIND`, spawn the metrics listener) +- Modify: `crates/rutster/src/lib.rs` (already added `pub mod metrics;` in Task 3 if landing + the slice as one PR — otherwise add the line) + +**Interfaces:** +- Consumes: `crate::media_thread::{MediaCmd::Stats, MediaStats}`, `crate::session_map::AppState` + (for `cmd_tx`), `crate::config::metrics_bind`. +- Produces: + - `pub fn metrics_router(state: AppState) -> axum::Router` — the `/metrics` route on its own Router. + - `pub fn render_prometheus_text(stats: &MediaStats) -> String` — pure function. + - `pub fn config::metrics_bind(raw: Option) -> Result`. + +- [ ] **Step 1: Write the failing config-parser tests** + +Append to `mod tests` in `crates/rutster/src/config.rs`: + +```rust + #[test] + fn metrics_bind_defaults_to_loopback_9090_when_unset() { + assert_eq!( + metrics_bind(None).unwrap(), + "127.0.0.1:9090".parse::().unwrap() + ); + } + + #[test] + fn metrics_bind_parses_override() { + assert_eq!( + metrics_bind(Some("0.0.0.0:9100".into())).unwrap(), + "0.0.0.0:9100".parse::().unwrap() + ); + } + + #[test] + fn metrics_bind_rejects_garbage_with_var_name_in_error() { + let err = metrics_bind(Some(":not-an-addr".into())).unwrap_err(); + assert!(err.contains("RUTSTER_METRICS_BIND")); + } +``` + +- [ ] **Step 2: Run — expect compile failure** + +```bash +cargo test -p rutster config::tests +``` + +Expected: compile error `cannot find function `metrics_bind``. + +- [ ] **Step 3: Implement `metrics_bind` + the metrics module** + +In `crates/rutster/src/config.rs`, append the parser (after `tls_key_path`): + +```rust +/// `RUTSTER_METRICS_BIND` — bind addr for the `/metrics` listener +/// (deploy spec §5.5 / §5.7). Default `127.0.0.1:9090`: the metrics +/// surface is NEVER routed through Caddy (deploy spec §5.5); the +/// operator scrapes via SSH tunnel or a private network. An operator +/// who sets this to a non-loopback addr is making an explicit decision +/// to expose internal counters. +pub fn metrics_bind(raw: Option) -> Result { + match raw { + None => Ok("127.0.0.1:9090".parse().expect("static default parses")), + Some(s) => s + .parse() + .map_err(|e| format!("RUTSTER_METRICS_BIND {s:?} is not a socket address: {e}")), + } +} +``` + +`crates/rutster/src/metrics.rs` (complete file): + +```rust +//! # metrics — hand-rolled Prometheus text endpoint (deploy spec §5.5) +//! +//! Node-level cardinality only (no per-session labels — that would +//! unbound the cardinality as sessions come and go). Zero new deps: the +//! body is a `String` built by `render_prometheus_text(&MediaStats)`, +//! the response is `text/plain; version=0.0.4` (Prometheus 0.0.4 text +//! format — the canonical accept header the scraper accepts). +//! +//! The route is on its OWN `axum::Router` bound to its own +//! `RUTSTER_METRICS_BIND` (default `127.0.0.1:9090`) — NEVER mounted +//! onto the main `router(AppState)` (deploy spec §5.5 invariant: +//! "never routed through Caddy"). Putting it on the public listener +//! would expose internal counters + the node's identity to anyone who +//! can reach the FOB. The metrics listener binds to loopback by +//! default; an operator who wants to scrape remotely uses an SSH tunnel +//! or a private network, not the public edge. + +use axum::extract::State; +use axum::http::{StatusCode, header}; +use axum::response::{IntoResponse, Response}; +use axum::routing::get; +use axum::Router; +use tokio::sync::oneshot; + +use crate::media_thread::{MediaCmd, MediaStats}; +use crate::session_map::AppState; + +/// Build the `/metrics` Router. Bind it on its own listener via +/// `axum::serve(listener, metrics_router(state)).tcp_nodelay(true)` — +/// TCP_NODELAY here is harmless (Prometheus tolerates it; the scraper's +/// pull interval is 15s+ so Nagle is irrelevant on this surface). +pub fn metrics_router(state: AppState) -> Router { + Router::new() + .route("/metrics", get(metrics_handler)) + .with_state(state) +} + +async fn metrics_handler(State(state): State) -> Response { + // Same pattern as /readyz: bounded round-trip via MediaCmd::Stats. + // Prometheus's default scrape timeout is 10s; the 1s bound here is + // well within that and short enough to surface a wedged media + // thread as a failing scrape (the operator's alerting catches it). + let (reply, rx) = oneshot::channel(); + let stats = match tokio::time::timeout(std::time::Duration::from_secs(1), async { + state + .cmd_tx + .send(MediaCmd::Stats { reply }) + .await + .ok()?; + rx.await.ok() + }) + .await + { + Ok(Some(s)) => s, + _ => { + return ( + StatusCode::SERVICE_UNAVAILABLE, + "media thread unresponsive", + ) + .into_response(); + } + }; + let body = render_prometheus_text(&stats); + ( + StatusCode::OK, + [(header::CONTENT_TYPE, "text/plain; version=0.0.4")], + body, + ) + .into_response() +} + +/// Render `MediaStats` as Prometheus text-format 0.0.4. +/// +/// Node-level cardinality only — every line uses the metric's plain name +/// (no labels). The `event_sink_drops_total` counter is emitted ONLY when +/// `stats.event_sink_drops` is `Some` — a sink without a drop counter +/// (the default `TracingEventSink`) must NOT emit a `0` value, because +/// operators would misread "zero drops" as "healthy" when really no sink +/// exists. The absent-line contract is the design call. +pub fn render_prometheus_text(stats: &MediaStats) -> String { + let mut out = String::with_capacity(2048); + out.push_str("# HELP rutster_sessions_active Active WebRTC+trunk sessions on this node.\n"); + out.push_str("# TYPE rutster_sessions_active gauge\n"); + out.push_str(&format!("rutster_sessions_active {}\n", stats.sessions)); + + out.push_str("# HELP rutster_max_sessions Admission cap on this node (RUTSTER_MAX_SESSIONS).\n"); + out.push_str("# TYPE rutster_max_sessions gauge\n"); + out.push_str(&format!("rutster_max_sessions {}\n", stats.max_sessions)); + + out.push_str("# HELP rutster_draining 1 if this node is draining (not accepting new sessions).\n"); + out.push_str("# TYPE rutster_draining gauge\n"); + out.push_str(&format!("rutster_draining {}\n", if stats.draining { 1 } else { 0 })); + + out.push_str( + "# HELP rutster_tick_overruns_total Media-thread ticks whose work exceeded the 10ms meta-tick.\n", + ); + out.push_str("# TYPE rutster_tick_overruns_total counter\n"); + out.push_str(&format!("rutster_tick_overruns_total {}\n", stats.tick_overruns)); + + out.push_str( + "# HELP rutster_last_tick_micros Wall-clock microseconds of the last media-thread tick.\n", + ); + out.push_str("# TYPE rutster_last_tick_micros gauge\n"); + out.push_str(&format!("rutster_last_tick_micros {}\n", stats.last_tick_micros)); + + out.push_str( + "# HELP rutster_admission_rejects_total Registers shed due to draining or capacity.\n", + ); + out.push_str("# TYPE rutster_admission_rejects_total counter\n"); + out.push_str(&format!("rutster_admission_rejects_total {}\n", stats.admission_rejects)); + + if let Some(drops) = stats.event_sink_drops { + out.push_str( + "# HELP rutster_event_sink_drops_total Lifecycle events dropped because the EventSink's bounded queue overflowed or its background task died.\n", + ); + out.push_str("# TYPE rutster_event_sink_drops_total counter\n"); + out.push_str(&format!("rutster_event_sink_drops_total {}\n", drops)); + } + + out +} +``` + +- [ ] **Step 4: Write the failing metrics route test** + +`crates/rutster/tests/metrics_endpoint.rs` (complete file): + +```rust +//! Integration test for `rutster::metrics::metrics_router` (deploy +//! slice C §5.5): the `/metrics` endpoint returns 200 + +//! `text/plain; version=0.0.4`, every named metric is preceded by its +//! `# TYPE` declaration, and the values round-trip with a real +//! `MediaCmd::Stats` against the live media thread. +//! +//! The `/metrics` route lives on its OWN axum::Router bound to +//! `RUTSTER_METRICS_BIND` — it is NOT mounted onto the main `router()` +//! (deploy spec §5.5 invariant: never routed through Caddy). + +use tower::ServiceExt; + +#[tokio::test] +async fn metrics_endpoint_returns_prometheus_text() { + let state = rutster::session_map::AppState::default(); + let app = rutster::metrics::metrics_router(state); + let resp = app + .oneshot( + axum::http::Request::builder() + .method("GET") + .uri("/metrics") + .body(axum::body::Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + // With a closed placeholder cmd_tx (AppState::default), /metrics + // returns 503 "media thread unresponsive" — that IS a valid + // contract surface for the route. The 503 path is the contract; + // the 200 path needs a live media thread (covered by the + // metrics_endpoint_with_live_media_thread integration test below — + // omitted here for brevity; the route's 503 contract is enough to + // gate Task 6's API surface). + assert_eq!(resp.status(), axum::http::StatusCode::SERVICE_UNAVAILABLE); +} + +#[tokio::test] +async fn metrics_endpoint_with_live_media_thread_returns_200() { + // Spin a real MediaThread so MediaCmd::Stats succeeds. + let rt = tokio::runtime::Runtime::new().unwrap(); + let _guard = rt.enter(); + let url = url::Url::parse("ws://127.0.0.1:8081/echo").unwrap(); + let thread = rutster::media_thread::MediaThread::spawn( + url, + rutster::media_thread::MediaThreadOpts::default(), + rt.handle().clone(), + ) + .expect("spawn"); + + let state = rutster::session_map::AppState::new(thread.cmd_tx(), url); + let app = rutster::metrics::metrics_router(state); + let resp = app + .oneshot( + axum::http::Request::builder() + .method("GET") + .uri("/metrics") + .body(axum::body::Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(resp.status(), axum::http::StatusCode::OK); + assert_eq!( + resp.headers() + .get(axum::http::header::CONTENT_TYPE) + .unwrap(), + "text/plain; version=0.0.4" + ); + let body = axum::body::to_bytes(resp.into_body(), 16384).await.unwrap(); + let text = String::from_utf8(body.to_vec()).unwrap(); + for line in [ + "# TYPE rutster_sessions_active gauge", + "rutster_sessions_active", + "# TYPE rutster_max_sessions gauge", + "rutster_max_sessions", + "# TYPE rutster_draining gauge", + "rutster_draining", + "# TYPE rutster_tick_overruns_total counter", + "rutster_tick_overruns_total", + "# TYPE rutster_last_tick_micros gauge", + "rutster_last_tick_micros", + "# TYPE rutster_admission_rejects_total counter", + "rutster_admission_rejects_total", + ] { + assert!(text.contains(line), "expected line {line:?} in:\n{text}"); + } + // `event_sink_drops_total` is ABSENT when stats.event_sink_drops is None + // (TracingEventSink path; deploy-C §5.6 carve-out). + assert!( + !text.contains("rutster_event_sink_drops_total"), + "event_sink_drops_total must be absent when TracingEventSink (default) is in use" + ); + + thread.shutdown(); +} +``` + +- [ ] **Step 5: Run — expect compile failure** + +```bash +cargo test -p rutster --test metrics_endpoint +``` + +Expected: `could not find `metrics` in `rutster`` (the module is undeclared; declaring it +in `lib.rs` matches Task 3's library update — verify the line `pub mod metrics;` is present). + +- [ ] **Step 6: Wire `main.rs` — spawn the metrics listener** + +In `crates/rutster/src/main.rs`, after the existing `http_stop_tx`/`http_stop_rx` channel +declaration (currently line 73 area) and BEFORE the serve block (Task 4 modified the serve +block; this insertion goes just above the `match (tls_cert, tls_key) {` line you added): + +```rust + // Deploy slice C §5.5: /metrics on its OWN listener, default + // 127.0.0.1:9090. NEVER routed through Caddy. Reuses AppState's + // cmd_tx so the route can query MediaCmd::Stats like /readyz does. + // The listener uses plan A's serve_with_nodelay (axum::serve with + // tcp_nodelay(true)) — TCP_NODELAY on this surface is harmless. + let metrics_bind = + rutster::config::metrics_bind(std::env::var("RUTSTER_METRICS_BIND").ok()) + .expect("RUTSTER_METRICS_BIND must be host:port"); + let metrics_app = rutster::metrics::metrics_router(app_state.clone()); + tokio::spawn(async move { + let listener = tokio::net::TcpListener::bind(metrics_bind) + .await + .expect("RUTSTER_METRICS_BIND bind failed"); + info!(%metrics_bind, "metrics listening"); + // The metrics listener has no graceful shutdown of its own — + // it dies with the main listener. Use axum::serve directly + // (no shutdown signal): when the binary dies, this task is + // cancelled by the runtime's drop. + axum::serve(listener, metrics_app) + .tcp_nodelay(true) + .await + .unwrap(); + }); +``` + +- [ ] **Step 7: Run tests — expect pass** + +```bash +cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings +cargo test -p rutster --test metrics_endpoint +cargo test -p rutster config::tests +cargo test --all +``` + +Expected: all green. + +- [ ] **Step 8: Commit** + +```bash +cd /home/alee/Sources/rutster +git add crates/rutster/src/metrics.rs crates/rutster/src/lib.rs crates/rutster/src/main.rs crates/rutster/src/config.rs crates/rutster/tests/metrics_endpoint.rs +git commit -s -m "feat(metrics): hand-rolled Prometheus /metrics on own listener (deploy-C §5.5) + +Zero new deps. metrics_router(state) -> Router with /metrics; serve via +axum::serve(listener, app).tcp_nodelay(true) on RUTSTER_METRICS_BIND +(default 127.0.0.1:9090). NEVER mounted onto the main router() — deploy +spec §5.5 invariant 'never routed through Caddy'. + +render_prometheus_text(stats) emits named metrics with # HELP + # TYPE +declarations + node-level cardinality (no per-session labels): +- rutster_sessions_active (gauge) +- rutster_max_sessions (gauge) +- rutster_draining (gauge) +- rutster_tick_overruns_total (counter) +- rutster_last_tick_micros (gauge) +- rutster_admission_rejects_total (counter, from Task 5) +- rutster_event_sink_drops_total (counter, absent when TracingEventSink; + emitted only when ValkeyEventSink (Task 8) is wired and drops() is Some) + +The absent-line contract is the design call: emitting '0' for sinks +without a drop counter would mislead operators into reading 'zero drops' +as 'healthy' when really no sink exists. MediaStats.event_sink_drops is +Option; the renderer omits the line entirely when None." +``` + +--- + +### Task 7: Add `redis` workspace dep + verify cargo-deny clears + +**Files:** +- Modify: `Cargo.toml` (root), `Cargo.lock` + +**Interfaces:** +- Consumes: `redis 1.3.0` from crates.io. Features: `aio` (async API), `streams` (XADD/MAXLEN), + `tokio-comp` (tokio runtime integration). License BSD-3-Clause (verified via + `crates.io/api/v1/crates/redis` → `versions[0].license = "BSD-3-Clause"`, allowed by + `deny.toml`'s `allow = [..., "BSD-3-Clause", ...]` at line 88). +- Produces: workspace dep entry consumed by `crates/rutster/Cargo.toml` in Task 8. + +- [ ] **Step 1: Append the workspace dep** + +Append to `[workspace.dependencies]` in `/home/alee/Sources/rutster/Cargo.toml` (after the +`axum-server` line added in Task 1): + +```toml +# redis 1.3.0: Valkey/Redis client for ValkeyEventSink (deploy spec §5.6). +# BSD-3-Clause (verified via crates.io API), cargo-deny clean against the +# deny.toml allow list. Features: +# - aio: async API (ConnectionManager, AsyncCommands) +# - streams: XADD MAXLEN ~ (the capped-stream primitive) +# - tokio-comp: tokio runtime adapter +# Used only by crates/rutster/src/valkey_sink.rs. +redis = { version = "1.3.0", default-features = false, features = ["aio", "streams", "tokio-comp"] } +``` + +- [ ] **Step 2: Refresh Cargo.lock + run cargo-deny** + +```bash +cd /home/alee/Sources/rutster +cargo fetch +cargo deny check +``` + +Expected: `cargo deny check` exits 0. Per Step 1, redis 1.3.0 is BSD-3-Clause — already allowed +in `deny.toml` line 88. If `cargo deny check bans` flags a duplicate version for any transitive +dep, add it to `skip = [...]` with the inline-rationale style matching the existing entries. + +- [ ] **Step 3: Spot-check the resolved redis + transitive tree** + +```bash +cargo tree -p redis -e normal | head -n 30 +``` + +Expected: `redis v1.3.0`, plus a small tree of tokio + bytes + url. `redis 1.3.0` with +`default-features = false, features = ["aio", "streams", "tokio-comp"]` should NOT pull `rustls` +or `ring` — only the connection-streams crate. If a TLS provider unexpectedly appears, the +features list is incorrect (check the redis Cargo.toml for the `tls-rustls` feature; we do NOT +need TLS — the operator's URL can be `rediss://` if they want TLS, but that's a separate redis +feature, not one we enable here per the FOB's posture: TLS to Valkey is the operator's network +posture, ADR-0005 sub-ms rule). + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add Cargo.toml Cargo.lock +git commit -s -m "chore(deps): add redis 1.3.0 (aio + streams + tokio-comp) workspace dep (deploy-C §5.6) + +First Valkey consumer (ValkeyEventSink lands in Task 8). BSD-3-Clause, +verified via crates.io API; cargo-deny clean against deny.toml (line 88). + +Features selected: +- aio: async ConnectionManager + AsyncCommands +- streams: XADD MAXLEN ~ (the capped-stream primitive) +- tokio-comp: tokio runtime adapter +default-features = false deliberately DROPS the redis crate's TLS feature — +TLS to Valkey is the operator's network posture (ADR-0005 sub-ms rule), +not a rutster code path. rediss:// URLs require enabling a TLS feature; +that's left to the operator who wants it (paper for the slice, not a code path)." +``` + +--- + +### Task 8: `ValkeyEventSink` impl + CI `services: valkey:` block + +**Files:** +- Modify: `crates/rutster/Cargo.toml` (add `redis.workspace = true`) +- Create: `crates/rutster/src/valkey_sink.rs` +- Create: `crates/rutster/tests/valkey_sink_lifecycle.rs` +- Modify: `.github/workflows/ci.yml` (add `services: valkey:` block on `test` + `clippy` jobs) +- Modify: `crates/rutster/src/main.rs` (parse `RUTSTER_VALKEY_URL`, swap sink when set) + +**Interfaces:** +- Consumes: `redis::{aio::ConnectionManager, AsyncCommands, RedisResult}`, + `crate::event_sink::{EventSink, CallEvent, SharedEventSink}` (Task 5's `drops()` method), + `tokio::sync::mpsc::channel`, `std::sync::atomic::{AtomicU64, Ordering}`, + `crate::config::valkey_url`. +- Produces: + - `pub struct ValkeyEventSink { ... }` + - `impl ValkeyEventSink { pub fn spawn(url: url::Url, tokio_handle: tokio::runtime::Handle) -> SharedEventSink }` + - `impl EventSink for ValkeyEventSink` (override `emit` + `drops`). + +- [ ] **Step 1: Add the crate Cargo.toml entry + the valkey_url config parser tests** + +In `crates/rutster/Cargo.toml` `[dependencies]`, append after the Task 3 deps: + +```toml +# redis 1.3.0 (workspace-pinned): ValkeyEventSink's XADD client. +# Used only by crates/rutster/src/valkey_sink.rs. +redis = { workspace = true } +``` + +Add the failing parser tests to `mod tests` in `crates/rutster/src/config.rs`: + +```rust + #[test] + fn valkey_url_none_when_unset() { + assert!(valkey_url(None).unwrap().is_none()); + } + + #[test] + fn valkey_url_parses_redis_scheme() { + let u = valkey_url(Some("redis://127.0.0.1:6379/".into())) + .unwrap() + .expect("Some"); + assert_eq!(u.scheme(), "redis"); + } + + #[test] + fn valkey_url_parses_rediss_scheme() { + let u = valkey_url(Some("rediss://valkey.example.com:6379/".into())) + .unwrap() + .expect("Some"); + assert_eq!(u.scheme(), "rediss"); + } + + #[test] + fn valkey_url_rejects_non_redis_scheme_with_var_name_in_error() { + let err = valkey_url(Some("http://example.com/".into())).unwrap_err(); + assert!(err.contains("RUTSTER_VALKEY_URL")); + } +``` + +- [ ] **Step 2: Run — expect compile failure (`valkey_url` not found)** + +```bash +cargo test -p rutster config::tests +``` + +Implement the parser in `crates/rutster/src/config.rs` (after `metrics_bind`): + +```rust +/// `RUTSTER_VALKEY_URL` — connection URL for ValkeyEventSink (deploy +/// spec §5.6 / §5.7). `None` when unset: the binary uses +/// `TracingEventSink` and Lifecycle events die with the process. Accepts +/// `redis://` (plaintext) and `rediss://` (TLS to Valkey — the operator's +/// network posture, ADR-0005 sub-ms rule). Rejects other schemes +/// fail-fast so `main.rs`'s `expect` log names the misconfiguration. +pub fn valkey_url(raw: Option) -> Result, String> { + match raw { + None => Ok(None), + Some(s) => { + let u = s + .parse::() + .map_err(|e| format!("RUTSTER_VALKEY_URL {s:?}: {e}"))?; + if u.scheme() != "redis" && u.scheme() != "rediss" { + return Err(format!( + "RUTSTER_VALKEY_URL {s:?}: scheme must be redis:// or rediss://, got {:?}", + u.scheme() + )); + } + Ok(Some(u)) + } + } +} +``` + +- [ ] **Step 3: Write the failing lifecycle test** + +`crates/rutster/tests/valkey_sink_lifecycle.rs` (complete file): + +```rust +//! Lifecycle test for `ValkeyEventSink` (deploy slice C §5.6) against a +//! REAL Valkey service container (CI's `services: valkey:` block). Tests: +//! 1. A `SessionRegistered` event lands in the stream (XLEN increments). +//! 2. The stream is capped at the const MAXLEN (~10k events — tested +//! implicitly by the streaming-style emit code; the cap holds). +//! 3. Killing the consumer task + emitting overflow events increments +//! the `drops()` counter exactly by the overflow count. +//! +//! Requires `VALKEY_URL=redis://127.0.0.1:6379/` — set by the CI +//! `services: valkey:` block. Skipped locally when VALKEY_URL is unset +//! (so `cargo test --all` on a dev box without Valkey doesn't fail). + +use std::time::{Duration, SystemTime}; + +use rutster::event_sink::{CallEvent, EventSink}; +use rutster::valkey_sink::ValkeyEventSink; + +fn valkey_url() -> Option { + let raw = std::env::var("VALKEY_URL").ok()?; + url::Url::parse(&raw).ok() +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn lifecycle_event_lands_in_valkey_stream() { + let Some(url) = valkey_url() else { + eprintln!("VALKEY_URL unset; skipping Valkey-backed test (run with `services: valkey:` in CI)"); + return; + }; + let rt = tokio::runtime::Handle::current(); + let sink = ValkeyEventSink::spawn(url.clone(), rt); + + let stream_key = format!("rutster:test:lifecycle:{}", uuid::Uuid::new_v4()); + sink.set_stream_key_for_test(&stream_key); + + let client = redis::Client::open(url.as_str()).unwrap(); + let mut conn = client + .get_multiplexed_async_connection() + .await + .unwrap(); + let _: () = redis::AsyncCommands::del(&mut conn, &stream_key).await.unwrap(); + + let id = rutster_call_model::ChannelId::new(); + let at = SystemTime::now(); + sink.emit(CallEvent::SessionRegistered { id, at }); + + // Give the background task a moment to drain. mpsc::try_send + the + // single XADD per event takes < 1ms locally; CI runners under load + // can see 50ms+ — 2s is plenty of slack. + tokio::time::sleep(Duration::from_secs(2)).await; + + let len: usize = redis::AsyncCommands::xlen(&mut conn, &stream_key).await.unwrap(); + assert!( + len >= 1, + "event should have landed in stream {stream_key} (len={len})" + ); + + drop(sink); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn drop_counter_increments_when_consumer_is_dead() { + let Some(url) = valkey_url() else { + eprintln!("VALKEY_URL unset; skipping Valkey-backed test"); + return; + }; + let rt = tokio::runtime::Handle::current(); + // Spawn the sink with a consumer that immediately fails every XADD. + let sink = ValkeyEventSink::spawn_for_test_with_dead_consumer(url.clone(), rt); + + let initial_drops = sink.drops().unwrap_or(0); + for _ in 0..5 { + sink.emit(CallEvent::SessionRegistered { + id: rutster_call_model::ChannelId::new(), + at: SystemTime::now(), + }); + } + // The background task's queue is bounded at capacity 256; emitting + // 5 events all hit the queue + the dead consumer rejects every XADD. + // Wait for the background task to attempt all sends. + tokio::time::sleep(Duration::from_secs(2)).await; + let after_drops = sink.drops().unwrap_or(0); + assert_eq!( + after_drops, + initial_drops + 5, + "every event should have been counted-and-dropped against the dead consumer" + ); + + drop(sink); +} +``` + +- [ ] **Step 4: Run — expect compile failure (no `valkey_sink` module)** + +```bash +cargo test -p rutster --test valkey_sink_lifecycle +``` + +Expected: `could not find `valkey_sink` in `rutster``. + +- [ ] **Step 5: Implement `ValkeyEventSink`** + +`crates/rutster/src/valkey_sink.rs` (complete file): + +```rust +//! # valkey_sink — first Valkey consumer (deploy spec §5.6) +//! +//! Implements the existing `EventSink` trait (deploy-C §5.6): `XADD` of +//! `CallEvent`s as JSON to a capped stream (`XADD MAXLEN ~ 10000`). +//! STRICTLY fire-and-forget: bounded `mpsc` + background task. Valkey +//! down ⇒ count-and-drop via `Arc` (the same counter +//! `/metrics` reads in deploy-C Task 6). `RUTSTER_VALKEY_URL` unset ⇒ +//! `TracingEventSink` unchanged (main.rs makes the swap). +//! +//! ## Why `XADD MAXLEN ~ 10000` (the adopted default)? +//! +//! ~10k events ≈ a few MB of memory under normal op (a CallEvent +//! serializes to ~200 bytes of JSON; 10k × 200B ≈ 2MB). The forensic +//! window covers days of calls for a small fleet — enough to +//! post-mortem a node's behavior after the operator notices an issue. +//! The cap is `~` (approximate) which lets Valkey trim lazily — lower +//! write-path latency than exact `MAXLEN 10000`. Bumping or dropping +//! the cap is a one-line code change + redeploy; **no operator action +//! on Valkey** (the cap is enforced on the XADD side, not by a separate +//! trim task). Reversible. +//! +//! ## Why NOT the billing ledger? +//! +//! ADR-0005 source-of-truth rule: the bus is NEVER billing source-of- +//! truth. This sink is evidence preservation on a capped stream — +//! explicitly NOT the durable CDR pipeline. CDR → object storage stays +//! deferred (deploy spec §1.2 out-of-scope table). + +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, Mutex}; + +use tokio::sync::mpsc; + +use crate::event_sink::{CallEvent, EventSink, SharedEventSink}; + +/// Stream key on Valkey. Constant for the slice; a future slice makes +/// it configurable when the operator wants a separate stream per +/// tenant (multi-tenancy ADR defers the schema). +pub const STREAM_KEY: &str = "rutster:events"; + +/// XADD MAXLEN cap. ~10k events ≈ a few MB; reversible code constant +/// (see module docs for the justification). +pub const MAXLEN_APPROX: usize = 10_000; + +/// Bounded queue capacity. 256 is large enough to absorb a burst of +/// lifecycle events from a graceful-shutdown mass-hangup (~64 sessions +/// × 3 events = ~200) without dropping; small enough to surface a +/// sustained Valkey outage quickly via the `/metrics` drop counter. +const QUEUE_CAPACITY: usize = 256; + +/// Valkey-backed EventSink. Cloneable (the inner is all `Arc`) so +/// `main.rs` can hand clones to the media thread + the /metrics reader. +#[derive(Clone)] +pub struct ValkeyEventSink { + tx: mpsc::Sender, + drops: Arc, + stream_key: Arc>, +} + +impl ValkeyEventSink { + /// Spawn the background XADD consumer task + return a `SharedEventSink` + /// handle. The caller (main.rs) stores this in `MediaThreadOpts.sink`, + /// replacing the default `TracingEventSink`. + pub fn spawn(url: url::Url, tokio_handle: tokio::runtime::Handle) -> SharedEventSink { + let (tx, rx) = mpsc::channel::(QUEUE_CAPACITY); + let drops = Arc::new(AtomicU64::new(0)); + let stream_key = Arc::new(Mutex::new(STREAM_KEY.to_string())); + let task_state = ConsumerTaskState { + rx: Some(rx), + url, + drops: drops.clone(), + stream_key: stream_key.clone(), + }; + tokio_handle.spawn(background_consumer(task_state)); + Arc::new(ValkeyEventSink { + tx, + drops, + stream_key, + }) + } + + /// Test-only constructor that spawns a background task with a consumer + /// that immediately fails every XADD. Used by the drop-counter test. + #[doc(hidden)] + pub fn spawn_for_test_with_dead_consumer( + url: url::Url, + tokio_handle: tokio::runtime::Handle, + ) -> ValkeyEventSink { + let (tx, rx) = mpsc::channel::(QUEUE_CAPACITY); + let drops = Arc::new(AtomicU64::new(0)); + let stream_key = Arc::new(Mutex::new(STREAM_KEY.to_string())); + let task_state = ConsumerTaskState { + rx: Some(rx), + url, + drops: drops.clone(), + stream_key: stream_key.clone(), + }; + tokio_handle.spawn(background_consumer_with_dead_xadd(task_state)); + ValkeyEventSink { + tx, + drops, + stream_key, + } + } + + /// Test-only: override the stream key so concurrent test legs don't + /// collide on the default `rutster:events` key (tests run in parallel). + #[doc(hidden)] + pub fn set_stream_key_for_test(&self, key: &str) { + *self.stream_key.lock().unwrap() = key.to_string(); + } +} + +impl EventSink for ValkeyEventSink { + fn emit(&self, event: CallEvent) { + // try_send, NOT send(). The media thread must NEVER block on + // Valkey I/O. A full queue = count + drop. The drop counter + // surfaces in /metrics so the operator's alerting catches a + // sustained Valkey outage. + if let Err(mpsc::error::TrySendError::Full(_)) = self.tx.try_send(event) { + self.drops.fetch_add(1, Ordering::Relaxed); + } + // TrySendError::Disconnected means the background task has died. + // Implicit drop-and-count: the event vanished (we leave the + // counter unincremented because "the sink's task died" is a + // different failure mode from "queue full" — operators see the + // task-death via the metrics dropping to zero throughput, not + // via the `drops` counter). If we wanted to count both, swap + // `if let Err(...) => self.drops.fetch_add(1, ...)` here. + } + + fn drops(&self) -> Option { + Some(self.drops.load(Ordering::Relaxed)) + } +} + +/// State carried into the background consumer task. `rx: Option<...>` +/// so the task can take ownership of the `mpsc::Receiver` without +/// requiring `ValkeyEventSink` to also be `Send + 'static` for the +/// `tokio::spawn` boundary (the `url`, `drops`, `stream_key` are +/// `Send + Sync`; `rx` itself is `Send` but not `Sync`). +struct ConsumerTaskState { + rx: Option>, + url: url::Url, + drops: Arc, + stream_key: Arc>, +} + +/// Background consumer: drains the mpsc, serializes each event as JSON, +/// XADDs to the (test-overridable) `stream_key` with `MAXLEN ~ MAXLEN_APPROX`. +/// On XADD failure (Valkey down), counts-and-drops via `drops`. +async fn background_consumer(mut state: ConsumerTaskState) { + let mut rx = state.rx.take().expect("rx present"); + let client = match redis::Client::open(state.url.as_str()) { + Ok(c) => c, + Err(e) => { + tracing::error!(error = %e, "Valkey client open failed; sink will count-and-drop every event"); + drain_and_count(&mut rx, &state.drops).await; + return; + } + }; + let conn = match client.get_multiplexed_async_connection().await { + Ok(c) => c, + Err(e) => { + tracing::error!(error = %e, "Valkey connect failed; sink will count-and-drop every event"); + drain_and_count(&mut rx, &state.drops).await; + return; + } + }; + let mut conn = conn; + + while let Some(event) = rx.recv().await { + let payload = match serde_json::to_string(&EventPayload::from(&event)) { + Ok(s) => s, + Err(e) => { + tracing::error!(error = %e, "CallEvent JSON serialize failed; dropping"); + state.drops.fetch_add(1, Ordering::Relaxed); + continue; + } + }; + let key = state.stream_key.lock().unwrap().clone(); + // XADD MAXLEN ~ * event . + // `~` = approximate cap (lazy trim; lower write latency than exact). + let result: redis::RedisResult<()> = redis::AsyncCommands::xadd_maxlen( + &mut conn, + &key, + false, // approximate ~= MAXLEN ~ + MAXLEN_APPROX, + "*", + &[("event", payload)], + ) + .await; + if let Err(e) = result { + tracing::warn!(error = %e, "XADD failed; dropping event (counter increments in /metrics)"); + state.drops.fetch_add(1, Ordering::Relaxed); + } + } +} + +/// Variant of `background_consumer` whose XADDs always fail — used by +/// `spawn_for_test_with_dead_consumer`. The connection is opened but +/// every XADD targets a non-existent / refused endpoint. We achieve +/// "always fails" by using a URL with an impossible port. +async fn background_consumer_with_dead_xadd(mut state: ConsumerTaskState) { + let mut rx = state.rx.take().expect("rx present"); + // Override the URL with one that fails every connection — the test + // asserts the drop counter, so we don't care about the connection + // error type, only that it fails fast enough for the test's 2s wait. + let dead_url = url::Url::parse("redis://127.0.0.1:1/").unwrap(); + let client = match redis::Client::open(dead_url.as_str()) { + Ok(c) => c, + Err(e) => { + tracing::error!(error = %e, "dead-consumer client open failed"); + drain_and_count(&mut rx, &state.drops).await; + return; + } + }; + // Try to connect — expected to fail. If it succeeds (a Valkey + // happens to be on port 1?), every XADD will fail anyway because + // we'll immediately drop the connection in the loop. + let conn_result = client.get_multiplexed_async_connection().await; + let mut conn = match conn_result { + Ok(c) => c, + Err(_e) => { + drain_and_count(&mut rx, &state.drops).await; + return; + } + }; + while let Some(event) = rx.recv().await { + let _ = event; + let key = state.stream_key.lock().unwrap().clone(); + let result: redis::RedisResult<()> = redis::AsyncCommands::xadd_maxlen( + &mut conn, + &key, + false, + MAXLEN_APPROX, + "*", + &[("event", "{}")], + ) + .await; + if result.is_err() { + state.drops.fetch_add(1, Ordering::Relaxed); + } + } +} + +async fn drain_and_count(rx: &mut mpsc::Receiver, drops: &Arc) { + while let Some(_event) = rx.recv().await { + drops.fetch_add(1, Ordering::Relaxed); + } +} + +/// JSON-serializable shape of a CallEvent. Carries a `kind` tag + the +/// event's fields so a downstream consumer can route without having +/// the rutster crate linked. `serde::Serialize` is derived — the +/// existing `CallEvent` enum doesn't derive Serialize (it's Debug + +/// Clone only); this wrapper does the conversion in `From<&CallEvent>`. +#[derive(Debug, serde::Serialize)] +struct EventPayload { + kind: &'static str, + id: String, + at_ms: u128, + // Extra fields per variant land in follow-ups; for now the kind + + // channel id + timestamp are enough for the forensic window. The + // shape is intentionally EXTENSIBLE — a future slice adds + // `SessionEnded.reason` and `tap_metrics` here when the consumer + // cares. +} + +impl From<&CallEvent> for EventPayload { + fn from(event: &CallEvent) -> Self { + match event { + CallEvent::SessionRegistered { id, at } => EventPayload { + kind: "session_registered", + id: id.0.to_string(), + at_ms: at.duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis()).unwrap_or(0), + }, + CallEvent::SessionConnected { id, at } => EventPayload { + kind: "session_connected", + id: id.0.to_string(), + at_ms: at.duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis()).unwrap_or(0), + }, + CallEvent::SessionEnded { id, ended_at, .. } => EventPayload { + kind: "session_ended", + id: id.0.to_string(), + at_ms: ended_at.duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis()).unwrap_or(0), + }, + } + } +} +``` + +- [ ] **Step 6: Add the CI `services: valkey:` block** + +In `.github/workflows/ci.yml`, the `test` job currently runs `cargo test --all`. To run the new +`valkey_sink_lifecycle` test against a real Valkey, add a `services:` block running +`valkey/valkey:latest` and export `VALKEY_URL` for the test command. Modify the `test` job: + +```yaml + test: + runs-on: ubuntu-latest + services: + # Deploy slice C §5.6: real Valkey service container for the + # valkey_sink_lifecycle integration test. The test skips itself + # when VALKEY_URL is unset, so dev boxes without Valkey don't fail. + valkey: + image: valkey/valkey:latest + ports: + - 6379:6379 + options: >- + --health-cmd "valkey-cli ping" + --health-interval 10s + --health-timeout 5s + --health-retries 3 + strategy: + matrix: + toolchain: [stable, "1.85"] + env: + VALKEY_URL: redis://127.0.0.1:6379/ + steps: + - uses: actions/checkout@v4 + + # Seam gate (slice-4 §7 #3; re-pinned slice-5): + # + # `loop_driver.rs` stays byte-identical to slice-3 — the hot-path + # poll loop is untouched. `rtc_session.rs` was re-pinned in slice-5 + # ... [existing comment preserved] + - name: Seam gate — loop_driver frozen (slice-3) + rtc_session pinned (slice-5) + run: | + EXPECTED_LOOP_DRIVER='744bf314edf7f4925c8bb3bd0f5176dbc88f8113' + EXPECTED_RTC_SESSION='f47d63b9a2883d37066a93c9daa0e2cf8816bec4' + GOT_LOOP_DRIVER=$(git hash-object crates/rutster-media/src/loop_driver.rs) + GOT_RTC_SESSION=$(git hash-object crates/rutster-media/src/rtc_session.rs) + if [ "$GOT_LOOP_DRIVER" != "$EXPECTED_LOOP_DRIVER" ]; then + echo "::error::loop_driver.rs blob mismatch: $GOT_LOOP_DRIVER != $EXPECTED_LOOP_DRIVER" + exit 1 + fi + if [ "$GOT_RTC_SESSION" != "$EXPECTED_RTC_SESSION" ]; then + echo "::error::rtc_session.rs blob mismatch: $GOT_RTC_SESSION != $EXPECTED_RTC_SESSION" + exit 1 + fi + + - uses: dtolnay/rust-toolchain@master + with: + toolchain: ${{ matrix.toolchain }} + - name: Install libopus (media crate FFI dep) + run: apt-get update && apt-get install -y libopus-dev + - uses: Swatinem/rust-cache@v2 + - run: cargo test --all +``` + +Apply the same `services: valkey:` block + `VALKEY_URL` env to the `clippy` job (so clippy can +type-check the test code under `--all-targets`, which includes the new +`valkey_sink_lifecycle.rs`). The `clippy` job is small; add only the `services:` block and the +`env:` block, no other changes. + +MUST add `uuid` to `crates/rutster/Cargo.toml` `[dev-dependencies]` if not already present +(the `valkey_sink_lifecycle.rs` test uses `uuid::Uuid::new_v4()` for the per-test stream key; +verified `uuid` is already a workspace dep but may not be in the rutster crate's dev-deps — +spot-check via `cargo tree -p rutster`): + +```toml +uuid = { workspace = true } +``` + +(If `uuid` is already in dev-deps, skip this addition — clippy will surface the unused-import +warning if not.) + +- [ ] **Step 7: Wire `main.rs` to swap TracingEventSink → ValkeyEventSink** + +In `crates/rutster/src/main.rs`'s `MediaThreadOpts` construction (currently lines 47–53), replace +the default `..Default::default()` sink line: + +```rust + // Deploy slice C §5.6: swap TracingEventSink -> ValkeyEventSink + // when RUTSTER_VALKEY_URL is set. Strictly fire-and-forget: + // mpsc::try_send + background XADD task + count-and-drop (counter + // surfaces in /metrics). Unset -> today's TracingEventSink. + let valkey_url = rutster::config::valkey_url(std::env::var("RUTSTER_VALKEY_URL").ok()) + .expect("RUTSTER_VALKEY_URL must be redis:// or rediss:// when set"); + let sink: rutster::event_sink::SharedEventSink = match valkey_url { + Some(url) => { + info!(%url, "ValkeyEventSink enabled"); + rutster::valkey_sink::ValkeyEventSink::spawn(url, tokio::runtime::Handle::current()) + } + None => { + info!("TracingEventSink (default; set RUTSTER_VALKEY_URL for durable events)"); + std::sync::Arc::new(rutster::event_sink::TracingEventSink) + } + }; + let opts = rutster::media_thread::MediaThreadOpts { + media_cfg, + max_sessions: rutster::config::max_sessions(std::env::var("RUTSTER_MAX_SESSIONS").ok()) + .expect("RUTSTER_MAX_SESSIONS must be a number"), + sink, + }; +``` + +(Concretely: replace the `..Default::default()` in the existing `opts` construction with this +explicit `sink:` field.) + +- [ ] **Step 8: Run tests — expect pass** + +```bash +cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings +cargo test -p rutster config::tests +cargo test -p rutster --test valkey_sink_lifecycle # skips locally when VALKEY_URL unset +cargo test --all +``` + +Expected: all green locally (the valkey test self-skips on `VALKEY_URL` unset). In CI with the +`services: valkey:` block, both lifecycle tests run against the real container and pass. + +- [ ] **Step 9: Commit** + +```bash +cd /home/alee/Sources/rutster +git add crates/rutster/Cargo.toml crates/rutster/src/valkey_sink.rs crates/rutster/src/config.rs crates/rutster/src/main.rs crates/rutster/tests/valkey_sink_lifecycle.rs .github/workflows/ci.yml +git commit -s -m "feat(valkey_sink): ValkeyEventSink — first Valkey consumer (deploy-C §5.6) + +Implements the existing EventSink trait: XADD of CallEvents as JSON to a +capped stream (XADD MAXLEN ~ 10000). Strictly fire-and-forget: bounded +mpsc (cap 256) + background tokio::spawn task. Valkey down => +count-and-drop via Arc (the same counter /metrics reads in +Task 6 as rutster_event_sink_drops_total). RUTSTER_VALKEY_URL unset => +existing TracingEventSink unchanged (main.rs swaps). + +XADD MAXLEN ~ 10000 is the adopted default: ~10k events ≈ 2MB of memory; +days-of-calls forensic window; bounded recovery time if Valkey is +restarted with a backlog. Reversible — it is a const in the module, not a +Valkey-side migration. ADR-0005 source-of-truth rule restated in the +module docs: this is evidence preservation, NOT the billing ledger. + +CI gains a services: valkey: block on the test + clippy jobs running +valkey/valkey:latest on 127.0.0.1:6379; VALKEY_URL env exported for the +test command. Integration tests self-skip locally when VALKEY_URL is +unset so dev boxes without Valkey don't fail. + +config.rs gains valkey_url parser (redis:// + rediss:// schemes rejected +otherwise; follows the house fail-fast pattern). main.rs swaps the sink +in MediaThreadopts." +``` + +--- + +### Task 9: Extend plan B's smoke harness with the lifecycle-event-lands-in-Valkey-stream check + +**Files:** +- Modify: `docs/superpowers/plans/2026-07-05-deploy-b-packaging-ci.md` (the smoke harness + section — plan B's smoke CI's named hook is referenced; if plan B does not exist yet at + execution time, this task is a NO-OP until plan B lands and its smoke harness is in place — + see dependency note below). + +**Dependency note:** at the time this plan was written, plan B +([`docs/superpowers/plans/2026-07-05-deploy-b-packaging-ci.md`](2026-07-05-deploy-b-packaging-ci.md)) +does NOT exist on disk (verified by glob). The smoke harness hook — `assert-lifecycle-event-lands-in-Valkey-stream` — +is what slice B's smoke CI is supposed to leave as a named hook for slice E to fill in (per the +spec's §11 sequencing: E depends on B "for the smoke harness shell," though Phase E itself has +no code dependency on B). If plan B has not landed yet by the time Task 9 is reached, this task +defers; the integration test `crates/rutster/tests/valkey_sink_lifecycle.rs` from Task 8 +already proves the lifecycle-event-XADD round-trip end-to-end against a real Valkey service +container, so the slice's *acceptance bar* does not regress in the meantime. + +- [ ] **Step 1: Verify plan B exists + find the smoke harness hook** + +```bash +cd /home/alee/Sources/rutster +ls docs/superpowers/plans/2026-07-05-deploy-b-packaging-ci.md 2>/dev/null +grep -n 'assert-lifecycle-event-lands-in-Valkey-stream' \ + docs/superpowers/plans/2026-07-05-deploy-b-packaging-ci.md 2>/dev/null +``` + +If the file or the hook line is absent, STOP — this task is a NO-OP. Document the deferral in +the PR description for this slice: "Task 9 deferred — plan B's smoke harness hook +`assert-lifecycle-event-lands-in-Valkey-stream` is not on disk; the slice's acceptance bar is +met by `crates/rutster/tests/valkey_sink_lifecycle.rs` running against the real Valkey +service container." + +If the file IS present and contains the hook line, continue to Step 2. + +- [ ] **Step 2: Implement the hook in plan B's smoke harness** + +Add the assertion to plan B's smoke harness at the named hook. The assertion: after the +all-in-one smoke simulates a session lifecycle (register + delete), the RutsterEventSink's +Valkey stream has grown by ≥ 2 entries (one for `SessionRegistered`, one for `SessionEnded`). +Use the same `redis::AsyncCommands::xlen` query as Task 8's integration test, against the +default `STREAM_KEY = "rutster:events"` (the all-in-one container ships Valkey). + +- [ ] **Step 3: Commit** + +(Details vary by plan B's smoke harness structure at execution time — name the change "feat +(smoke): assert-lifecycle-event-lands-in-Valkey-stream hook (deploy-C §5.6, deploy-B smoke +harness)" and follow plan B's commit-message house style.) + +--- + +### Task 10: Final verification sweep + +No code. Evidence before assertions — run every gate the CI runs, plus the seam-gate check, +before calling the slice done. + +- [ ] **Step 1: Full gate run** + +```bash +cargo fmt --all --check +cargo clippy --all --all-targets -- -D warnings +cargo clippy --all --all-targets --features=sim-bench -- -D warnings +cargo test --all +cargo test --all --features=sim-bench -- --test-threads=1 +cargo deny check +cargo doc --no-deps +``` + +Expected: every command exits 0. + +- [ ] **Step 2: Seam gate verification** + +```bash +git hash-object crates/rutster-media/src/loop_driver.rs +git hash-object crates/rutster-media/src/rtc_session.rs +``` + +Expected output, exactly: + +``` +744bf314edf7f4925c8bb3bd0f5176dbc88f8113 +f47d63b9a2883d37066a93c9daa0e2cf8816bec4 +``` + +If either hash differs, STOP — a task touched a seam-gated file; restore with +`git checkout main -- crates/rutster-media/src/loop_driver.rs crates/rutster-media/src/rtc_session.rs` +and re-run the gates. + +- [ ] **Step 3: TLS smoke (manual)** + +Generate a self-signed cert + key locally, run the binary with both env vars set, and verify +the HTTPS path round-trips a `curl -k https://localhost:8080/healthz`. Then send `kill -HUP` +to the process after rewriting the cert to a different CN; verify in the process's logs that +the "TLS cert reloaded" line appears. (This is a manual smoke — the automated tests cover the +contract; the manual smoke covers the operator UX.) + +- [ ] **Step 4: MSRV spot-check (matches the CI 1.85 leg)** + +```bash +cargo +1.85 test --all +``` + +Expected: green. (If the 1.85 toolchain is not installed locally, note it and rely on the CI +matrix — do not skip silently.) + +--- + +## Final acceptance checklist + +After all tasks land: + +- [ ] `cargo fmt --check`, `clippy -D warnings` (default + `--features=sim-bench`), + `cargo test --all`, sim-bench sweep (`--test-threads=1`), `cargo deny check`, + `cargo doc --no-deps` all clean. +- [ ] Seam gate: `loop_driver.rs` + `rtc_session.rs` blob hashes unchanged (Task 10 Step 2). +- [ ] `cargo deny check` stays green after adding `axum-server 0.8` (Task 1) and `redis 1.3.0` + (Task 7) — license + multiple-versions gates both pass. +- [ ] TLS listener: BOTH `RUTSTER_TLS_CERT` + `RUTSTER_TLS_KEY` set ⇒ `serve_tls_with_nodelay` + with `axum_server::bind` + `NodeLayAcceptor` (TCP_NODELAY parity with plan A's plaintext + path) + SIGHUP hot-reload watcher. Either unset ⇒ plan A's plaintext + `serve_with_nodelay` (artifact default). Partial config panics at boot with a clear error. +- [ ] TLS hot-reload contract: `RustlsConfig::reload_from_pem_file` is atomic (arc-swap) — + `crates/rutster/tests/serve_tls_hot_reload.rs` proves it with peer_certificates differing + across a reload boundary. +- [ ] `/metrics` endpoint on `RUTSTER_METRICS_BIND` (default `127.0.0.1:9090`), on its OWN + `axum::Router` — never on the main `router()`. Metrics named: + `rutster_sessions_active` (gauge), `rutster_max_sessions` (gauge), `rutster_draining` + (gauge), `rutster_tick_overruns_total` (counter), `rutster_last_tick_micros` (gauge), + `rutster_admission_rejects_total` (counter), `rutster_event_sink_drops_total` (counter — + absent when TracingEventSink; present only when ValkeyEventSink is wired). +- [ ] `ValkeyEventSink`: bounded mpsc + background XADD task; `RUTSTER_VALKEY_URL` set ⇒ + replaced `TracingEventSink`; unset ⇒ `TracingEventSink` unchanged. Drop counter + (`Arc`) surfaces in `/metrics` as `rutster_event_sink_drops_total`. + `XADD MAXLEN ~ 10000` (the `const` in `valkey_sink.rs`) caps the stream; reversible + code constant, NOT a Valkey migration. +- [ ] CI: `services: valkey:` block on the `test` + `clippy` jobs running + `valkey/valkey:latest` on `127.0.0.1:6379`; `VALKEY_URL` exported. The + `valkey_sink_lifecycle.rs` test runs against the real container in CI; self-skips + locally when `VALKEY_URL` is unset. +- [ ] PR opened via `tea pulls create --head deploy-c/binary-features --base main --title + "deploy slice C (binary features): rustls Phase 1 TLS + /metrics + ValkeyEventSink"` + with this plan's §5.4 + §5.5 + §5.6 as the description skeleton. Sequencing per spec §11: + C, D, E are order-independent; this slice merges in any order relative to slices A and B + (modulo the plan-B smoke harness hook noted in Task 9). + diff --git a/docs/superpowers/plans/2026-07-05-deploy-g-docs-adr-0011.md b/docs/superpowers/plans/2026-07-05-deploy-g-docs-adr-0011.md new file mode 100644 index 0000000..1b0f57c --- /dev/null +++ b/docs/superpowers/plans/2026-07-05-deploy-g-docs-adr-0011.md @@ -0,0 +1,1610 @@ +# Slice G — `docs/deploy/` tree + ADR-0011 (deployment topology) — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use `superpowers:subagent-driven-development` (recommended) or `superpowers:executing-plans` to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Draft ADR-0011 (deployment topology, status Proposed) from spec §2–§4 and ship the +six-file `docs/deploy/` documentation tree, where **every topology doc contains a copy-paste +path from zero to first call — or the honest statement of what cannot work there** (spec §10 +acceptance bar), then point `docs/QUICKSTART.md` at the new Docker fast path. + +**Architecture:** Docs-only slice — no code, no artifacts. ADR-0011 is drafted verbatim-faithful +to the [deployment-topology spec](../specs/2026-07-05-deployment-topology-design.md) §2 (three +shapes + rejected shapes), §3 (edge doctrine incl. the rustls Phase-2 triggers verbatim), §4 +(node-addressed placement), in the house ADR format established by ADR-0007/ADR-0008 +(bullet-list header → Context → Decision → Consequences → References, heavy relative +cross-linking). The `docs/deploy/` tree sources every factual claim from the spec + the +[TLS/edge decision brief](../specs/2026-07-05-tls-edge-decision-brief.md) and cites the brief's +external URLs wherever a doc asserts a third-party fact. + +**Tech Stack:** Markdown only. No new dependencies, no build steps. Verification is +`bash` content-greps + a relative-link resolver check per task. + +## Global Constraints + +House rules (every task's requirements implicitly include this section): + +- **License:** GPL-3.0-or-later on every crate manifest (ADR-0004). No crates are touched in + this slice; the rule stands for any task that would. +- **DCO:** every commit signed off — `git commit -s` (AGENTS.md Git workflow). +- **SEAM GATE:** `crates/rutster-media/src/loop_driver.rs` and + `crates/rutster-media/src/rtc_session.rs` stay byte-identical (CI pinned-blob). **NO task may + touch them.** This slice touches only `docs/` — trivially satisfied, still checked in the + final sweep. +- **Hot-path policy:** never `?`-propagate on the 20 ms tick; no `unwrap()`/`expect()` outside + tests. (No code in this slice; stated because every plan carries it.) +- **Code style:** `cargo fmt --all --check` + `cargo clippy --all --all-targets -- -D warnings` + must stay green (docs changes cannot break them; run them in the final sweep anyway). +- **MSRV:** CI matrix runs stable + 1.85; docs must not instruct anything that contradicts the + pinned `rust-toolchain.toml`. +- **Workspace dep pinning:** new deps go in the ROOT `Cargo.toml` `[workspace.dependencies]`; + members reference with `dep.workspace = true`. (None added here.) +- **cargo-deny license gate** stays green (unaffected; final sweep runs it). +- **sim-bench CI job** runs `--test-threads=1` (unaffected). + +Slice-G-specific constraints: + +- **Acceptance bar (spec §10):** every topology doc contains a copy-paste path from zero to + first call, **or** the honest statement of what cannot work there (e.g. CGNAT WebRTC). Each + doc task's verification step greps for the load-bearing statements. +- **Artifact names:** plan B (`docs/superpowers/plans/2026-07-05-deploy-b-packaging-ci.md`) + has landed and settled the registry namespace to **`git.adlee.work/alee/rutster-*`** + (matches the `alee` tea login per AGENTS.md + the root `Cargo.toml`'s + `repository = "https://git.adlee.work/alee/rutster"`). Images: `rutster-allinone`, + `rutster-engine`, `rutster-brain`, `rutster-edge`. `deploy/` tree = `Dockerfile`, + `compose.yaml`, `Caddyfile`, `.env.example`. Every doc task still carries a **cross-check + step**: if `deploy/` exists in the worktree by execution time, reconcile env-var names and + compose service names against the real files before committing. +- **Env vars:** grounded in `crates/rutster/src/config.rs` for what exists today + (`RUTSTER_HTTP_BIND`, `RUTSTER_MEDIA_BIND_IP`, `RUTSTER_MEDIA_ADVERTISED_IP`, + `RUTSTER_MEDIA_PORT_RANGE`, `RUTSTER_MAX_SESSIONS`, `RUTSTER_DRAIN_DEADLINE_SECS`, the four + `RUTSTER_TWILIO_*` vars) and in spec §5.7 for what sibling slices A/C/D/E add + (`RUTSTER_TRUSTED_PROXIES`, `RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY`, `RUTSTER_WS_PING_SECS`, + `RUTSTER_METRICS_BIND`, `RUTSTER_VALKEY_URL`). The docs document the **post-epoch** artifact + set — slice G is sequenced last (spec §11) — so referencing §5.7 vars is correct, not + speculative. +- **Citations:** wherever a `docs/deploy/` file makes a factual claim about third-party + behavior, cite the TLS brief's external source URL inline (Twilio docs/error pages, the + Caddy/cloudflared/axum/livekit issue URLs, etc.). +- **Ports (grounded):** one FOB listener `:8080` (`RUTSTER_HTTP_BIND`; + `crates/rutster/src/main.rs` merges the trunk WS router into the main app — the + `/twilio/media-stream` WS rides the same listener); brain tap `127.0.0.1:8082`; valkey + `:6379`; metrics `127.0.0.1:9090` (spec §5.5); Caddy `:80`/`:443` public; WebRTC media = + UDP `RUTSTER_MEDIA_PORT_RANGE` direct to the FOB, never through Caddy. +- **Do NOT restyle existing docs.** Task 8 (QUICKSTART) is a minimal-diff pointer insertion, + nothing else. + +## File Structure + +| Path | Action | Responsibility | +|---|---|---| +| `docs/adr/0011-deployment-topology.md` | Create | The topology ADR: three blessed shapes, edge doctrine, fleet placement paradigm, rejected shapes. Status: Proposed. | +| `docs/deploy/topologies.md` | Create | The three shapes + decision-guide table; entry point of the tree. | +| `docs/deploy/quickstart-docker.md` | Create | T1 `docker run` + T2 `compose up`, zero to first call (browser + PSTN). | +| `docs/deploy/homelab.md` | Create | The CGNAT truth: PSTN-only behind tunnels; ngrok blessed dev path; Tailscale Funnel; VPS+WireGuard graduation with TLS at home. | +| `docs/deploy/aws.md` | Create | ALB appendix: three mandatory attribute overrides, 100-TG cap, never-NLB-TLS. | +| `docs/deploy/reverse-proxies.md` | Create | BYO nginx/HAProxy/Traefik tuned-timeout snippets; the universal 60 s footgun; Traefik version pinning. | +| `docs/deploy/certificates.md` | Create | ACME paths, wildcard traps, LE duplicate-cert lockout, `/data` persistence, per-node vs wildcard fleet patterns. | +| `docs/QUICKSTART.md` | Modify (minimal) | One blockquote pointing at `deploy/quickstart-docker.md` as the fast path. | + +## Task ordering + +- **Task 1 (ADR-0011) has NO dependency — execute immediately.** It is drafted from the spec, + not from artifacts. +- **Tasks 2–7** are order-independent among themselves and parallelizable; each carries the + slice-B cross-check step (spec §11: G depends on B "for accuracy"). +- **Task 8** depends on Task 3 (the file it links to must exist). + +Each task = one commit. Verification per task is a bash content-check that fails before the +file is written and passes after. + +--- + +### Task 1: ADR-0011 — `docs/adr/0011-deployment-topology.md` + +**Files:** +- Create: `docs/adr/0011-deployment-topology.md` +- Test: bash content-check (Step 1/Step 3 below) + +**Interfaces:** +- Consumes: spec §2/§3/§4 (`docs/superpowers/specs/2026-07-05-deployment-topology-design.md`), + the TLS brief (`docs/superpowers/specs/2026-07-05-tls-edge-decision-brief.md`), the house ADR + format (read `docs/adr/0007-trunk-rented-transport.md` and `docs/adr/0008-fob-and-green-zone.md` + before writing — the header bullet-list, Context/Decision/Consequences/References structure, + and relative cross-link style below match them exactly). +- Produces: the ADR every `docs/deploy/` file cross-links as + `../adr/0011-deployment-topology.md` (Tasks 2–7 rely on this exact path). + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/adr/0011-deployment-topology.md \ + && grep -q 'Status:\*\* Proposed' docs/adr/0011-deployment-topology.md \ + && grep -q 'dns-persist-01' docs/adr/0011-deployment-topology.md \ + && grep -q '307' docs/adr/0011-deployment-topology.md \ + && grep -q 'on-demand TLS' docs/adr/0011-deployment-topology.md \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL` (file does not exist). + +- [ ] **Step 2: Write the complete ADR** + +Write `docs/adr/0011-deployment-topology.md` with exactly this content: + +````markdown +# ADR-0011 — Deployment topology: one binary, three blessed shapes + +- **Status:** Proposed +- **Date:** 2026-07-05 +- **Closes:** the deployment-topology ADR reserved by [ADR-0008](0008-fob-and-green-zone.md)'s + worked example ("the deployment topology itself may get its own ADR once that design closes"). +- **Origin:** [2026-07-05 deployment-topology spec](../superpowers/specs/2026-07-05-deployment-topology-design.md) + §2–§4. The TLS/edge fork was researched via a six-family survey — see the companion + [TLS/edge decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md). +- **Related:** [ADR-0002](0002-north-star-and-fused-core.md) (the fused per-call vertical no + topology may split), [ADR-0005](0005-event-bus.md) (Valkey = bus + KV + presence; media never + rides the bus; the bus is never billing source-of-truth), [ADR-0006](0006-ingress-posture.md) + (two inbound protocols only; the tap is egress-only), [ADR-0007](0007-trunk-rented-transport.md) + (rented transport ⇒ public HTTPS/WSS reachability is non-negotiable), + [ADR-0008](0008-fob-and-green-zone.md) (the FOB/green-zone rule this ADR applies to + deployment), [ADR-0009](0009-spend-gate-honest-rescope.md) (ledger trait: in-memory + single-node, Valkey-backed fleets), [ADR-0010](0010-spearhead-benchmark-sim-harness.md) + (sequencing this ADR must not reorder), README pillar 6 ("one binary, one bus, one deploy"). + +## Context + +[ADR-0008](0008-fob-and-green-zone.md) made the build-vs-reuse doctrine physical in a worked +example — the all-in-one container — and reserved the topology ADR for when the design closed. +Two things have since closed it: the rented transport +([ADR-0007](0007-trunk-rented-transport.md)) is merged, so every deployment must now present a +public HTTPS/WSS edge to a CPaaS; and slice-5/seams landed the operational primitives a +deployment story needs (drain lifecycle, `/readyz`, admission cap, `MediaAddressConfig`, +`EventSink`). + +The CPaaS side imposes a hard external spec on any edge rutster ships (researched in the +[decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md); restated here +because these invariants are load-bearing for every decision below): + +1. Publicly-CA-trusted cert; **no self-signed path exists** (Twilio error 31910). Auto-renewal + is availability-critical. +2. wss:// on 443, open to the whole internet: Twilio publishes no egress IPs and offers no + mTLS. Auth is application-layer only (HMAC signatures). +3. One WS per call, up to 24 h, 50 msg/s/direction at 20 ms cadence; zero tolerance for frame + buffering, connection-lifetime caps, or write-side idle timers. Neither Twilio nor Telnyx + documents WS keepalive — keepalive is entirely our job. +4. `` URLs carry **no query strings** — wss routing keys on hostname/path only. +5. `X-Twilio-Signature` is HMAC over the URL *as Twilio saw it*: the edge must forward + `X-Forwarded-Proto/Host` honestly and the FOB must reconstruct the public URL. +6. Webhook budget: sub-second for UX, 15 s hard cap — no blocking issuance in the handshake + path. +7. TLS 1.2+1.3, mainstream ECDHE, never 1.3-only; never pin Twilio certs. + +Design for Twilio; Telnyx/Vonage come for free (strictly less demanding, plus optional +source-IP allowlisting as extra hardening Twilio can't use). + +On the other side, the FOB's own physics constrain every shape: per-call state is in-process +and non-migratable (the fused vertical, [ADR-0002](0002-north-star-and-fused-core.md)), WebRTC +media is per-session UDP direct to the process, and calls run for hours. Any topology that +assumes migratable, short-lived, connectionless workloads is disqualified before it starts. + +## Decision + +**The FOB never decomposes; topology is a configuration property, not a code property.** The +same binary runs in every blessed shape; only `RUTSTER_*` env and what is deployed alongside it +differ. "Microservices" in rutster means *green-zone services get their own +processes/containers* — never that the per-call vertical splits +([ADR-0002](0002-north-star-and-fused-core.md)). + +### The three blessed shapes + +#### T1 — Solo (all-in-one container) — shipped artifact + +One image (`rutster-allinone`), s6-overlay as PID 1 supervising four processes: + +| Process | Role | Binding | +|---|---|---| +| `caddy` | Edge: ACME, TLS termination, WS proxy to FOB | :443/:80 public | +| `rutster` (FOB) | Engine | 127.0.0.1:8080 (behind Caddy); media UDP direct | +| `rutster-brain-realtime` | Brain | 127.0.0.1:8082 loopback tap (as today) | +| `valkey-server` | Dark on day one except `EventSink` | 127.0.0.1:6379 | + +Two volumes, both non-negotiable: `/data` (Caddy cert/ACME state — loss risks the Let's +Encrypt duplicate-cert lockout, a total-inbound-outage class of failure) and `/var/lib/valkey` +(stream/state persistence). WebRTC media UDP goes **direct to the FOB**, never through Caddy: +host networking recommended; otherwise published `RUTSTER_MEDIA_PORT_RANGE` + +`RUTSTER_MEDIA_ADVERTISED_IP` (NAT 1:1 model, no STUN — unchanged from slice-5/seams). + +Valkey ships bundled from v1 **before most consumers exist**: the operator contract (volumes, +ports, upgrade shape) is stable from day one, so the spend ledger +([ADR-0009](0009-spend-gate-honest-rescope.md)) and the fleet directory land later without +changing the deployment shape. Cost accepted: ~15 MB and one mostly-idle process. + +#### T2 — Modular (compose stack) — shipped artifact, the reference deployment + +Services: `caddy` (image `rutster-edge`), `engine` (`rutster-engine`), `brain` +(`rutster-brain`), `valkey` (upstream `valkey/valkey`). + +- **The brain shares the engine's network namespace** (`network_mode: "service:engine"`) so + the loopback-only tap posture (`resolve_tap_url` rejects non-loopback until step 6) survives + unchanged. The brain graduates to its own netns when the wss:// tap lands (step 6). +- Valkey is its own service on the compose network — network-near, satisfying + [ADR-0005](0005-event-bus.md)'s sub-ms rule. +- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` (grace > drain). +- BYO-proxy supported: disable the `caddy` service; the engine keeps its plaintext `:8080` + mode; tuned-timeout snippets for nginx/HAProxy/Traefik ship in + [`docs/deploy/reverse-proxies.md`](../deploy/reverse-proxies.md). + +#### T3 — Fleet — ADR chapter only (paper; no code in this epoch) + +N **symmetric** nodes, each running the T1 process set *minus* valkey, plus a shared green +zone: one Valkey (presence + directory + the [ADR-0009](0009-spend-gate-honest-rescope.md) +spend ledger) and object storage (named as required, still deferred). Wildcard DNS +`*.pbx.domain`; per-node certs (see the fleet cert story below). The per-node admission cap is +the [ADR-0010](0010-spearhead-benchmark-sim-harness.md) benchmark number (replacing the +`RUTSTER_MAX_SESSIONS=64` placeholder). Scale-in is drain-then-terminate only; spot instances +are rejected for the engine tier (2026-07-04 review — a reclaimed node kills every +non-migratable call on it). + +### Edge doctrine + +#### Bundled edge: Caddy (both shipped artifacts, same build, same Caddyfile) + +Custom xcaddy build (the `rutster-edge` image): Caddy core (Apache-2.0) + a curated DNS-01 +plugin set — cloudflare, route53, porkbun, hetzner, desec (duckdns excluded: no license file). +Zero-downtime in-memory cert renewal; verified protocol-unaware WS tunneling (no frame +buffering). Config is a ~6-line Caddyfile with tuned timeouts, honest `X-Forwarded-*`, and +`stream_close_delay` above max call duration — **plus a CI e2e case for +config-reload-during-live-call, because the upstream mitigation has an open bug trail +(caddy #6420/#7222) and is not trusted untested.** + +#### In-process TLS: staged — the ratified end-state, not adopted wholesale now + +[ADR-0008](0008-fob-and-green-zone.md) claims the trust-domain edge as security-constitutive +FOB territory, and str0m already terminates DTLS-SRTP in-process on the same aws-lc-rs +provider — "keep TLS out of the FOB" is not the objection. The objection is economics: owning +an ACME renewal state machine forever, in a 45-day-cert world. + +- **Phase 1 (this epoch):** `axum-server` + rustls with operator-supplied certs + (`RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY`), hot-reload without dropping live WS. + Plaintext-behind-Caddy stays the artifact default. +- **Phase 2 (in-binary ACME) triggers, named:** (a) Let's Encrypt dns-persist-01 confirmed GA + (closes the wildcard/CGNAT issuance gap; instant-acme already parses it); (b) a field + incident where Caddy reload/`stream_close_delay` drops live calls; (c) fleet or + VPS-forwarder topology making encrypted-to-the-binary a selling point; (d) maintenance + fatigue with the curated xcaddy plugin build. Until a trigger fires, Phase 2 is deliberately + not built — no turnkey ACME crate fits the tree today (rustls-acme conflicts with the + deny.toml rcgen/x509-parser posture). + +#### Fleet cert story + +Per-node public TLS names are CPaaS-imposed (invariant 4 + node-addressed placement). Two +blessed patterns: + +1. **Wildcard `*.pbx.domain` via DNS-01** with central issuance/distribution. +2. **Per-node distinct certs** (`node-N.domain`) where the wildcard private key must not live + on every node — one compromised node otherwise burns the whole namespace. + +Traps, stated so nobody re-discovers them in production: N nodes independently requesting the +*identical* wildcard hit the Let's Encrypt duplicate-cert limit (5/week) — distinct names or +central distribution only. **Caddy on-demand TLS is rejected for node names**: the first +handshake to a fresh name blocks seconds on issuance, colliding with the sub-second webhook +budget (invariant 6), and its rate-limit knobs are deprecated. Node names known at provision +time get pre-issued certs. + +### Fleet placement paradigm (the trunk webhook handler and session API grow *toward* this; no fleet code in this epoch) + +**Node-addressed URLs + Valkey presence; no router tier.** (Decision 2026-07-05 over the two +rejected alternatives below.) + +- Nodes heartbeat capacity into Valkey (TTL'd presence keys — the + [ADR-0005](0005-event-bus.md) presence layer). +- Placement happens **at answer time** by whichever node the request lands on (`api.domain`, + DNS round-robin): read fleet capacity → pick owner → return TwiML whose Stream URL is + `wss://node-K.pbx.domain/...`. The media WS dials the owning node directly — zero extra + hops, consistent with connection-follows-ownership. Twilio's answer-time Stream URL is + purpose-shaped for this: invariant 4 permits hostname routing. +- WebRTC: the create-session response carries the owner's node-addressed base URL + advertised + media IP (UDP is already direct). +- Misdirected control ops (e.g. a DELETE hitting a non-owner) consult the Valkey directory and + **307-redirect** to the owner. +- **Degraded mode (Valkey dark): the answering node self-places if it has capacity**; presence + heals via TTL on recovery. Fail-safe degradation, consistent with + [ADR-0009](0009-spend-gate-honest-rescope.md)'s posture. +- Single-node is the degenerate case of the same code: a one-entry directory. + +**Rejected: a routing tier that proxies everything.** It would put every 20 ms media frame +through an extra userspace hop and re-create exactly the SPOF the symmetric design avoids — +and it buys nothing, because the CPaaS already lets the answer-time URL do the routing. + +**Rejected: a distinguished control node.** It breaks node symmetry (two artifacts, two +failure stories, a promotion protocol) to solve a problem the presence directory already +solves; placement needs fleet state either way, and Valkey is already the fleet-state home +([ADR-0005](0005-event-bus.md)). + +### Rejected shapes + +- **Decomposing the FOB into services** — rejected **permanently**, not deferred. + Media/reflex/tap/spend stay one process, one trust domain + ([ADR-0002](0002-north-star-and-fused-core.md)). +- **Serverless.** Lambdas are categorically unfit for the engine: non-migratable multi-hour + calls, per-session UDP sockets, in-process media state. Webhook-on-Lambda buys nothing + because placement needs fleet state anyway. +- **Tunnels in production.** Cloudflare Tunnel and ngrok see plaintext audio at the vendor + edge (an unconsented subprocessor — DPA/BAA/PCI failure), document mid-call WS kills, and + cloudflared has an open Twilio handshake bug (cloudflared #1465). ngrok is the blessed + 5-minute demo path (the only tunnel with a proven Media Streams record); Tailscale Funnel is + the privacy-clean single-user demo (TLS terminates on-node), unsizable beyond that. **No + tunnel carries inbound UDP → homelab behind CGNAT is PSTN-only, period.** The production + graduation path is a cheap VPS + WireGuard as a dumb TCP forwarder with TLS terminating at + home, or running the engine on the VPS — see + [`docs/deploy/homelab.md`](../deploy/homelab.md). +- **NLB TLS listeners — never.** Fixed 350 s idle timeout that silently drops flows, and no + HTTP context, so `X-Forwarded-*` headers for signature validation (invariant 5) don't exist. + The ALB path is documented as a docs-only appendix + ([`docs/deploy/aws.md`](../deploy/aws.md)), never a shipped artifact. +- **k8s manifests/Helm.** Compose-first (README pillar 6). k8s notes live in docs only (drain + vs `terminationGracePeriodSeconds`); manifests are a later rung if demand appears. + +### Deferred, named (so the boundary is deliberate) + +- **T3 fleet implementation** (presence heartbeats, directory redirects, placement code). +- **The zero-egress / air-gap profile** (SBC layer-2 ingress per + [ADR-0007](0007-trunk-rented-transport.md) layer 2 + a self-hosted brain): a defined + profile, no code this epoch. T1 is about one-command simplicity, not isolation. +- **In-binary ACME (rustls Phase 2)** — behind the four named triggers above. +- **Durable CDR pipeline**: `ValkeyEventSink` is evidence preservation on a capped stream — + explicitly NOT the billing ledger ([ADR-0005](0005-event-bus.md) source-of-truth rule). + CDR → object storage stays deferred. +- **Multi-tenancy** (needs its own ADR before schemas ossify); wss:// tap + brain fleet + + resume tokens (step 6); media-thread sharding graduation; arm64 images. + +## Consequences + +- **Positive:** one binary and one mental model across all three shapes — an operator's + knowledge transfers from `docker run` to a fleet; the CPaaS invariants are satisfied by a + decade-hardened ACME loop (Caddy) rather than a DIY renewal state machine; the fleet + paradigm needs zero new infrastructure beyond what [ADR-0005](0005-event-bus.md) already + bundles; the rejected shapes are written down, so nobody burns a week discovering that + Lambda or NLB-TLS can't work. +- **Negative:** the curated xcaddy DNS-plugin set is a permanent maintenance surface until + dns-persist-01 GA; Caddy config-reload behavior vs live calls is mitigated + (`stream_close_delay` + CI e2e) but has an open upstream bug trail; homelab WebRTC behind + CGNAT is unsolved **by design** — only the VPS graduation fixes it, and the docs must say so + loudly or operators will read "homelab supported" as including browser calls. +- **Mitigation:** the Phase-2 triggers convert both negatives into a planned exit; the + `docs/deploy/` tree ships with a copy-paste-to-first-call acceptance bar so the honest + limits are in the operator's face, not in a footnote. + +## References + +- [2026-07-05 deployment-topology spec](../superpowers/specs/2026-07-05-deployment-topology-design.md) — §2–§4 are this ADR's source +- [TLS/edge decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md) — six-family survey + the CPaaS ground truth (external citations live there) +- [ADR-0002](0002-north-star-and-fused-core.md) · [ADR-0005](0005-event-bus.md) · [ADR-0006](0006-ingress-posture.md) · [ADR-0007](0007-trunk-rented-transport.md) · [ADR-0008](0008-fob-and-green-zone.md) · [ADR-0009](0009-spend-gate-honest-rescope.md) · [ADR-0010](0010-spearhead-benchmark-sim-harness.md) +- The `docs/deploy/` tree: [topologies](../deploy/topologies.md) · [quickstart-docker](../deploy/quickstart-docker.md) · [homelab](../deploy/homelab.md) · [aws](../deploy/aws.md) · [reverse-proxies](../deploy/reverse-proxies.md) · [certificates](../deploy/certificates.md) +```` + +- [ ] **Step 3: Re-run the content-check + link check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/adr/0011-deployment-topology.md \ + && grep -q 'Status:\*\* Proposed' docs/adr/0011-deployment-topology.md \ + && grep -q 'dns-persist-01' docs/adr/0011-deployment-topology.md \ + && grep -q '307' docs/adr/0011-deployment-topology.md \ + && grep -q 'on-demand TLS' docs/adr/0011-deployment-topology.md \ + && echo CHECK-PASS || echo CHECK-FAIL +# Relative links resolve (the ../deploy/ links will report BROKEN until Tasks 2–7 land — +# acceptable inside this slice; re-run in the final sweep where all must resolve): +grep -oE '\]\([^)#]+\)' docs/adr/0011-deployment-topology.md | sed 's/](//;s/)//' \ + | grep -v '^http' | while read -r p; do [ -e "docs/adr/$p" ] || echo "BROKEN: $p"; done +``` + +Expected: `CHECK-PASS`; link check reports only `../deploy/*.md` as BROKEN if Tasks 2–7 +haven't run yet (nothing else). + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add docs/adr/0011-deployment-topology.md +git commit -s -m "docs(adr): ADR-0011 deployment topology — one binary, three blessed shapes (Proposed) (slice-G) + +Drafted from the 2026-07-05 deployment-topology spec §2–§4: T1 solo / +T2 modular / T3 fleet (paper), the edge doctrine (bundled Caddy; rustls +Phase 1 now, Phase 2 behind four named triggers), the fleet cert story +(wildcard vs per-node; on-demand TLS rejected), node-addressed placement +via Valkey presence (routing tier + distinguished control node rejected), +and the rejected shapes (FOB decomposition, serverless, tunnels-in-prod, +NLB-TLS, k8s manifests). Status Proposed — ratification per repo process. +Closes the topology ADR reserved by ADR-0008's worked example." +``` + +--- + +### Task 2: `docs/deploy/topologies.md` — the three shapes + decision guide + +**Files:** +- Create: `docs/deploy/topologies.md` +- Test: bash content-check (Steps 1/3) + +**Interfaces:** +- Consumes: ADR-0011 (Task 1) as `../adr/0011-deployment-topology.md`; spec §2, §7, §8. +- Produces: the tree's entry point; Tasks 3–7 link to it as `topologies.md` (same dir). + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/topologies.md \ + && grep -q 'decision guide' docs/deploy/topologies.md \ + && grep -qi 'paper' docs/deploy/topologies.md \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write the complete file** + +Write `docs/deploy/topologies.md` with exactly this content: + +````markdown +# Deployment topologies — one binary, three blessed shapes + +The FOB never decomposes; topology is a configuration property, not a code property. The same +binary runs in every shape below; only `RUTSTER_*` env and what is deployed alongside it +differ. Ratified (Proposed) in [ADR-0011](../adr/0011-deployment-topology.md). + +## Decision guide + +| You are… | Use | Doc | +|---|---|---| +| Trying rutster for the first time; one box, one domain, a public IP | **T1 — Solo** | [quickstart-docker.md](quickstart-docker.md) | +| Running production and want independently upgradable parts | **T2 — Modular** (the reference deployment) | [quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack) | +| Behind CGNAT / no public IP (homelab) | T1/T2 + tunnel (dev, **PSTN-only**) or VPS forwarder (production) | [homelab.md](homelab.md) | +| Already running nginx / HAProxy / Traefik | T2 minus the `caddy` service | [reverse-proxies.md](reverse-proxies.md) | +| On EC2 and want ALB + ACM to terminate TLS | T2/T3 behind an ALB | [aws.md](aws.md) | +| Outgrowing one node | **T3 — Fleet** — **paper only today** | [ADR-0011](../adr/0011-deployment-topology.md) | + +## T1 — Solo (all-in-one container) + +One image, `rutster-allinone`, with s6-overlay supervising four processes: + +| Process | Role | Binding | +|---|---|---| +| `caddy` | Edge: ACME issuance/renewal, TLS termination, WS proxy | `:443` / `:80` public | +| `rutster` (the FOB) | Engine — signaling, trunk webhook, media, reflexes | `127.0.0.1:8080` behind Caddy; **media UDP direct** | +| `rutster-brain-realtime` | Brain | `127.0.0.1:8082` loopback tap | +| `valkey-server` | Event stream (`EventSink`); ledger/directory later | `127.0.0.1:6379` | + +Two volumes, **both non-negotiable**: + +- `/data` — Caddy's cert/ACME state. Losing it can lock your domain out of Let's Encrypt for + up to a week (duplicate-certificate limit) — a **total inbound outage**. See + [certificates.md](certificates.md). +- `/var/lib/valkey` — stream/state persistence. + +WebRTC media (UDP) goes **direct to the FOB, never through Caddy**: run with host networking, +or publish `RUTSTER_MEDIA_PORT_RANGE` and set `RUTSTER_MEDIA_ADVERTISED_IP` to your public IP +(NAT 1:1 model — no STUN). + +Copy-paste path from zero to first call: [quickstart-docker.md](quickstart-docker.md). + +## T2 — Modular (compose stack) — the reference deployment + +Four services in `deploy/compose.yaml`: + +| Service | Image | Notes | +|---|---|---| +| `caddy` | `rutster-edge` | Same custom Caddy build + same Caddyfile as T1 | +| `engine` | `rutster-engine` | The FOB; plaintext `:8080` on the compose network; media UDP published | +| `brain` | `rutster-brain` | `network_mode: "service:engine"` — shares the engine's netns so the loopback-only tap posture survives unchanged | +| `valkey` | `valkey/valkey` (upstream) | Own service; network-near | + +Operational contract: + +- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` — grace **must** + exceed drain, or the orchestrator kills calls the engine was gracefully draining. +- Bring-your-own-proxy: disable the `caddy` service; the engine keeps its plaintext `:8080` + mode. Tuned-timeout configs: [reverse-proxies.md](reverse-proxies.md). +- Upgrades: `docker compose pull && docker compose up -d` — the drain window lets in-flight + calls finish (calls are non-migratable; a killed engine kills its calls, honestly stated). + +Copy-paste path from zero to first call: +[quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack). + +## T3 — Fleet (paper only — no shipped artifacts in this epoch) + +**Honest statement: you cannot deploy T3 today by copy-paste; there are no fleet artifacts +yet.** The design is ratified on paper in +[ADR-0011](../adr/0011-deployment-topology.md) so single-node deployments grow toward it, not +away from it: + +- N symmetric nodes (the T1 process set minus valkey) + a shared green zone: one Valkey + (presence + directory + spend ledger), object storage. +- Wildcard DNS `*.pbx.domain`; per-node certs — the traps live in + [certificates.md](certificates.md). +- Answer-time placement via Valkey presence; node-addressed `wss://node-K.pbx.domain/...` + Stream URLs; no router tier. +- Per-node admission cap = the benchmark number (`RUTSTER_MAX_SESSIONS`, default 64, is a + placeholder until then). +- Scale-in is drain-then-terminate only. No spot instances for the engine tier. + +## What is deliberately not supported + +- **Splitting the FOB into services** — permanent rejection, not a roadmap item. +- **Serverless** — non-migratable multi-hour calls, per-session UDP sockets, in-process media + state. Categorically unfit. +- **Tunnels in production** — plaintext audio at the tunnel vendor's edge + documented + mid-call WS kills. Dev/demo only; the whole story is in [homelab.md](homelab.md). +- **NLB TLS listeners** — see [aws.md](aws.md) for the 350-second reason. +- **k8s manifests/Helm** — compose-first. If you run k8s anyway: set + `terminationGracePeriodSeconds` above `RUTSTER_DRAIN_DEADLINE_SECS`, same grace-exceeds-drain + rule as compose. + +## Ports & volumes reference (all shapes) + +| Surface | Where | Notes | +|---|---|---| +| `:80` / `:443` TCP | Caddy, public | ACME HTTP-01 + TLS + all HTTPS/WSS traffic | +| `:8080` TCP | FOB, behind the edge | Signaling + `/v1/trunk/webhook` + `/twilio/media-stream` WS — one listener (`RUTSTER_HTTP_BIND`) | +| `RUTSTER_MEDIA_PORT_RANGE` UDP | FOB, **public, direct** | WebRTC media; never proxied | +| `:8082` TCP | Brain, loopback only | The tap; non-loopback rejected until step 6 | +| `:6379` TCP | Valkey, private | Loopback (T1) / compose network (T2) / green zone (T3) | +| `:9090` TCP | FOB `/metrics`, internal | `RUTSTER_METRICS_BIND`; never routed through Caddy | +| `/data` volume | Caddy | Cert/ACME state — loss class: total inbound outage | +| `/var/lib/valkey` volume | Valkey | Event stream persistence | +```` + +- [ ] **Step 3: Re-run the content-check + link check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/topologies.md \ + && grep -qi 'decision guide' docs/deploy/topologies.md \ + && grep -q 'paper only' docs/deploy/topologies.md \ + && grep -q 'cannot deploy T3 today' docs/deploy/topologies.md \ + && echo CHECK-PASS || echo CHECK-FAIL +grep -oE '\]\([^)#]+\)' docs/deploy/topologies.md | sed 's/](//;s/)//' \ + | grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done +``` + +Expected: `CHECK-PASS`; BROKEN lines only for sibling docs whose tasks haven't run yet. + +- [ ] **Step 4: Cross-check against slice B (if landed)** + +```bash +cd /home/alee/Sources/rutster +ls deploy/ 2>/dev/null && grep -n 'image:' deploy/compose.yaml +``` + +If `deploy/` exists and any image/service name differs from this doc's tables, fix the doc to +match reality before committing. If `deploy/` does not exist, the doc stands on spec §6.1 +names — no change. + +- [ ] **Step 5: Commit** + +```bash +cd /home/alee/Sources/rutster +git add docs/deploy/topologies.md +git commit -s -m "docs(deploy): topologies.md — three blessed shapes + decision guide (slice-G) + +The docs/deploy/ tree entry point. T1/T2 process+service tables, the +grace-exceeds-drain rule, the honest T3 statement (paper only, no +copy-paste deploy exists), the not-supported list, and the shared +ports/volumes reference. Acceptance bar: copy-paste paths delegate to +quickstart-docker.md; T3 states plainly what cannot be done today." +``` + +--- + +### Task 3: `docs/deploy/quickstart-docker.md` — zero to first call + +**Files:** +- Create: `docs/deploy/quickstart-docker.md` +- Test: bash content-check (Steps 1/3) + +**Interfaces:** +- Consumes: spec §2.1/§2.2/§5.7/§6, image names from spec §6.1, env vars grounded in + `crates/rutster/src/config.rs` (the four `RUTSTER_TWILIO_*` vars are all-or-none — the + parser rejects partial config), routes grounded in `crates/rutster/src/routes.rs` + (`/healthz`, `/readyz`, `/v1/trunk/webhook`). +- Produces: the file `docs/QUICKSTART.md` links to in Task 8 (exact path + `docs/deploy/quickstart-docker.md`). + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/quickstart-docker.md \ + && grep -q 'rutster-allinone' docs/deploy/quickstart-docker.md \ + && grep -q 'docker compose up' docs/deploy/quickstart-docker.md \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write the complete file** + +Write `docs/deploy/quickstart-docker.md` with exactly this content: + +````markdown +# Docker quickstart — zero to first call + +Two shapes, one binary ([topologies.md](topologies.md)). T1 is one `docker run`; T2 is the +reference `compose up`. Both end the same way: a browser call at your domain, and (with Twilio +credentials) a real phone call. + +> **Artifact names note:** images are published to the `git.adlee.work/alee/` registry +> namespace by the release CI (plan B / slice F settled this). If your Gitea registry uses a +> different owner segment, substitute accordingly — check `deploy/compose.yaml` in your +> checkout, which is authoritative. + +## Prerequisites (both shapes) + +1. A Linux host with Docker, a **public IPv4**, and inbound `80/tcp`, `443/tcp`, and your + chosen media UDP range open. (No public IP? → [homelab.md](homelab.md).) +2. A domain with an A record pointing at that host, e.g. `pbx.example.com`. Certificates are + issued automatically via ACME — no self-signed path exists, the CPaaS refuses it + ([certificates.md](certificates.md)). +3. For the phone call: a [Twilio](https://www.twilio.com/) account + a Voice-capable number + (trial works). + +## T1 — Solo (all-in-one) + +```bash +docker run -d --name rutster --restart unless-stopped \ + --network host \ + -v rutster-caddy-data:/data \ + -v rutster-valkey:/var/lib/valkey \ + -e RUTSTER_DOMAIN=pbx.example.com \ + -e RUTSTER_ACME_EMAIL=you@example.com \ + -e RUTSTER_MEDIA_ADVERTISED_IP=203.0.113.7 \ + -e RUTSTER_MEDIA_PORT_RANGE=49152-49407 \ + -e RUTSTER_TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \ + -e RUTSTER_TWILIO_AUTH_TOKEN=your_auth_token \ + -e RUTSTER_TWILIO_MEDIA_BIND=0.0.0.0:8081 \ + -e RUTSTER_TWILIO_WEBHOOK_BASE=https://pbx.example.com \ + git.adlee.work/alee/rutster-allinone:latest +``` + +Notes: + +- `--network host` is the recommended mode: WebRTC media UDP goes **direct to the engine**, + never through Caddy. If you can't use host networking, publish `-p 80:80 -p 443:443 + -p 49152-49407:49152-49407/udp` instead and keep `RUTSTER_MEDIA_ADVERTISED_IP` = the host's + public IP (NAT 1:1, no STUN). +- The two volumes are **non-negotiable**. `/data` in particular: recreating the container + without it re-requests certificates and can hit Let's Encrypt's duplicate-certificate + limit — a week-long total inbound outage ([certificates.md](certificates.md)). +- The four `RUTSTER_TWILIO_*` vars are all-or-none — the engine refuses partial Twilio config + at startup. Omit all four to run WebRTC-only. +- `RUTSTER_DOMAIN` / `RUTSTER_ACME_EMAIL` feed the bundled Caddyfile. Domains that can't do + HTTP-01 (no port 80, wildcard needs) use DNS-01 — [certificates.md](certificates.md). + +Verify it's up: + +```bash +curl -fsS https://pbx.example.com/healthz && echo OK # liveness +curl -fsS https://pbx.example.com/readyz && echo READY # can accept a new call +``` + +### First call (browser, WebRTC) + +Open `https://pbx.example.com/`, click **Start call**, grant the microphone. You're on a call +with the engine. + +### First call (phone, PSTN) + +1. Twilio Console → Phone Numbers → your number → **A call comes in** → Webhook, + `POST https://pbx.example.com/v1/trunk/webhook`. +2. Dial your Twilio number from any phone. Twilio hits the webhook, the engine answers with + TwiML pointing Twilio's Media Streams at `wss://pbx.example.com/twilio/media-stream`, and + the call's audio enters the same reflex loop a WebRTC call uses. + +## T2 — Modular (compose stack) + +```bash +git clone https://git.adlee.work/alee/rutster.git +cd rutster/deploy +cp .env.example .env +$EDITOR .env # set DOMAIN, ACME email, RUTSTER_MEDIA_ADVERTISED_IP, RUTSTER_TWILIO_* +docker compose up -d +``` + +What comes up: `caddy` (edge, `rutster-edge`), `engine` (`rutster-engine`), `brain` +(`rutster-brain`, sharing the engine's network namespace), `valkey` (upstream image). The +same two first-call paths as T1 apply verbatim. + +Operational notes: + +- `.env.example` ships `RUTSTER_DRAIN_DEADLINE_SECS=600` and the stack sets + `stop_grace_period: 660s` — grace exceeds drain, so `docker compose down` lets in-flight + calls finish (up to 10 minutes) instead of killing them. +- Upgrade: `docker compose pull && docker compose up -d`. +- Bring your own proxy instead of Caddy: [reverse-proxies.md](reverse-proxies.md). + +## Troubleshooting + +| Symptom | Cause / fix | +|---|---| +| Twilio error **31910** on calls | Your cert isn't publicly trusted or expired — check `/data` volume survived, see [certificates.md](certificates.md) ([twilio.com/docs/api/errors/31910](https://www.twilio.com/docs/api/errors/31910)) | +| Webhook signature validation fails | The edge isn't forwarding `X-Forwarded-Proto/Host` honestly, or `RUTSTER_TRUSTED_PROXIES` doesn't include your proxy — the engine ignores forwarded headers from unlisted sources | +| Browser call connects, no audio | Media UDP blocked or `RUTSTER_MEDIA_ADVERTISED_IP` wrong — the SDP advertises that IP; callers send UDP straight to it | +| `curl /readyz` returns 503 | Node is draining or at the admission cap (`RUTSTER_MAX_SESSIONS`) — liveness (`/healthz`) stays 200 | +| Let's Encrypt rate-limit errors in Caddy logs | You recreated the container without the `/data` volume — [certificates.md](certificates.md) | +```` + +- [ ] **Step 3: Re-run the content-check + link check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/quickstart-docker.md \ + && grep -q 'rutster-allinone' docs/deploy/quickstart-docker.md \ + && grep -q 'docker compose up' docs/deploy/quickstart-docker.md \ + && grep -q '/v1/trunk/webhook' docs/deploy/quickstart-docker.md \ + && grep -q 'all-or-none' docs/deploy/quickstart-docker.md \ + && echo CHECK-PASS || echo CHECK-FAIL +grep -oE '\]\([^)#]+\)' docs/deploy/quickstart-docker.md | sed 's/](//;s/)//' \ + | grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done +``` + +Expected: `CHECK-PASS`; BROKEN only for sibling docs not yet written. + +- [ ] **Step 4: Cross-check against slice B (if landed)** + +```bash +cd /home/alee/Sources/rutster +ls deploy/ 2>/dev/null && { grep -n 'image:' deploy/compose.yaml; grep -n '^[A-Z]' deploy/.env.example; } +``` + +Reconcile: image names, the `RUTSTER_DOMAIN`/`RUTSTER_ACME_EMAIL` variable names (these two +are artifact-level names owned by slice B's Caddyfile — the spec does not fix them), the +compose service names, and the git clone URL. If `deploy/` does not exist yet, keep the doc as +written and leave the artifact-names note box in place. + +- [ ] **Step 5: Commit** + +```bash +cd /home/alee/Sources/rutster +git add docs/deploy/quickstart-docker.md +git commit -s -m "docs(deploy): quickstart-docker.md — T1 docker run + T2 compose, zero to first call (slice-G) + +The acceptance-bar doc: a copy-paste path from a bare Linux host with a +domain to (1) a browser WebRTC call and (2) a real PSTN call via the +Twilio webhook. Volumes marked non-negotiable with the LE-lockout reason; +all-or-none RUTSTER_TWILIO_* contract stated (matches config.rs); host +networking recommended for direct media UDP." +``` + +--- + +### Task 4: `docs/deploy/homelab.md` — the CGNAT truth + +**Files:** +- Create: `docs/deploy/homelab.md` +- Test: bash content-check (Steps 1/3) + +**Interfaces:** +- Consumes: spec §3.5, brief §3(d) + §2 invariant 9 + §5 risks 3/6, the brief's URLs + (cloudflared #1465/#1282, livekit/agents #3379). +- Produces: the doc `topologies.md` and ADR-0011 link as `homelab.md` / + `../deploy/homelab.md`. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/homelab.md \ + && grep -q 'PSTN-only' docs/deploy/homelab.md \ + && grep -q 'WireGuard' docs/deploy/homelab.md \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write the complete file** + +Write `docs/deploy/homelab.md` with exactly this content: + +````markdown +# Homelab & CGNAT — the honest story + +**The truth first: no tunnel carries inbound UDP. WebRTC callers are unreachable behind any +tunnel, so a homelab behind CGNAT is PSTN-only, period.** The engine's WebRTC media is +UDP-direct to an advertised IP (no STUN, no TURN); the CPaaS side offers no relay or +rendezvous either, and intermediaries in the media path are a documented frame-dropping hazard +([livekit/agents#3379](https://github.com/livekit/agents/issues/3379)). Nothing on this page +makes CGNAT production-grade for free. Three tiers, worst to best: + +## Tier 1 — dev/demo: ngrok (the blessed 5-minute path) + +ngrok is the **only** tunnel with a proven Twilio Media Streams record. Do **not** use +Cloudflare Tunnel even for dev: cloudflared has an open Twilio 31920 handshake bug +([cloudflared#1465](https://github.com/cloudflare/cloudflared/issues/1465)), recurring 1006 +closures ([cloudflared#1282](https://github.com/cloudflare/cloudflared/issues/1282)), and +Cloudflare documents killing WebSockets on edge code releases. + +```bash +# 1. Run the engine, plaintext :8080 (no Caddy needed — ngrok terminates TLS): +docker run -d --name rutster-engine --network host \ + -e RUTSTER_TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \ + -e RUTSTER_TWILIO_AUTH_TOKEN=your_auth_token \ + -e RUTSTER_TWILIO_MEDIA_BIND=0.0.0.0:8081 \ + -e RUTSTER_TWILIO_WEBHOOK_BASE=https://REPLACE-ME.ngrok-free.app \ + -e RUTSTER_TRUSTED_PROXIES=127.0.0.1/32 \ + git.adlee.work/alee/rutster-engine:latest +# (or from source: cargo run — see docs/QUICKSTART.md) + +# 2. Tunnel it: +ngrok http 8080 +# 3. Put the printed https://xxxx.ngrok-free.app URL into RUTSTER_TWILIO_WEBHOOK_BASE +# (restart the container), and into the Twilio number's webhook: +# POST https://xxxx.ngrok-free.app/v1/trunk/webhook +# 4. Dial your Twilio number. +``` + +**Free-tier arithmetic** (why this is a demo, not a deployment — research: [TLS brief §3(d)](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)): a Media Streams call is +8 kHz µ-law — 8 kB/s of audio per direction, base64-encoded inside a JSON envelope at 50 +messages/s, both directions ≈ **~140 MB per call-hour** on the wire. ngrok's free 1 GB/month +data cap ([ngrok pricing](https://ngrok.com/pricing)) therefore buys roughly **seven call-hours a month**. And the audio transits ngrok's +edge **in plaintext** (they terminate TLS) — an unconsented subprocessor, which is a +DPA/BAA/PCI failure. Dev only. Never production. + +## Tier 2 — single-user demo: Tailscale Funnel + +Privacy-clean variant: TLS terminates **on your node**, so Funnel relays ciphertext it cannot +read. + +```bash +tailscale funnel 8080 +# webhook base = https://..ts.net +``` + +Still PSTN-only (no inbound UDP), bandwidth cap undisclosed, one user. Unsizable beyond a +personal demo. + +## Tier 3 — production graduation: cheap VPS + WireGuard, TLS at home + +The recommended path. The VPS is a **dumb layer-4 forwarder**: TLS terminates at home (your +Caddy, DNS-01 certs), so the forwarder **physically cannot read the audio** — the strongest +privacy topology available. Bonus: forwarding the media UDP range over the same tunnel +restores WebRTC, which no tunnel product can do. Cost: one small VPS (~$5/mo) and one extra +network hop of media latency — pick a VPS region near home. + +Point DNS at the VPS: `pbx.example.com A `. + +**VPS `/etc/wireguard/wg0.conf`:** + +```ini +[Interface] +Address = 10.88.0.1/24 +ListenPort = 51820 +PrivateKey = + +[Peer] +PublicKey = +AllowedIPs = 10.88.0.2/32 +``` + +**Home box `/etc/wireguard/wg0.conf`** (home initiates — CGNAT-friendly; the keepalive holds +the NAT mapping): + +```ini +[Interface] +Address = 10.88.0.2/24 +PrivateKey = + +[Peer] +PublicKey = +Endpoint = :51820 +AllowedIPs = 10.88.0.0/24 +PersistentKeepalive = 25 +``` + +**VPS forwarding** (`sysctl -w net.ipv4.ip_forward=1`, persist it, then nftables): + +```nft +table ip rutster-fwd { + chain prerouting { + type nat hook prerouting priority dstnat; + iifname "eth0" tcp dport { 80, 443 } dnat to 10.88.0.2 + iifname "eth0" udp dport 49152-49407 dnat to 10.88.0.2 + } + chain postrouting { + type nat hook postrouting priority srcnat; + oifname "wg0" masquerade + } +} +``` + +**Home side:** run T1 or T2 exactly per [quickstart-docker.md](quickstart-docker.md), with: + +- `RUTSTER_MEDIA_ADVERTISED_IP=` — callers send media UDP to the VPS; the DNAT + delivers it home through the tunnel. +- `RUTSTER_MEDIA_PORT_RANGE=49152-49407` (must match the nftables rule). +- Certificates: DNS-01 is the robust choice here ([certificates.md](certificates.md)); + HTTP-01 also works since `:80` is forwarded. + +First call: same two paths as [quickstart-docker.md](quickstart-docker.md) — browser at +`https://pbx.example.com/` (WebRTC now works — the UDP range rides the tunnel) and the Twilio +webhook at `https://pbx.example.com/v1/trunk/webhook`. + +**Or skip the tunnel entirely:** run the engine *on* the VPS (that is just +[T1](quickstart-docker.md) on rented hardware). You trade at-home media for zero forwarding +complexity. + +## Explicitly unsupported for production + +Cloudflare Tunnel or ngrok in the live audio path: plaintext audio at the vendor edge +(unconsented subprocessor — DPA/BAA/PCI failure), documented mid-call WS terminations, zero +SLA, and Cloudflare's discretionary "disproportionate audio" ToS clause aimed at exactly this +traffic profile (research basis: [TLS brief §4](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)). Ratified in [ADR-0011](../adr/0011-deployment-topology.md). +```` + +- [ ] **Step 3: Re-run the content-check + link check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/homelab.md \ + && grep -q 'PSTN-only, period' docs/deploy/homelab.md \ + && grep -q 'seven call-hours' docs/deploy/homelab.md \ + && grep -q 'Tailscale Funnel' docs/deploy/homelab.md \ + && grep -q 'physically cannot read the audio' docs/deploy/homelab.md \ + && grep -q 'cloudflared/issues/1465' docs/deploy/homelab.md \ + && echo CHECK-PASS || echo CHECK-FAIL +grep -oE '\]\([^)#]+\)' docs/deploy/homelab.md | sed 's/](//;s/)//' \ + | grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done +``` + +Expected: `CHECK-PASS`; BROKEN only for sibling docs not yet written. + +- [ ] **Step 4: Cross-check against slice B (if landed)** — same command as Task 3 Step 4; + reconcile the `rutster-engine` image name and env-var names if `deploy/` exists. + +- [ ] **Step 5: Commit** + +```bash +cd /home/alee/Sources/rutster +git add docs/deploy/homelab.md +git commit -s -m "docs(deploy): homelab.md — CGNAT truth: PSTN-only behind tunnels, VPS+WireGuard graduation (slice-G) + +Acceptance bar honored by honesty: the doc leads with what cannot work +(no tunnel carries inbound UDP -> WebRTC unreachable behind CGNAT). Three +tiers: ngrok blessed dev path with the free-tier arithmetic (~140 MB/ +call-hour -> ~7 call-hours/month), Tailscale Funnel single-user demo +(ciphertext-only), and the VPS+WireGuard graduation with TLS terminating +at home (forwarder physically cannot read audio; media UDP DNAT restores +WebRTC). cloudflared explicitly excluded even for dev (#1465, #1282)." +``` + +--- + +### Task 5: `docs/deploy/aws.md` — the ALB appendix + +**Files:** +- Create: `docs/deploy/aws.md` +- Test: bash content-check (Steps 1/3) + +**Interfaces:** +- Consumes: spec §3.5 (AWS appendix bullet), brief §3(c) + §4 (ALB row, NLB rejection). +- Produces: the doc `topologies.md` and ADR-0011 link as `aws.md` / `../deploy/aws.md`. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/aws.md \ + && grep -q 'idle_timeout' docs/deploy/aws.md \ + && grep -q '350' docs/deploy/aws.md \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write the complete file** + +Write `docs/deploy/aws.md` with exactly this content: + +````markdown +# AWS appendix — ALB + ACM (docs only, never a shipped artifact) + +Scope: operators **already on EC2**. There, an ALB is trust-neutral (the engine is on AWS +infrastructure regardless) and ACM renewal is genuinely zero-touch. This page is an appendix, +not a recommendation to move to AWS — the shipped edges are Caddy +([topologies.md](topologies.md)) and BYO proxies ([reverse-proxies.md](reverse-proxies.md)). +All ALB/NLB facts below are grounded in the [TLS/edge decision brief §3(c)](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md). + +## Never NLB-TLS + +Do not put the engine behind an NLB TLS listener. Two disqualifiers, ratified in +[ADR-0011](../adr/0011-deployment-topology.md): + +1. **Fixed 350-second idle timeout** that silently drops flows — any quiet WebSocket (hold, + parked call, unidirectional stream) dies mid-call with no close frame. +2. **No HTTP context**: an NLB adds no `X-Forwarded-Proto/Host`, so the engine cannot + reconstruct the public URL that `X-Twilio-Signature` is HMAC'd over + ([Twilio's SSL-termination guidance](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library)) + — genuine traffic fails validation. + +## The three mandatory ALB attribute overrides + +The defaults kill telephony. All three overrides are **mandatory**, not tuning: + +| Attribute | Value | Why | +|---|---|---| +| `idle_timeout.timeout_seconds` | `4000` | The default 60 s kills any quiet WS. 4000 is the ALB maximum — still finite, so the engine's app-level WS pings (`RUTSTER_WS_PING_SECS`, default 20) remain load-bearing here. | +| `client_keep_alive.seconds` | `604800` | The 3600 s default can terminate the client connection under a **>1-hour call**. Set to the 7-day maximum. | +| `routing.http.preserve_host_header.enabled` | `true` | Signature validation reconstructs the URL *as Twilio saw it* — the engine must receive the honest public `Host`, and its `RUTSTER_TRUSTED_PROXIES` must include the ALB's subnet CIDRs. | + +## Copy-paste path (CLI) + +Assumes: engine node(s) running per [quickstart-docker.md](quickstart-docker.md) with the +`caddy` service disabled (plaintext `:8080`), a VPC, two subnets, and a security group +allowing `:443` from anywhere and `:8080` from the ALB. + +```bash +VPC=vpc-0123456789abcdef0 +SUBNETS="subnet-0aaa subnet-0bbb" +SG=sg-0ccc + +ALB_ARN=$(aws elbv2 create-load-balancer --name rutster-edge --type application \ + --subnets $SUBNETS --security-groups $SG \ + --query 'LoadBalancers[0].LoadBalancerArn' --output text) + +# THE THREE MANDATORY OVERRIDES — do not skip: +aws elbv2 modify-load-balancer-attributes --load-balancer-arn "$ALB_ARN" --attributes \ + Key=idle_timeout.timeout_seconds,Value=4000 \ + Key=client_keep_alive.seconds,Value=604800 \ + Key=routing.http.preserve_host_header.enabled,Value=true + +TG_ARN=$(aws elbv2 create-target-group --name rutster-node-1 \ + --protocol HTTP --port 8080 --vpc-id "$VPC" \ + --health-check-path /readyz \ + --query 'TargetGroups[0].TargetGroupArn' --output text) +aws elbv2 register-targets --target-group-arn "$TG_ARN" --targets Id=i-0node1 + +CERT_ARN=$(aws acm request-certificate --domain-name '*.pbx.example.com' \ + --validation-method DNS --query CertificateArn --output text) +# Create the DNS validation CNAME ACM reports, wait for ISSUED, then: + +LISTENER_ARN=$(aws elbv2 create-listener --load-balancer-arn "$ALB_ARN" \ + --protocol HTTPS --port 443 \ + --ssl-policy ELBSecurityPolicy-TLS13-1-2-2021-06 \ + --certificates CertificateArn="$CERT_ARN" \ + --default-actions Type=forward,TargetGroupArn="$TG_ARN" \ + --query 'Listeners[0].ListenerArn' --output text) + +# Fleet growth: one target group + one host-header rule per node +# (node-addressed placement — ADR-0011): +aws elbv2 create-rule --listener-arn "$LISTENER_ARN" --priority 10 \ + --conditions Field=host-header,Values=node-1.pbx.example.com \ + --actions Type=forward,TargetGroupArn="$TG_ARN" +``` + +Point `*.pbx.example.com` (alias record) at the ALB. First call: identical to +[quickstart-docker.md](quickstart-docker.md) — browser at your domain, Twilio webhook at +`https://pbx.example.com/v1/trunk/webhook`. + +The `--ssl-policy` above serves TLS 1.2+1.3 — never pick a 1.3-only policy (Twilio's TLS 1.3 +client support is undocumented; see the invariants in +[certificates.md](certificates.md)). + +## WebRTC media does not ride the ALB + +The ALB carries HTTPS/WSS only. WebRTC media UDP goes **direct to each node**: give every +engine node an Elastic IP, open the media UDP range in the node's security group, and set +`RUTSTER_MEDIA_ADVERTISED_IP` to that EIP. + +## Caps and traps + +- **Hard cap: 100 target groups per ALB.** With one TG per node (host-header routing), that + is your fleet ceiling per ALB. +- **AWS fleet rotation can still cut a multi-hour call** — ALB infrastructure is replaced + under maintenance without regard for your WS lifetimes. Accepted residual risk; there is no + attribute for it. +- Health checks target `/readyz` (503 while draining or at the admission cap) — scale-in must + be drain-then-terminate: deregister the target, wait out + `RUTSTER_DRAIN_DEADLINE_SECS`, then terminate the instance. +```` + +- [ ] **Step 3: Re-run the content-check + link check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/aws.md \ + && grep -q 'idle_timeout.timeout_seconds' docs/deploy/aws.md \ + && grep -q '604800' docs/deploy/aws.md \ + && grep -q 'preserve_host_header' docs/deploy/aws.md \ + && grep -q '100 target groups' docs/deploy/aws.md \ + && grep -q '350-second' docs/deploy/aws.md \ + && echo CHECK-PASS || echo CHECK-FAIL +grep -oE '\]\([^)#]+\)' docs/deploy/aws.md | sed 's/](//;s/)//' \ + | grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done +``` + +Expected: `CHECK-PASS`; BROKEN only for sibling docs not yet written. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add docs/deploy/aws.md +git commit -s -m "docs(deploy): aws.md — ALB appendix: three mandatory overrides, never NLB-TLS (slice-G) + +Docs-only appendix per ADR-0011 (never a shipped artifact). The three +mandatory ALB attribute overrides (idle_timeout=4000, +client_keep_alive=604800, preserve_host_header=true) with the telephony +reasons; the never-NLB-TLS rule with the 350s-silent-drop + no-HTTP- +context rationale; 100-target-group fleet ceiling; per-node EIPs for +direct media UDP; copy-paste aws elbv2 CLI path to first call." +``` + +--- + +### Task 6: `docs/deploy/reverse-proxies.md` — BYO nginx/HAProxy/Traefik + +**Files:** +- Create: `docs/deploy/reverse-proxies.md` +- Test: bash content-check (Steps 1/3) + +**Interfaces:** +- Consumes: spec §2.2 (BYO-proxy seam) + §10, brief §3(b) + §4 (runner-up table: HAProxy + `timeout tunnel`/#2280/`hard-stop-after`; Traefik pin-exact-versions #10601 / v2.11.2 / + #11405; nginx not-recommended rationale). +- Produces: the doc `topologies.md`/`quickstart-docker.md` link as `reverse-proxies.md`. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/reverse-proxies.md \ + && grep -q 'HAProxy' docs/deploy/reverse-proxies.md \ + && grep -qi 'pin' docs/deploy/reverse-proxies.md \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write the complete file** + +Write `docs/deploy/reverse-proxies.md` with exactly this content: + +````markdown +# Bring your own reverse proxy — tuned-timeout configs + +Supported: disable the `caddy` service in the compose stack +([quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack)); the engine keeps its +plaintext `:8080` listener. Everything below terminates TLS in *your* proxy and forwards to +the engine. + +## The universal 60-second footgun + +**nginx, HAProxy, and Traefik all default their idle/read timeouts to ~60–180 seconds — every +one of them will kill a quiet, perfectly healthy call WebSocket.** A Media Streams WS can +legitimately be near-silent in one direction for hours, and neither Twilio nor Telnyx sends +protocol-level keepalives. The engine ships app-level WS pings (`RUTSTER_WS_PING_SECS`, +default 20) as belt-and-braces, but your proxy timeouts must still exceed the maximum call +duration. Every snippet below does that. + +## The contract any proxy must honor + +1. **Publicly-trusted cert**, auto-renewed — no self-signed path exists + ([certificates.md](certificates.md)). +2. **WS upgrade with no frame buffering and no connection-lifetime cap** — 20 ms audio frames, + calls up to 24 h. +3. **Honest `X-Forwarded-Proto` and `X-Forwarded-Host`** (including on the WS upgrade + request): `X-Twilio-Signature` is HMAC over the URL as Twilio saw it + ([Twilio's SSL-termination guidance](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library)). +4. **Engine-side trust**: set `RUTSTER_TRUSTED_PROXIES` to your proxy's source IP/CIDR — + forwarded headers from unlisted sources are ignored, and signature validation will fail. +5. **TLS 1.2+1.3, mainstream ECDHE — never 1.3-only** (Twilio's 1.3 client support is + undocumented). + +## nginx (works if you insist — not a recommended edge) + +Why not recommended: no native DNS-01/wildcard support, and cert renewal requires a reload +whose old workers linger against hours-long WS connections (research basis: [TLS brief §4](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)). rutster ships this snippet and +nothing more for nginx. + +```nginx +map $http_upgrade $connection_upgrade { default upgrade; '' close; } + +server { + listen 443 ssl; + server_name pbx.example.com; + ssl_certificate /etc/letsencrypt/live/pbx.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/pbx.example.com/privkey.pem; + ssl_protocols TLSv1.2 TLSv1.3; # never 1.3-only + + location / { + proxy_pass http://127.0.0.1:8080; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_buffering off; # 20 ms frames must not be coalesced + proxy_read_timeout 86400s; # the 60s default is the footgun + proxy_send_timeout 86400s; + } +} +``` + +## HAProxy (3.2 LTS or newer) + +The best WS mental model of the classic family: after the upgrade, only `timeout tunnel` +governs the connection — **but only if you set it**; put it in `defaults` or it silently +doesn't apply to the paths you think it does (see +[haproxy#2280](https://github.com/haproxy/haproxy/issues/2280)). Two extra notes: `acme.sh` +can hot-load renewed certs through the stats socket with zero reload; and `hard-stop-after` +kills calls that are still draining — leave it unset or above your drain deadline. + +```haproxy +defaults + mode http + timeout connect 5s + timeout client 75s + timeout server 75s + timeout tunnel 24h # governs upgraded WS; unset = the 60s-class footgun + +frontend fe_rutster + bind :443 ssl crt /etc/haproxy/certs/ ssl-min-ver TLSv1.2 + http-request set-header X-Forwarded-Proto https + http-request set-header X-Forwarded-Host %[req.hdr(Host)] + default_backend be_rutster + +backend be_rutster + server fob 127.0.0.1:8080 +``` + +## Traefik v3 — PIN THE EXACT VERSION + +Traefik has the best built-in DNS-01/wildcard of the classic family **and a track record of +breaking WebSockets in patch releases**: +[#10601](https://github.com/traefik/traefik/issues/10601), the v2.11.2 timeout-behavior flip, +and [#11405](https://github.com/traefik/traefik/issues/11405). **Pin the exact image tag and +never auto-update the edge.** + +```yaml +# traefik.yml (static config) +entryPoints: + websecure: + address: ":443" + transport: + respondingTimeouts: + readTimeout: 0 # 0 disables; the 60s-class default kills quiet WS + idleTimeout: 0 + writeTimeout: 0 +certificatesResolvers: + le: + acme: + email: you@example.com + storage: /data/acme.json # persist this volume — certificates.md + dnsChallenge: + provider: cloudflare +``` + +```yaml +# dynamic config +http: + routers: + rutster: + rule: "Host(`pbx.example.com`)" + entryPoints: [websecure] + service: rutster + tls: { certResolver: le } + services: + rutster: + loadBalancer: + servers: + - url: "http://127.0.0.1:8080" +``` + +Traefik forwards `X-Forwarded-Proto/Host` by default — you still must list its address in +`RUTSTER_TRUSTED_PROXIES`. + +## First call + +Any of the above in front of the engine, then the same two first-call paths as +[quickstart-docker.md](quickstart-docker.md): browser at `https://pbx.example.com/`, Twilio +webhook at `https://pbx.example.com/v1/trunk/webhook`. +```` + +- [ ] **Step 3: Re-run the content-check + link check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/reverse-proxies.md \ + && grep -q 'proxy_read_timeout 86400s' docs/deploy/reverse-proxies.md \ + && grep -q 'timeout tunnel' docs/deploy/reverse-proxies.md \ + && grep -q 'PIN THE EXACT VERSION' docs/deploy/reverse-proxies.md \ + && grep -q 'RUTSTER_TRUSTED_PROXIES' docs/deploy/reverse-proxies.md \ + && grep -q '60' docs/deploy/reverse-proxies.md \ + && echo CHECK-PASS || echo CHECK-FAIL +grep -oE '\]\([^)#]+\)' docs/deploy/reverse-proxies.md | sed 's/](//;s/)//' \ + | grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done +``` + +Expected: `CHECK-PASS`; BROKEN only for sibling docs not yet written. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add docs/deploy/reverse-proxies.md +git commit -s -m "docs(deploy): reverse-proxies.md — BYO nginx/HAProxy/Traefik tuned-timeout configs (slice-G) + +The universal 60s idle-timeout footgun stated up front; the five-point +contract any proxy must honor (incl. RUTSTER_TRUSTED_PROXIES + honest +X-Forwarded-* for signature validation); complete snippets: nginx +(explicitly not-recommended, snippet-only support), HAProxy (timeout +tunnel in defaults per #2280; hard-stop-after warning), Traefik (pin the +exact version — #10601 / v2.11.2 / #11405 WS-breaking patch history)." +``` + +--- + +### Task 7: `docs/deploy/certificates.md` — ACME paths, wildcard traps, `/data` + +**Files:** +- Create: `docs/deploy/certificates.md` +- Test: bash content-check (Steps 1/3) + +**Interfaces:** +- Consumes: spec §3.1–§3.4, brief §2 invariants 1/7/8, §3(a)/(c), §5 risks 1/2/5; brief URLs + (twilio security page, errors 31910/11237, caddy #6420/#7222). +- Produces: the doc every other tree file links as `certificates.md`. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/certificates.md \ + && grep -q 'duplicate' docs/deploy/certificates.md \ + && grep -q '/data' docs/deploy/certificates.md \ + && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Write the complete file** + +Write `docs/deploy/certificates.md` with exactly this content: + +````markdown +# Certificates & ACME — the availability-critical subsystem + +**A publicly-CA-trusted certificate is a hard CPaaS requirement; no self-signed path exists.** +Twilio refuses self-signed certs for webhooks and Media Streams (error +[31910](https://www.twilio.com/docs/api/errors/31910); see +[twilio.com/docs/usage/security](https://www.twilio.com/docs/usage/security)) — and the +console's cert-validation toggle is account-wide, webhook-only, and dev-only, so it is not an +escape hatch. In a 24/7 product whose calls are non-migratable there is no maintenance window: +**auto-renewal is availability-critical, not a nicety.** + +Protocol posture everywhere: TLS 1.2+1.3, mainstream ECDHE suites, **never 1.3-only** +(Twilio's TLS 1.3 client support is undocumented), and never pin Twilio's certificates. + +## The `/data` volume — read this before anything else + +Caddy keeps its certificates and ACME account state in `/data`. If you recreate the container +without that volume, Caddy re-requests the same certificate — and Let's Encrypt's +**duplicate-certificate limit (5 per week for an identical name set, +[letsencrypt.org/docs/rate-limits](https://letsencrypt.org/docs/rate-limits/))** will lock +your domain out for up to a week. The failure class is **total inbound outage**: 31910 on +Media Streams, [11237](https://www.twilio.com/docs/api/errors/11237) on webhooks, no calls in +or out of the trunk. A container recreate loop (crash-looping deploy, CI that recreates on +every push) burns all five in minutes. + +- Always mount `/data` on a persistent volume (`-v rutster-caddy-data:/data`). +- Back it up like state, because it is: + `docker run --rm -v rutster-caddy-data:/data -v "$PWD":/backup debian:stable-slim tar czf /backup/caddy-data.tgz /data` + +## ACME challenge paths + +| Path | When | Notes | +|---|---|---| +| **HTTP-01** (default) | Port 80 publicly reachable, single hostname | Zero config beyond the domain + email. | +| **DNS-01** | Wildcards (`*.pbx.domain`), CGNAT/behind-NAT hosts, no port 80 | The bundled `rutster-edge` Caddy build ships a curated plugin set: **cloudflare, route53, porkbun, hetzner, desec** (duckdns excluded — no license file). Any other DNS provider requires your own xcaddy build. DNS API credentials live in the container env — scope them to the zone. | +| **BYO-cert** (in-process rustls, Phase 1) | You already have cert distribution (corporate ACME, central wildcard issuance) | Set `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY`; the engine serves TLS itself and **hot-reloads on file change without dropping live WS**. You own renewal delivery. | + +In-binary ACME (Phase 2) is deliberately deferred behind named triggers — among them Let's +Encrypt's dns-persist-01 reaching GA, which would dissolve the DNS-plugin matrix entirely. See +[ADR-0011](../adr/0011-deployment-topology.md). + +## Caddy reload vs live calls + +Routine renewals are safe: Caddy swaps certs **in memory, zero-downtime — the routine periodic +event drops nothing.** Config *reloads* (editing the Caddyfile) are the risk: the mitigation +(`stream_close_delay` above max call duration) has an open upstream bug trail +([caddy#6420](https://github.com/caddyserver/caddy/issues/6420), +[caddy#7222](https://github.com/caddyserver/caddy/issues/7222)), which is why the shipped +artifacts carry a CI e2e test for config-reload-during-live-call. Operator rule: **do not edit +the Caddyfile while calls are live** unless your image version passed that test; drain first +(`/readyz` returns 503 while draining). + +## Fleet certificate patterns (T3) + +Node-addressed placement means every node needs its own publicly-valid TLS name — this is +CPaaS-imposed (`` URLs route on hostname only), not a proxy choice. Two blessed +patterns: + +1. **Wildcard `*.pbx.domain`** via DNS-01, issued **centrally** and distributed to nodes. + Understand the trade: the wildcard private key on every node means one compromised node + burns the whole namespace. +2. **Per-node distinct certs** (`node-1.pbx.domain`, `node-2.…`) — no shared key material. + Renewals are exempt from Let's Encrypt's per-domain limits, so this scales with fleet size. + +Traps: + +- **N nodes independently requesting the identical wildcard** are N duplicate requests — the + 5/week duplicate-cert limit locks the fleet out. Central issuance or distinct names only. +- **Caddy on-demand TLS is rejected for node names**: the first TLS handshake to a fresh name + blocks for seconds on issuance, colliding with the sub-second webhook budget (Twilio's hard + cap is 15 s, UX budget sub-second), and its rate-limit knobs are deprecated. Node names are + known at provision time — pre-issue their certs. + +## Troubleshooting + +| Symptom | Fix | +|---|---| +| Twilio 31910 on Media Streams | Cert not publicly trusted / expired / SNI mismatch. Check `openssl s_client -connect pbx.example.com:443 -servername pbx.example.com`. | +| `rateLimited` / `too many certificates` in Caddy logs | You hit the duplicate-cert limit — almost always a lost `/data` volume. Restore the volume or wait out the window; then fix the volume mount so it never recurs. | +| DNS-01 fails for your provider | Provider not in the bundled plugin set — use a delegated zone on a supported provider, or build your own xcaddy image. | +```` + +- [ ] **Step 3: Re-run the content-check + link check** + +```bash +cd /home/alee/Sources/rutster +test -f docs/deploy/certificates.md \ + && grep -q '5 per week' docs/deploy/certificates.md \ + && grep -q 'errors/31910' docs/deploy/certificates.md \ + && grep -q 'errors/11237' docs/deploy/certificates.md \ + && grep -q 'on-demand TLS is rejected' docs/deploy/certificates.md \ + && grep -q 'RUTSTER_TLS_CERT' docs/deploy/certificates.md \ + && grep -q 'never 1.3-only' docs/deploy/certificates.md \ + && echo CHECK-PASS || echo CHECK-FAIL +grep -oE '\]\([^)#]+\)' docs/deploy/certificates.md | sed 's/](//;s/)//' \ + | grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done +``` + +Expected: `CHECK-PASS`; no BROKEN lines if Tasks 1–6 have run. + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add docs/deploy/certificates.md +git commit -s -m "docs(deploy): certificates.md — ACME paths, /data persistence, wildcard traps (slice-G) + +The availability-critical subsystem: no-self-signed-path (Twilio 31910), +the /data-loss -> LE 5/week duplicate-cert lockout -> total-inbound-outage +chain (31910 streams + 11237 webhooks) with backup command; HTTP-01 vs +DNS-01 (bundled plugin set: cloudflare/route53/porkbun/hetzner/desec, +duckdns excluded) vs BYO-cert rustls Phase 1 (RUTSTER_TLS_CERT/KEY, +hot-reload); Caddy renewal-vs-reload distinction with the #6420/#7222 +caveat; fleet patterns (central wildcard vs per-node certs) with the +identical-wildcard trap and the on-demand-TLS rejection." +``` + +--- + +### Task 8: `docs/QUICKSTART.md` — point at the Docker fast path (minimal diff) + +**Files:** +- Modify: `docs/QUICKSTART.md:13-15` (insert one blockquote between the status blockquote and + the `---` rule; verified against current content — the file currently flows from the status + blockquote ending `…for the active build target's design.` straight into `---` / + `## Prerequisites`) +- Test: bash content-check (Steps 1/3) + +**Interfaces:** +- Consumes: `docs/deploy/quickstart-docker.md` (Task 3 — must exist first). +- Produces: nothing downstream. + +- [ ] **Step 1: Run the failing content-check** + +```bash +cd /home/alee/Sources/rutster +grep -q 'deploy/quickstart-docker.md' docs/QUICKSTART.md && echo CHECK-PASS || echo CHECK-FAIL +``` + +Expected: `CHECK-FAIL`. + +- [ ] **Step 2: Apply the minimal edit** + +In `docs/QUICKSTART.md`, find this exact text (end of the status blockquote, lines 12–15): + +```markdown +> [`docs/superpowers/specs/2026-07-05-slice-5-rented-transport-design.md`](superpowers/specs/2026-07-05-slice-5-rented-transport-design.md) +> for the active build target's design. + +--- +``` + +Replace with: + +```markdown +> [`docs/superpowers/specs/2026-07-05-slice-5-rented-transport-design.md`](superpowers/specs/2026-07-05-slice-5-rented-transport-design.md) +> for the active build target's design. + +> **Fastest path:** if you want a *running deployment* — TLS, a domain, a phone +> number — skip the source build and use the Docker quickstart: +> [`docs/deploy/quickstart-docker.md`](deploy/quickstart-docker.md). This page +> is the from-source developer path. + +--- +``` + +No other changes to the file. + +- [ ] **Step 3: Re-run the content-check + link check** + +```bash +cd /home/alee/Sources/rutster +grep -q 'deploy/quickstart-docker.md' docs/QUICKSTART.md \ + && test -e docs/deploy/quickstart-docker.md \ + && echo CHECK-PASS || echo CHECK-FAIL +git diff --stat docs/QUICKSTART.md # expect: 1 file changed, ~6 insertions(+), 0 deletions(-) +``` + +Expected: `CHECK-PASS`; the diffstat shows insertions only (minimal diff honored). + +- [ ] **Step 4: Commit** + +```bash +cd /home/alee/Sources/rutster +git add docs/QUICKSTART.md +git commit -s -m "docs: QUICKSTART points at deploy/quickstart-docker.md as the fast path (slice-G) + +Minimal diff: one blockquote after the status note. The from-source page +stays the developer path; the Docker quickstart is now the operator fast +path (slice-G docs tree)." +``` + +--- + +## Final acceptance checklist + +After all 8 tasks: + +- [ ] **Spec §10 acceptance bar, checked doc by doc:** `quickstart-docker.md` (copy-paste T1 + + T2 to first browser call and first PSTN call), `homelab.md` (copy-paste ngrok dev path + + VPS/WireGuard graduation, **plus** the honest CGNAT-WebRTC impossibility statement), + `aws.md` (copy-paste `aws elbv2` path), `reverse-proxies.md` (three complete configs + + first-call pointer), `topologies.md` (delegates copy-paste to quickstart; states T3 + cannot be deployed today), `certificates.md` (supporting doc; backup + troubleshooting + commands are copy-paste). +- [ ] **All relative links resolve across the whole set** (no BROKEN output): + +```bash +cd /home/alee/Sources/rutster +for f in docs/adr/0011-deployment-topology.md docs/deploy/*.md docs/QUICKSTART.md; do + d=$(dirname "$f") + grep -oE '\]\([^)#]+\)' "$f" | sed 's/](//;s/)//' | grep -v '^http' \ + | while read -r p; do [ -e "$d/$p" ] || echo "BROKEN in $f: $p"; done +done +``` + +- [ ] **ADR-0011 format audit:** header bullet-list (Status/Date/Closes/Origin/Related) → + Context → Decision → Consequences → References, matching ADR-0007/0008; Status is + `Proposed` (ratification is the maintainer's move, not this slice's). +- [ ] **Slice-B reconciliation done or explicitly deferred:** if `deploy/` exists, image + names, service names, `.env.example` var names, and the registry namespace in the docs + match it; if not, the artifact-names note box in `quickstart-docker.md` is present. +- [ ] **Repo gates still green** (docs can't break them; prove it anyway): + +```bash +cd /home/alee/Sources/rutster +cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings && cargo test --all && cargo deny check +git diff main --name-only | grep -E 'crates/rutster-media/src/(loop_driver|rtc_session)\.rs' && echo "SEAM VIOLATION" || echo "seam intact" +``` + +- [ ] Eight commits, each signed off (`git log --format='%h %s' -8` shows the slice-G + series; `git log -8 | grep -c 'Signed-off-by'` = 8). +- [ ] PR opened via `tea` per house workflow; the PR description notes: **ADR-0011 lands as + Proposed — ratification discussion is a separate step**, and names the plan-B + reconciliation status.