# Rutster slice 4½ — Benchmark + simulation harness (the `rutster-sim` seed) - **Status:** Draft (pending review) - **Date:** 2026-07-05 - **Spearhead step:** 4½ of 6 (inserted by [ADR-0010](../../adr/0010-spearhead-benchmark-sim-harness.md) — "after barge-in (4), before rented transport (5)") - **Origin:** [2026-07-03 adversarial review](../../reviews/2026-07-03-adversarial-review.md) (recommendations R1+R2) + [2026-07-03 market feature scan](../../reviews/2026-07-03-market-feature-scan.md) (F1 — simulation as the standout engine-fit feature) - **Depends on (already merged):** - [slice 1 — WebRTC media loopback](2026-06-28-slice-1-webrtc-loopback-design.md) — the media core + the `AudioPipe` trait slice 4½ simulates against - [slice 2 — The agent tap](2026-06-28-slice-2-agent-tap-design.md) — the `TapAudioPipe` slice 4½ measures end-to-end - 2026-06-30 slice-3 realtime brain (merged `c30a452`) — `MockRealtimeBrain` slice 4½ drives the harness against - [slice 4 — Barge-in / VAD-driven playout kill](2026-07-01-slice-4-barge-in-design.md) — the ≤60 ms kill budget this slice MEASURES (slice-4 §5.1) - 2026-07-04 slice-5/seams — `MediaCmd::Stats` exposes `MediaStats { tick_overruns, last_tick_micros }` (the "tick-lag gauge" readout this harness surfaces as the primary concurrency-sweep result) - **Related:** [ADR-0002](../../adr/0002-north-star-and-fused-core.md) (the wedge: "local real-time reflexes that don't need the brain" — currently an arithmetic claim, this slice proves it), [ADR-0008](../../adr/0008-fob-and-green-zone.md) (the harness is FOB: hot-path-adjacent + differentiating), [ARCHITECTURE.md §"Biggest technical risk"](../../ARCHITECTURE.md) (the reflex loop *is* the long pole; the harness is its measurement surface) --- ## TL;DR Stand up spearhead step 4½: a self-hostable **benchmark + simulation harness** in a new `crates/rutster-sim/` crate. It drives synthetic callers through the SAME media-leg path real callers use, measures **p50/p99 mouth-to-ear latency** and **barge-in kill-time** against slice-4's ≤60 ms kill budget, and runs the same measurements at **1 / 10 / 50 concurrent synthetic calls**. A separate CI job (opt-in via `--features=sim-bench`) asserts the per-budget thresholds every commit. **A latency regression fails the build the same way a broken test does** — ADR-0010's central demand. The harness is also the **doctrine-drift detector** for the timing-thread debt (slice-4 §1.2 deferred per-session threads; slice-5/seams landed a single shared std::thread + a tick-lag gauge without data). 4½'s concurrency sweep turns `MediaStats.tick_overruns` from a counter into a **decision artifact**: "does single-thread poll loop breach budget at realistic concurrency?" gets answered with data, not vibes. If yes, the dedicated threadpool-shard graduation (slice-4 deferral #2) gets scheduled on evidence. Three things this slice deliberately is NOT: - **Not an LLM-driven caller** (the scenario pack is scripted PCM segments; LLM-driven callers are post-spearhead). The harness ships with three deterministic scenarios that exercise the wedge. - **Not a fuzz target** (the harness exercises real latency under load; it is not designed to find panics in the PCM path — that's the deferred `cargo-fuzz` harness). - **Not a load test for production capacity** (p99 under 50 synthetic calls is the sperhead-scale limit; load testing for horizontally-scaled fleet deployments lands with the rung-2 escalation / step-6 spend-cap work). --- ## 1. Scope ### 1.1 In scope - A new `crates/rutster-sim/` crate (currently non-existent; this slice creates it from `Cargo.toml` + `lib.rs`). - A **`Scenario` data type** + a TOML scenario file format (`crates/rutster-sim/scenarios/*.toml`) describing scripted caller PCM playback with timing directives ("speak N frames, pause M frames, interrupt at T"). At least three shipped scenarios: `loud-barge.toml`, `quiet-advisory.toml`, `sustained-call.toml`. - A **`SimAudioPipe: AudioPipe`** implementation that drives a `Scenario` on `next_pcm_frame` (plays back scripted PCM frames) and captures received frames on `on_pcm_frame` (timestamps each — the latency-measurement boundary). - A **`LatencyProbe`** that computes the two metrics slice-4's design sets targets for: - **Mouth-to-ear round-trip** — caller-speech-onset → first frame returned at the caller's "ear" (the `SimAudioPipe`'s source path). - **Barge-in kill-time** — caller-speech-onset → reflex-barge-fire (the moment `next_pcm_frame` first returns `None` post-barge). - A **`ConcurrencyRunner`** that spawns N concurrent `SimCall`s against a target binary URL (default: in-process via `MediaThread` for determinism) and computes per-call latencies + the p50/p99 aggregates across the sample. - **Tick-lag gauge integration**: the runner reads `MediaStats { tick_overruns, last_tick_micros }` via `MediaCmd::Stats` (already exposed by slice-5/seams) and surfaces both as primary readouts in the sweep report — both the per-call `last_tick_micros` distribution and the cumulative `tick_overruns` count. - **CI-regressed thresholds**: a separate CI job runs `cargo test --all --features=sim-bench` per PR + nightly; thresholds asserted; a regression fails the build. - New learner-facing comments explaining the measurement discipline (the test corpus *is* the code-reading curriculum for "how do you measure a real-time system without lying to yourself?" — the slice-1 §7 verbosity standard carries over). ### 1.2 Out of scope (with scheduled return) | Deferred item | Returns in | Why deferred | |---|---|---| | LLM-driven synthetic callers | post-spearhead refinement | ADR-0010 says scripted scenarios are 4½; LLM callers are an extension. Scripted scenarios are deterministic (reproducibility is the point); LLM callers add variance that complicates threshold-assertion CI gates. | | Per-environment calibration of threshold values | post-spearhead | The MVP ships thresholds tuned for the dev loopback (smoke) + CI runner (deterministic box). Real-world noise calibration (per-CPU, per-RAM) is a tuning-framework problem — paired with slice-4's VAD-threshold tuning deferral. | | Multi-perspective scenario recording (the caller's audio + the operator's audio + the brain's audio saved per-run as WAV for review) | later rung (rung-2 escalation) | Recording per-call audio for supervisor review is a rung-2 capability (warm-handoff artifact). The harness measures; recording is a separate concern. | | Distributed / multi-binary fleet latency sweep | rung 3 (post-escalation) | ADR-0010's sweep targets a single binary at low concurrency. Fleet-scale (N binaries, NAT, autoscaling) lands after the trunk is real (step 5) + escalation is real (rung 2). | | Adaptive `VAD_RMS_THRESHOLD` tuning | post-spearhead | Slice-4 §1.2 defers; 4½ inherits the const threshold (500.0). Real-world noise-floor learning is a later rung. | | Concurrency > 50 calls | later rung | The spearhead's scale envelope caps at "dozens of PSTN calls on one box" (slice-4 §6.3); 50 is the upper claim. Beyond 50 ⇒ fleet (above), not single-binary. | | Latency under degraded brain conditions (mock brain stalls) | later refinement | The MVP measures against `MockRealtimeBrain` (deterministic). The real latency risk is "what if the brain takes 800 ms instead of 300 ms?" — addressed by a "brain slow" scenario ADDED LATER, not in the MVP. | | Hooking `cargo bench` / criterion into the CI artifact publishing | later | False precision: the harness is `cargo test --features=sim-bench`, not `cargo bench`. We are asserting threshold-shape gates, not micro-bench-diff regressions (which are noisy in CI). `criterion` is a different epistemology. | | Browser-based e2e (Playwright/Selenium) | post-spearhead | Unchanged from prior slices' deferral. The synthetic-peer harness is the test vehicle. | --- ## 2. Architecture delta ### 2.1 What changes vs slice-4 + slice-5/seams Slice 4½ adds ONE crate + one CI matrix entry. The fused vertical's existing hot path (`loop_driver.rs` + `rtc_session.rs` + `MediaThread` + `Reflex

` + `TapAudioPipe`) is **untouched**. The harness drives the same media-leg ingress path a real caller uses — so what it measures is what the customer experiences, by construction. ``` ┌─ crates/rutster-sim/ (NEW, this slice) ──────────┐ │ │ │ Scenario(.toml) ──► SimAudioPipe: AudioPipe │ │ │ │ │ LatencyProbe ◄── timestamps on frames passing │ │ through the pipe │ │ │ │ ConcurrencyRunner (1/10/50 SimCalls) │ │ │ │ │ └─► MediaCmd::Stats (slice-5/seams already) │ │ reads tick_overruns / last_tick_micros│ └──────────────────────┬────────────────────────────┘ │ drives via the existing │ media-leg ingress path ▼ ┌────────────────────────────────────────────── Rutster trust boundary (FOB) ────────────┐ │ WebRTC ingress / Trunk ingress (slice-5) → RtcSession → Reflex │ │ ↑ ↑ ↑ ↑ │ │ └── existing, untouched ───┘ │ │ │ │ │ │ │ │ loop_driver.rs + rtc_session.rs (byte-identical) ┘ │ │ │ │ │ │ MediaThread (slice-4) ─── MediaCmd::{Register, AcceptOffer, Delete, │ │ Shutdown, Stats, Drain, RegisterTrunk[slice-5]} │ │ │ │ Tick-lag gauge: MediaStats { tick_overruns, last_tick_micros } (slice-5) │ └───────────────────────────────────────────────────────────────────────────────────────────┘ ``` ### 2.2 The measurement boundary (load-bearing — gets it wrong, the harness lies) A latency measurement is honest only if the timestamp is captured at the same point the customer experiences the audio. The harness employs TWO clocks: - **Caller-side onset timestamp** (`t_onset`): captured inside `SimAudioPipe::on_pcm_frame` when a scenario-step directive says "speak now." This is the wall-clock the *caller* started speaking. - **Caller-side receipt timestamp** (`t_receipt`): captured inside `SimAudioPipe::next_pcm_frame` when the brain's response PCM frame is returned to the "caller's ear" (the `SimAudioPipe`'s source path returns `Some(frame)`). Latency = `t_receipt - t_onset`. Both timestamps come from `Instant::now()` (monotonic, unaffected by NTP) within the same `SimAudioPipe` instance — so wall-clock skew between nodes is structurally eliminated. The harness measures *the system's response to a synthetic caller*, not "what time it is on the operator's box vs the caller's box." This is the design choice that distinguishes the harness from "just add some `Instant::now()` calls in tracing." The `SimAudioPipe` IS the caller — it owns both the onset timestamp (it decided when to "speak") and the receipt timestamp (it observed when the system "replied"). The harness can't lie about latency because the only clock it uses is the caller's clock. ### 2.3 Why `SimAudioPipe` instead of a tracing-only approach The alternative: instrument `loop_driver.rs` / `rtc_session.rs` / the tap with `tracing::Span` timestamps and aggregate them post-hoc. We reject this design because: 1. **It violates the seam.** `loop_driver.rs` + `rtc_session.rs` are byte-identical seams; timing instrumentation would either change those files (slice-1 §8.5 #6 violated) or layer on top via decorators (complex, flaky under load). 2. **It measures wall-clock at multiple points, not customer experience.** A `tracing::Span` that says "encode took 4 ms" tells us nothing about whether the brain's reply actually reached the caller's ear in ≤60 ms — there's queueing + jitter + WS transport between "encode" and "heard." 3. **It produces observability dashboards, not CI gates.** Operators need dashboards (separate concern — the `EventSink` from slice-5/seams emits per-call CDR-anchored fields); the spearhead needs *regression-failing thresholds* stated as test assertions, which is the harness's job. The `SimAudioPipe` is the right architectural choice for **CI-suite measurements**. Tracing dashboards land with ADR-0005 Valkey wiring (a later rung) — both necessary, both different. ### 2.4 Why 1 / 10 / 50 (and not other values) - **1 call** isolates the *baseline* — the cold-path latency with zero concurrency pressure. If this regresses, the bug is in the loop itself, not in contention. Slice-4's §5.1 ≤60 ms kill budget is asserted here. - **10 calls** is the *warm working set.* Approximately the peak spearhead-scale; the std thread's 10 ms meta-tick comfortably fits 10 sessions per tick (each session costs <100 µs in `loop_driver::drive`), so this asserts budget at warm-but-uncontended conditions. - **50 calls** is the *saturation point.* ADR-0010's "single-poll-task head-of-line-blocking debt" (review P2) lives here: 50 sessions per 10 ms tick = 200 µs per session in the meta-tick before contention squeezes; if `last_tick_micros` stays under 10 ms the single-thread design holds; if `tick_overruns` grows past some threshold the dedicated-threadpool-shard graduation (slice-4 §1.2 deferral) gets its data-driven case. We do NOT test 100/500/5000 — that's fleet-scale (rung 3). 50 is the upper edge of the spearhead's "one binary, one city" claim. ### 2.5 The CI gate shape A regression-failing threshold is a test assertion. The crate `crates/rutster-sim/` ships: - `#[cfg(feature = "sim-bench")]` modules + tests — default OFF, so `cargo test --all` (the routine CI gate on every PR) stays fast (assertions of *correctness*, not measurement). - A `cargo test --all --features=sim-bench` invocation in a SEPARATE CI job. This job runs on every PR + nightly. Failure ⇒ red X ⇒ PR does not merge. The thresholds are encoded as Rust `assert!` statements in `crates/rutster-sim/src/thresholds.rs`: ```rust // crates/rutster-sim/src/thresholds.rs /// Slice-4 spec §5.1 + §7 done-criteria #8: kill-time budget is /// ≤60 ms (3 debounce frames × 20 ms tick) + 1 tick to drain + apply. /// Observer slack to make CI deterministic-but-not-flaky on a slow runner: /// effective CI assertion ≤80 ms (60 ms budget + 20 ms slack). pub const BARGE_IN_KILL_TIME_P99_MS: f64 = 80.0; /// Slice-1 + slice-3 mouth-to-ear budget: 200 ms (slice-1 notification) + /// 250 ms mock brain + 100 ms playout buffer. CI assertion ceiling: /// 700 ms (allowance for CI runner variance against dev machine). pub const MOUTH_TO_EAR_P99_MS: f64 = 700.0; /// Slice-5/seams tick-lag gauge: the meta-tick must stay under 10 ms /// (the loop's nominal period). At 1 call: ≤2 ms. At 50 calls: ≤10 ms. /// Tick overruns (count of ticks exceeding 10 ms) at p50 across the sweep: /// ≤1% of total ticks. pub const TICK_LAG_MAX_MS: f64 = 10.0; pub const TICK_OVERRUN_PCT_MAX: f64 = 1.0; /// Concurrency-sweep sample sizes. pub const SWEEP_CONCURRENCIES: &[usize] = &[1, 10, 50]; ``` These are *constants* for the MVP. They become *env-var configurable* in a post-spearhead tuning-framework (paired with slice-4's VAD threshold tuning deferral). --- ## 3. Component design ### 3.1 `Scenario` + `ScenarioStep` ```rust // crates/rutster-sim/src/scenario.rs /// A scripted caller scenario. Read from TOML (a scenario file under /// `crates/rutster-sim/scenarios/*.toml`). Deterministic by construction — /// the entire point is reproducible thresholds in CI. /// /// # Why TOML (not YAML, not RON) /// /// `serde` + `toml` is already a workspace member (cargo-deny licenses track). /// TOML keeps the scenario file readable as a one-shot script (a sequence /// of named steps + numbers); YAML would invite flow-mapping complexity /// the scenario format doesn't need. #[derive(Debug, Clone, serde::Deserialize)] pub struct Scenario { /// Human-readable identifier; shows up in the CI failure message. pub name: String, /// The sequence of caller-side actions; played front-to-back. pub steps: Vec, } /// One axis of caller behavior. A scenario is a time-ordered sequence of /// these. The `SimAudioPipe` consumes them in order during `on_pcm_frame`. #[derive(Debug, Clone, serde::Deserialize)] #[serde(tag = "kind", rename_all = "snake_case")] pub enum ScenarioStep { /// Send N loud PCM frames (sample value 1000, well above VAD_RMS_THRESHOLD=500.0). /// Triggers the local VAD via slice-4's `LocalVadReflex::on_pcm_frame`. SpeakLoud { frames: u32 }, /// Send N zero frames (sample value 0, well below VAD_RMS_THRESHOLD). /// Used by the quiet-caller advisory scenario: drives mock-brain advisory path. SpeakQuiet { frames: u32 }, /// Insert N zero frames before the next step (silence pacing). Pause { frames: u32 }, /// Wait until the harness receives M "ear" frames before advancing. /// Used as a barrier: the brain's reply must arrive before the next caller action. AwaitReply { frames: u32 }, /// End the scenario. The `SimAudioPipe` returns None from next_pcm_frame thereafter. End, } ``` Scenario file example (`crates/rutster-sim/scenarios/loud-barge.toml`): ```toml # Drives the PRIMARY barge-in path (slice-4 §5.1). The caller says one loud # burst of audio; the local VAD trips; playout dies; no brain advisory needed. # Asserts the wedge-#1 path: "VAD killing TTS the instant the caller speaks, # without the brain." name = "loud-barge" [[steps]] kind = "speak_loud" frames = 20 # 20 frames @ 20ms = 400 ms of speech; comfortably past the 60 ms debounce [[steps]] kind = "await_reply" frames = 0 # barrier: the FOB should be muted at this point (Reflex::muted == true) [[steps]] kind = "end" ``` ### 3.2 `SimAudioPipe: AudioPipe` ```rust // crates/rutster-sim/src/sim_audio_pipe.rs /// A test-double `AudioPipe` that simulates a caller. Drives a `Scenario` /// on `on_pcm_frame` (the sink path: caller speaks); receives brain /// response frames on `next_pcm_frame` (the source path: caller hears). /// Captures `Instant::now()` at every meaningful event for the /// `LatencyProbe` to consume. /// /// # Why this is THE measurement boundary /// /// Both clocks live INSIDE this pipe. The wall-clock the *caller* started /// speaking is captured here (we decided when to "speak"); the wall-clock /// the *caller* heard the reply is captured here (we observed the system's /// reply). See §2.2 — the harness can't lie about latency because the only /// clock it uses is the caller's. pub struct SimAudioPipe { /// The scenario under playback. scenario: Scenario, /// Cursor into `scenario.steps`. step_idx: usize, /// Frames remaining in the current step (decrements per `on_pcm_frame` /// for `SpeakLoud`/`SpeakQuiet`/`Pause`; freezes for `AwaitReply`). step_frames_remaining: u32, /// Frames received from `next_pcm_frame` while in `AwaitReply`. /// When this reaches the step's `frames` target, advance. reply_frames_received: u32, /// Captured timestamps (the `LatencyProbe` consumes this via /// `take_captures()` after the run). Discarded on every `on_pcm_frame` /// call once the capture buffer is at capacity (bounded; hot-path). captures: Vec, /// A pre-allocated frame returned from `next_pcm_frame` when we have a /// pending reply (the harness intercepts frames routed back through the /// existing media loop and returns them here). See §3.4 for how the /// `SimCall` wires this pipe to a real `MediaThread`. reply_ring: std::collections::VecDeque, } /// A timestamped event captured by the `SimAudioPipe`. Read by the /// `LatencyProbe` post-run to compute p50/p99 latencies. #[derive(Debug, Clone, Copy)] pub enum Capture { /// The caller started speaking loudly (a `SpeakLoud` step began). CallerLoudOnset { at: Instant }, /// The FOB killed playout (a `next_pcm_frame` call returned None /// immediately after a barge event). The wall-clock this slice cares /// about for kill-time. BargeKillObserved { at: Instant }, /// The caller heard a brain reply (a `next_pcm_frame` returned Some /// after the barge cleared). The wall-clock this slice cares about /// for mouth-to-ear. CallerHeardReply { at: Instant }, } ``` #### 3.2.1 The two key methods ```rust // crates/rutster-sim/src/sim_audio_pipe.rs (continued) impl AudioSource for SimAudioPipe { fn next_pcm_frame(&mut self) -> Option { // The source path: the caller's "ear." Brain replies (frames the // system produced on the egress side and routed back to us via the // SimCall's mpsc wiring — see §3.4) land here. match self.reply_ring.pop_front() { Some(frame) => { if self.is_in_await_reply_step() { self.reply_frames_received += 1; if self.reply_frames_received >= self.current_step_target() { self.advance_step(); } } // Capture: this is the "caller heard" wall-clock. self.captures.push(Capture::CallerHeardReply { at: Instant::now() }); Some(frame) } None => { // The reflex muted us (slice-4's Reflex

::muted == true). // Capture: this is the "barge kill observed" wall-clock. // (Only capture if we are mid-AwaitReply post-barge; see the // LatencyProbe's classification §3.3 for the dedup logic.) self.captures.push(Capture::BargeKillObserved { at: Instant::now() }); None } } } } impl AudioSink for SimAudioPipe { fn on_pcm_frame(&mut self, _frame: PcmFrame) { // The sink path: the caller "speaks." The scenario drives here. // Decode the current step + emit the appropriate PCM signal. self.dispatch_step_action(); // (We discard the inbound frame — the caller doesn't hear itself; // the SimCall's wiring pushes the caller-side frame into the // `tx_pcm_in` channel for the tap to forward to the brain.) } } ``` ### 3.3 `LatencyProbe` ```rust // crates/rutster-sim/src/latency.rs /// Computes the two metrics slice-4's design sets budgets for, from a /// captured-stream of `Capture` events produced by a `SimAudioPipe`. /// /// # Verification discipline /// /// The probe is the single source of truth for "did latency regress?" /// Assertions are made against `LatencyProbe::p99_kill_ms()` and /// `LatencyProbe::p99_mouth_to_ear_ms()`. A failure here is the build-red /// signal ADR-0010 demands. pub struct LatencyProbe { captures: Vec, } impl LatencyProbe { pub fn from_captures(captures: Vec) -> Self { Self { captures } } /// Barge-in kill-time: caller-speech-onset → first `BargeKillObserved`. /// Returns Duration per call (we capture one onset+kill pair per barge). pub fn kill_times(&self) -> Vec { /* pair captures */ } /// Mouth-to-ear: caller-speech-onset → next `CallerHeardReply`. pub fn mouth_to_ear_times(&self) -> Vec { /* pair captures */ } pub fn p50_kill_ms(&self) -> f64 { percentile(&self.kill_times(), 50) } pub fn p99_kill_ms(&self) -> f64 { percentile(&self.kill_times(), 99) } pub fn p50_mouth_to_ear_ms(&self) -> f64 { percentile(&self.mouth_to_ear_times(), 50) } pub fn p99_mouth_to_ear_ms(&self) -> f64 { percentile(&self.mouth_to_ear_times(), 99) } } fn percentile(durations: &[Duration], pct: u8) -> f64 { if durations.is_empty() { return f64::NAN; } let mut sorted: Vec = durations.iter().map(|d| d.as_millis()).collect(); sorted.sort_unstable(); let idx = ((sorted.len() as f64 - 1.0) * (pct as f64 / 100.0)).round() as usize; sorted[idx] as f64 } ``` The `LatencyProbe` is **post-hoc**: a single `SimCall` runs to completion, the `SimAudioPipe`'s captures are drained, the probe computes the metrics. No per-tick instrumentation cost during the hot path itself — just the `Instant::now()` calls inside `SimAudioPipe::next_pcm_frame` (which isn't the hot path anyway; it's the simulated-caller epilogue). ### 3.4 `SimCall` (the wiring) + `ConcurrencyRunner` ```rust // crates/rutster-sim/src/runner.rs /// One simulated call: a `SimAudioPipe` + the wiring to drive it against the /// existing `MediaThread`. Single binary; no separate process. pub struct SimCall { /// The scenario-driven caller pipe. pipe: SimAudioPipe, /// The `MediaThread` cmd_tx — we register a session, drive the pipe /// via `tx_pcm_in`, capture frame receipts in `next_pcm_frame`. media_cmd_tx: mpsc::Sender, /// Latency probe populated post-run. probe: Option, } impl SimCall { pub async fn run(mut self) -> LatencyProbe { // 1. Register a session with the MediaThread. // 2. Wire self.pipe as the session's AudioPipe (MediaCmd::RegisterSim). // 3. Drive the scenario: each scenario step emits `on_pcm_frame` // calls against the SimAudioPipe; the MediaThread's loop_driver // echoes frames back via next_pcm_frame. // 4. On End: drain captures + return the probe. todo!("see §4 data flow") } } /// The concurrency sweep runner. Spawns N `SimCall`s in parallel (tokio), /// awaits all, aggregates per-call latencies into the sweep report. pub struct ConcurrencyRunner { /// Target binary in-process MediaThread cmd_tx. Passed in by the test fixture. media_cmd_tx: mpsc::Sender, /// Concurrency levels to sweep (slice-4½ hardcoded [1, 10, 50]). concurrencies: Vec, } impl ConcurrencyRunner { /// Run the full sweep; return the per-concurrency-level report. pub async fn run(&self, scenario: Scenario) -> SweepReport { /* ... */ } } /// The artifact feeding the CI assertions. The thresholds.rs asserts /// `report.barge_kill_p99_ms <= BARGE_IN_KILL_TIME_P99_MS` etc. #[derive(Debug)] pub struct SweepReport { pub per_concurrency: Vec, } #[derive(Debug)] pub struct PerConcurrencyReport { pub concurrency: usize, pub p50_kill_ms: f64, pub p99_kill_ms: f64, pub p50_mouth_to_ear_ms: f64, pub p99_mouth_to_ear_ms: f64, /// From slice-5/seams MediaCmd::Stats. The "doctrine-drift detector" /// for the timing-thread debt — ADR-0010's debt-pairing readout. pub max_tick_lag_micros: u64, pub tick_overruns: u64, pub total_ticks: u64, pub tick_overrun_pct: f64, } ``` ### 3.5 `MediaCmd::RegisterSim` (the seam — one new enum variant) Slice-5/seams already exists with `MediaCmd::Register`, `AcceptOffer`, `Delete`, `Shutdown`, `Stats`, `Drain`. Slice 4½ adds ONE variant: `RegisterSim`, which lets a `SimCall` register a session whose `AudioPipe` is a `SimAudioPipe` instead of a WebRTC-backed `RtcSession`. This is the minimum extension to drive the harness without needing to spin up a real WebRTC peer. ```rust // crates/rutster/src/media_thread.rs (extended this slice) pub enum MediaCmd { // ... existing variants (unchanged from slice-5) /// slice-4½: harness-side session registration. The /// `SimAudioPipe` lives entirely on the binary side; no WebRTC /// handshake needed. The `tx_pcm_in` channel is the existing /// sink-input seam (the harness emits `on_pcm_frame(frame)` directly /// rather than the loop_driver pulling RTP + decoding first). RegisterSim { pipe: Box, reply: oneshot::Sender, }, } ``` The std thread's `run_media_thread` handles `RegisterSim` by inserting a synthetic "session" entry that drives the harness's `SimAudioPipe` through the same `loop_driver::drive` calls as real WebRTC sessions — the seam holds (`loop_driver.rs` + `rtc_session.rs` byte-identical). The harness measures what real callers experience because **it drives the same code path**. ### 3.6 Tick-lag gauge integration (the doctrine-drift detector) Slice-5/seams added `MediaStats { tick_overruns, last_tick_micros }` to `MediaCmd::Stats`. Slice 4½ consumes both. The `ConcurrencyRunner` polls `MediaCmd::Stats` once per second during the sweep; the per-concurrency-level report carries: - `max_tick_lag_micros` — the maximum observed `last_tick_micros` during the sweep at this concurrency level. Indicates "the worst tick the loop experienced." - `tick_overruns` / `total_ticks` — cumulative count of ticks where `last_tick_micros > 10_000` (10 ms); converts to percent. Indicates "what fraction of ticks overflowed." The thresholds (`TICK_LAG_MAX_MS = 10.0`, `TICK_OVERRUN_PCT_MAX = 1.0`) are the answer to ADR-0010's "if the concurrency sweep shows the shared-tokio poll loop breaching budget at realistic call counts, the dedicated-timing-thread work gets scheduled on data, not vibes." If slice 4½'s sweep shows `tick_overrun_pct > 1.0` at 50 calls, **the FOB reflex loop's single-thread debt is real and graduates from doctrine to data** — that's the slice's load-bearing finding regardless of whether the latency thresholds pass. --- ## 4. Data flow ### 4.1 Single `SimCall` (the unit of measurement) ``` 1. SimCall::run() starts. 2. Send MediaCmd::RegisterSim { pipe: Box, reply } to media_cmd_tx. 3. The MediaThread handles RegisterSim: - Constructs a "synthetic session" entry in its HashMap. - The session's `pipe` field is the SimAudioPipe. - Subsequent loop_driver::drive(now) calls touch this session identically to a WebRTC session (the seam holds). 4. The harness drives the scenario: - For each SpeakLoud/SpeakQuiet step: the SimCall emits `pipe.on_pcm_frame(frame)` calls at the 20 ms tick cadence. loop_driver::drive's encode path is bypassed for the SimPipe (we wrote the frame directly into the pipe). - loop_driver::drive immediately treats the next_pcm_frame call as the source path: it pulls from the SimPipe's reply_ring (where brain replies populate when the SimCall sees them via the tap/engine path). 5. Brain replies (from MockRealtimeBrain or the in-process tap): - Routed back into the SimPipe's reply_ring via an mpsc the SimCall holds. - loop_driver::drive picks them up on next_pcm_frame, encodes if real-WebRTC, but for sim we just observe — captures CallerHeardReply timestamp. 6. Reflex barge-in (slice-4 already merged): - If the SimPipe emitted SpeakLoud frames, the LocalVadReflex

(which wraps the pipe in the session_map/MediaThread composition site, slice-4 Task 6) trips the SpeechStarted advisory → Reflex

::muted = true → next_pcm_frame returns None → capture BargeKillObserved timestamp. 7. On End step: harness stops driving, returns LatencyProbe. ``` ### 4.2 Concurrency sweep (the doctrine detector) ``` 1. ConcurrencyRunner::run(scenario) launches N Tokio tasks, each running SimCall::run against the SAME media_cmd_tx (shared media thread). 2. The MediaThread drives ALL N sessions per 10 ms meta-tick (slice-4 §2.2 unchanged shape). 3. Per second during the sweep: ConcurrencyRunner fires MediaCmd::Stats; accumulates tick_overruns + last_tick_micros samples. 4. On all SimCalls completing: aggregate per-call LatencyProbes → p50/p99 vector. 5. Build SweepReport with per-concurrency rows, asserting thresholds.rs constants. ``` ### 4.3 Why in-process (not client-server) The harness DOES NOT stand up the binary as a server + a separate sim-client process. Why: 1. **Determinism.** Loopback within the same process eliminates websocket / socket / TCP jitter as a confounder. The thresholds are assertions about the FOB reflex loop itself — not assertions about NTPD variance on the CI runner. 2. **CI simplicity.** `cargo test --features=sim-bench` runs in-process; no port binding, no test-orchestration container, no race against `epoll` initialization. 3. **Direct seam access.** The harness can construct a `SimAudioPipe` and ship it via `MediaCmd::RegisterSim` directly — same path the production binary would use if it had a "synthetic caller" feature, no client-server glue needed. A separate client-server mode (true loopback against the binary's HTTP/WebSocket surface) IS deferred — it's needed when the harness gains LLM-driven callers (post-spearhead refinement) and needs network realism. --- ## 5. Measurement plan + thresholds ### 5.1 The budgets (concrete numbers) | Metric | Budget (slice-4 design) | CI assertion (slice 4½) | Rationale for slack | |---|---|---|---| | Barge-in kill-time, p99 | ≤60 ms (3 debounce × 20 ms + 1 drain tick) | ≤80 ms | CI runner has known variance against dev; the budget is 60 ms; the assertion is 80 to avoid flakiness. | | Mouth-to-ear round-trip, p99 | ≤200 ms slice-1 + ≤300 ms mock brain + ≤100 ms playout = ~600 ms | ≤700 ms | Same logic; the mock brain is deterministic but the harness adds observer cost. | | Tick-lag (max prev-poll duration) | unspecified | ≤10 ms | The slice-5/seams META_TICK const; the invariant the assertion makes explicit. | | Tick overruns (fraction of ticks > 10 ms) | unspecified | ≤1% | At 50 calls × 1000 ticks each = ≥99% need to be ≤10 ms. Allows for one scheduling hiccup per ~99 well-behaved ticks. | ### 5.2 Per-concurrency swept assertions ``` For each N ∈ [1, 10, 50]: run scenario loud-barge.toml against N concurrent SimCalls. assert p99_kill_ms <= 80 ms; assert p99_mouth_to_ear_ms <= 700 ms; assert max_tick_lag_micros <= 10_000; // 10 ms assert tick_overrun_pct <= 1.0; ``` ### 5.3 The scenarios (3 shipped) | Scenario | Path | What it asserts | |---|---|---| | `loud-barge.toml` | Caller speaks 20 loud frames → awaits reply → end. | The PRIMARY barge-in path (slice-4 §5.1): local VAD fires, kill within ≤80 ms at p99, NO brain advisory required. | | `quiet-advisory.toml` | Caller speaks 20 quiet frames (sub-VAD-threshold) → awaits reply → end. | The SECONDARY barge-in path (slice-4 §5.2): brain advisory fires from `MockRealtimeBrain`, kill flows through slice-3 plumbing + slice-4 `advisory_tx` → `Reflex`. | | `sustained-call.toml` | Caller speaks 10 loud → 10 quiet → 10 loud → 10 quiet → 10 loud → end (5 minutes of talk). The fatigue / sustained-load check. | Multi-barge: 3 `SpeechStarted` advisories should fire in sequence; the `Reflex::barge_epoch` increments 3×; latency is asserted across all three bars (the second + third bar shouldn't drift > 1.5× the first). | ### 5.4 CI integration The CI workflow gains a new job: ```yaml # .github/workflows/ci.yml (additive) sim-bench: name: sim-bench (stable) runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable - uses: Swatinem/rust-cache@v2 - name: Install libopus run: sudo apt-get update && sudo apt-get install -y libopus-dev - name: Run sim-bench threshold sweep run: cargo test --all --features=sim-bench -- --test-threads=1 ``` Notes: - `--test-threads=1` — concurrent sim-bench tests would pollute each other's `MediaStats` polling (the tick-lag gauge measures the SHARED media thread; concurrent runs of the sweep would contaminate each other). - Run on `stable` only — the matrix already runs `cargo test --all` on stable + 1.85; the bench feature lives on stable. ### 5.5 The thresholds-as-test contract ```rust // crates/rutster-sim/src/thresholds.rs (continuation — the CI test entry) #[cfg(test)] mod threshold_assertions { use super::*; #[tokio::test] #[cfg(feature = "sim-bench")] async fn loud_barge_at_each_concurrency_passes_thresholds() { let scenario = Scenario::load("scenarios/loud-barge.toml").unwrap(); for &n in SWEEP_CONCURRENCIES { let report = ConcurrencyRunner::in_process(n).run(scenario.clone()).await; let row = report.per_concurrency.iter() .find(|r| r.concurrency == n).expect("concurrency row"); assert!( row.p99_kill_ms <= BARGE_IN_KILL_TIME_P99_MS, "p99 kill-time at N={}: {}ms > {}ms (budget overflow)", n, row.p99_kill_ms, BARGE_IN_KILL_TIME_P99_MS, ); assert!( row.p99_mouth_to_ear_ms <= MOUTH_TO_EAR_P99_MS, "p99 mouth-to-ear at N={}: {}ms > {}ms", n, row.p99_mouth_to_ear_ms, MOUTH_TO_EAR_P99_MS, ); assert!( (row.max_tick_lag_micros as f64) / 1000.0 <= TICK_LAG_MAX_MS, "max tick-lag at N={}: {}us > {}ms", n, row.max_tick_lag_micros, TICK_LAG_MAX_MS, ); assert!( row.tick_overrun_pct <= TICK_OVERRUN_PCT_MAX, "tick overrun % at N={}: {}% > {}%", n, row.tick_overrun_pct, TICK_OVERRUN_PCT_MAX, ); } } #[tokio::test] #[cfg(feature = "sim-bench")] async fn quiet_advisory_at_1_concurrency_passes_thresholds() { /* ... */ } #[tokio::test] #[cfg(feature = "sim-bench")] async fn sustained_call_multibarge_does_not_drift() { /* ... */ } } ``` --- ## 6. Why these decisions ### 6.1 Why a new crate (`crates/rutster-sim/`) instead of in-tree tests A test under `crates/rutster/tests/sim_*.rs` would absorb the harness into the binary crate — encouraging the harness to depend on binary internals. The separate crate keeps the dependency direction: `rutster-sim` → `rutster-media` + `rutster-call-model` + `rutster` (for `MediaCmd`). The sim crate is a **FOB member** per ADR-0008: hot-path-adjacent (drives the loop), differentiating (the proof artifact). It earns cratehood the same way `rutster-tap` did. ### 6.2 Why scripted scenarios, not LLM-driven callers (deferred) - **Reproducibility** is the entire point. A CI gate that asserts "p99 ≤ 80 ms" can't be flaky on LLM variance — the LLM might take 2 s to respond, the threshold fails, the build is red, the dev attributes it to the LLM provider. The signal is corrupted. - **Isolation** of what's being measured: scripted scenarios measure the FOB + the loop + the reflex; LLM callers introduce brain-side variance that isn't a FOB property. - LLM-driven callers land in a post-spearhead refinement tier (ADR-0010 explicit deferral). They measure a different question ("how does it feel to talk to my brain?") than the MVP harness asks ("does the FOB reflex loop meet its budget under load?"). ### 6.3 Why in-process, not client-server (the false-precision risk) A client-server test harness gives Network-Realism™ but takes away the **measurement discipline**: a TCP retransmit on the CI runner would inflate p99 unattributably, and a "drift over threshold" failure becomes a triage time-sink ("is the FOB regressing, or did the CI runner's network blip?"). The in-process mode gives **single-process determinism** at the cost of network realism — which the spearhead explicitly doesn't need (it's measuring the FOB loop, not the deployed system). A future post-spearhead tier adds a client-server mode for integration realism + a different set of thresholds. ### 6.4 Why tick-lag as a primary readout (the doctrine-drift contract) The single-thread media loop (slice-4 §6.3) is a spearhead-scale decision explicitly deferred to a "load demands it" trigger. Without the harness measuring it under load, the deferral has no data-driven graduation criterion — it'd be doctrine ("faster per-session threads later") instead of evidence ("measured `tick_overrun_pct = 12%` at 50 calls, the threadpool shard lands now"). ADR-0010 explicitly pairs the debt — `MediaStats.tick_overruns` exists because of slice-5/seams' seam work; slice 4½ makes the gauge OBSERVED. Both pass and the single-thread design is validated; both breach and the graduation is scheduled with data. ### 6.5 Why `--features=sim-bench` (default off) `cargo test --all` runs in 12 s on the CI runner today (fmt+clippy+test sweep, no bench). Turning the threshold sweep on by default would either (a) slow down every PR's turnaround to the sim sweep duration (estimated 30–60 s, dominated by the 50-concurrency sweep), or (b) make the threshold sweep's failure mode just-another-failing-test that gets ignored. The opt-in feature makes the threshold sweep a **separate concern with its own CI job**, surfacing its results prominently in the PR status checks, and small enough to not block the routine `cargo test --all` gate. This mirrors how performance-sensitive open-source projects gate `cargo bench` PRs. ### 6.6 Why p99 (not p50) as the load-bearing assertion p50 = "the typical experience." p99 = "the worst acceptable case." The contact-center wedge (README §"Why it exists") rests on tight-tail-latency: a p50 of 60 ms with a p99 of 1.5 s is INDISTINGUISHABLE from a cloud CCaaS provider that averaged down to 60 ms but had bad tails. The assertion HAS to fire on the tail to be meaningful. p999 (or max) IS deferred — too noisy for CI gate assertions (one scheduler hiccup blowing past the ceiling would block every PR for unrelated reasons). p99 is the empirical sweet spot: sensitive enough to catch real regressions, lenient enough to survive CI runner variance. --- ## 7. Done-criteria 1. `cargo test --all` passes (stable + 1.85) — the routine gate, UNCHANGED. The sim-bench feature is opt-in; default `cargo test --all` does NOT run the threshold sweep. 2. `cargo fmt --check` + `cargo clippy -- -D warnings` clean on the new crate. 3. `cargo test --all --features=sim-bench` passes — the new gate, on stable. CI runs this in a separate `sim-bench` job per PR + nightly. 4. `cargo deny check` passes — no new dep conflicts (`toml` is already a workspace member; `serde` already workspace member). 5. `cargo doc --no-deps` renders the new `crates/rutster-sim/` cleanly with learner-facing comments per AGENTS.md code style. 6. Loop driver + rtc_session seam STILL holds: `loop_driver.rs` + `rtc_session.rs` byte-identical to slice-3 (CI pinned-blob gate from slice-4 Task 10 unchanged). The new `MediaCmd::RegisterSim` variant lives in `media_thread.rs`, NOT in the seam files. 7. The lance `loud-barge.toml` scenario passes the threshold sweep at all of [1, 10, 50] concurrency. 8. The `quiet-advisory.toml` scenario passes at 1 concurrency (the secondary-path focus). 9. The `sustained-call.toml` scenario's 3-barge sequence shows performance drift ≤ 1.5× across barges (anti-fatigue assertion). 10. Tick-lag gauge reads a `MediaStats.{tick_overruns, last_tick_micros}` value during the sweep and surfaces it in the SweepReport. 11. SweepReport's per-concurrency rows are logged to stderr in a structured format (CI failure messages are readable; "p99 kill-time at N=50: 84ms > 80ms" not "test_sim_thresholds failed"). 12. The single-thread-vs-threadpool question has a data-point answer documented ("slice 4½ found p99 tick-lag = Xms at 50 calls; the threadpool shard remains deferred / the threadpool shard should land now"). Even if the answer is "data confirms the deferral is fine for now," the decision is no longer vibe-based. --- ## 8. Open decisions ### 8.1 Should the harness also assert against the HTTP/WebSocket out-of-process surface? **Decision (slice 4½):** no. In-process measurement only. The out-of-process mode is a future post-spearhead refinement (paired with LLM-driven callers). Reasoning in §6.3. ### 8.2 Should the sim-bench CI job also run on the 1.85 toolchain? **Decision (slice 4½):** no. The matrix already runs `cargo test --all` on both. The sim-bench job is a stable-only opt-in feature; 1.85 doesn't get a second tier of benches. Rationale: the sim-bench feature is `cfg(feature)` — feature-gated code paths need their own gates, not a toolchain proliferation. ### 8.3 Threshold values: hardcodedconsts vs env-overridable for the CI runner operator? **Decision (slice 4½):** hardcoded consts in `thresholds.rs`. Post-spearhead, with the per-environment tuning framework (paired with slice-4 §1.2 VAD-threshold tuning deferral), they become env-driven. Hardcoded now makes the budget-vs-assertion-slack reasoning (§5.1) explicit in source — not subject to runner-env drift. ### 8.4 Should `MediaCmd::RegisterSim` carry a `pipe: Box` or a more structured "sim descriptor" the media thread materializes? **Decision (slice 4½):** `Box` — same shape as the existing `RtcSession`'s pipe construction (which has `pipe: Box`). A "sim descriptor" would add a layer of indirection the harness doesn't benefit from; the harness already constructs the `SimAudioPipe` and is fully prepared to ship it across the channel. The single variant is the minimal seam. ### 8.5 Concurrency-sweep sample size / iteration count per concurrency level **Decision (slice 4½):** 1000 ticks per concurrency level (the meta-tick count for the sweep durations). At 10 ms per tick = 10 seconds of sweep per concurrency × 3 levels = 30s sweep total. Sufficient sample size for a stable p99 (n = ~1000 ticks); bounded enough to keep CI fast. ### 8.6 Should the harness record per-call audio (WAV capture of the `SimAudioPipe`'s frames) for supervisor review? **Decision (slice 4½):** no. That's a rung-2 escalation feature (warm-handoff artifact). The harness measures; recording is a separate concern. The `SimAudioPipe` exposes a `take_captures()` API for the `LatencyProbe` only; raw frame capture is out of scope. --- ## 9. Cross-references - [ADR-0010](../../adr/0010-spearhead-benchmark-sim-harness.md) — centralized rationale: why 4½ exists, what it should produce, what it should defer (LLM callers), how it pairs the timing-thread debt. - [slice-4 spec §5.1](2026-07-01-slice-4-barge-in-design.md) — the ≤60 ms kill budget + the latency-arithmetic this slice asserts against. - [slice-1 spec §8.5 #6](2026-06-28-slice-1-webrtc-loopback-design.md) — the seam gate (`loop_driver.rs` + `rtc_session.rs` byte-identical) slice 4½ re-affirms (NO changes to those files). - [slice-5/seams plan](../plans/2026-07-04-slice-5-scalability-seams.md) (the infrastructure pre-paving this slice consumes) — `MediaCmd::Stats` exposes `MediaStats { tick_overruns, last_tick_micros }`, the readout this slice's concurrency sweep surfaces. - [ADR-0002](../../adr/0002-north-star-and-fused-core.md) — the fused vertical; the in-process measurement IS the fused-vertical seam (no gRPC hop between the harness + the loop). - [ADR-0008](../../adr/0008-fob-and-green-zone.md) — FOB/green-zone doctrine; the harness is a FOB member (hot-path-adjacent + differentiating). No green-zone dep added. - [PORT_PLAN.md §Phasing](../../PORT_PLAN.md) — step 4½ = sim harness (per ADR-0010 insertion).