rutster/docs/superpowers/specs/2026-07-05-slice-4-half-benchmark-sim-design.md

Rutster slice 4½ — Benchmark + simulation harness (the rutster-sim seed)

• Status: Draft (pending review)
• Date: 2026-07-05
• Spearhead step: 4½ of 6 (inserted by ADR-0010 — "after barge-in (4), before rented transport (5)")
• Origin: 2026-07-03 adversarial review (recommendations R1+R2) + 2026-07-03 market feature scan (F1 — simulation as the standout engine-fit feature)
• Depends on (already merged):
  • slice 1 — WebRTC media loopback — the media core + the AudioPipe trait slice 4½ simulates against
  • slice 2 — The agent tap — the TapAudioPipe slice 4½ measures end-to-end
  • 2026-06-30 slice-3 realtime brain (merged c30a452) — MockRealtimeBrain slice 4½ drives the harness against
  • slice 4 — Barge-in / VAD-driven playout kill — the ≤60 ms kill budget this slice MEASURES (slice-4 §5.1)
  • 2026-07-04 slice-5/seams — MediaCmd::Stats exposes MediaStats { tick_overruns, last_tick_micros } (the "tick-lag gauge" readout this harness surfaces as the primary concurrency-sweep result)
• Related: ADR-0002 (the wedge: "local real-time reflexes that don't need the brain" — currently an arithmetic claim, this slice proves it), ADR-0008 (the harness is FOB: hot-path-adjacent + differentiating), ARCHITECTURE.md §"Biggest technical risk" (the reflex loop is the long pole; the harness is its measurement surface)

---

TL;DR

Stand up spearhead step 4½: a self-hostable benchmark + simulation harness in a new crates/rutster-sim/ crate. It drives synthetic callers through the SAME media-leg path real callers use, measures p50/p99 mouth-to-ear latency and barge-in kill-time against slice-4's ≤60 ms kill budget, and runs the same measurements at 1 / 10 / 50 concurrent synthetic calls. A separate CI job (opt-in via --features=sim-bench) asserts the per-budget thresholds every commit. A latency regression fails the build the same way a broken test does — ADR-0010's central demand.

The harness is also the doctrine-drift detector for the timing-thread debt (slice-4 §1.2 deferred per-session threads; slice-5/seams landed a single shared std::thread + a tick-lag gauge without data). 4½'s concurrency sweep turns MediaStats.tick_overruns from a counter into a decision artifact: "does single-thread poll loop breach budget at realistic concurrency?" gets answered with data, not vibes. If yes, the dedicated threadpool-shard graduation (slice-4 deferral #2) gets scheduled on evidence.

Three things this slice deliberately is NOT:

• Not an LLM-driven caller (the scenario pack is scripted PCM segments; LLM-driven callers are post-spearhead). The harness ships with three deterministic scenarios that exercise the wedge.
• Not a fuzz target (the harness exercises real latency under load; it is not designed to find panics in the PCM path — that's the deferred cargo-fuzz harness).
• Not a load test for production capacity (p99 under 50 synthetic calls is the sperhead-scale limit; load testing for horizontally-scaled fleet deployments lands with the rung-2 escalation / step-6 spend-cap work).

---

1. Scope

1.1 In scope

• A new crates/rutster-sim/ crate (currently non-existent; this slice creates it from Cargo.toml
  • lib.rs).
• A Scenario data type + a TOML scenario file format (crates/rutster-sim/scenarios/*.toml) describing scripted caller PCM playback with timing directives ("speak N frames, pause M frames, interrupt at T"). At least three shipped scenarios: loud-barge.toml, quiet-advisory.toml, sustained-call.toml.
• A SimAudioPipe: AudioPipe implementation that drives a Scenario on next_pcm_frame (plays back scripted PCM frames) and captures received frames on on_pcm_frame (timestamps each — the latency-measurement boundary).
• A LatencyProbe that computes the two metrics slice-4's design sets targets for:
  • Mouth-to-ear round-trip — caller-speech-onset → first frame returned at the caller's "ear" (the SimAudioPipe's source path).
  • Barge-in kill-time — caller-speech-onset → reflex-barge-fire (the moment next_pcm_frame first returns None post-barge).
• A ConcurrencyRunner that spawns N concurrent SimCalls against a target binary URL (default: in-process via MediaThread for determinism) and computes per-call latencies + the p50/p99 aggregates across the sample.
• Tick-lag gauge integration: the runner reads MediaStats { tick_overruns, last_tick_micros } via MediaCmd::Stats (already exposed by slice-5/seams) and surfaces both as primary readouts in the sweep report — both the per-call last_tick_micros distribution and the cumulative tick_overruns count.
• CI-regressed thresholds: a separate CI job runs cargo test --all --features=sim-bench per PR + nightly; thresholds asserted; a regression fails the build.
• New learner-facing comments explaining the measurement discipline (the test corpus is the code-reading curriculum for "how do you measure a real-time system without lying to yourself?" — the slice-1 §7 verbosity standard carries over).

1.2 Out of scope (with scheduled return)

Deferred item	Returns in	Why deferred
LLM-driven synthetic callers	post-spearhead refinement	ADR-0010 says scripted scenarios are 4½; LLM callers are an extension. Scripted scenarios are deterministic (reproducibility is the point); LLM callers add variance that complicates threshold-assertion CI gates.
Per-environment calibration of threshold values	post-spearhead	The MVP ships thresholds tuned for the dev loopback (smoke) + CI runner (deterministic box). Real-world noise calibration (per-CPU, per-RAM) is a tuning-framework problem — paired with slice-4's VAD-threshold tuning deferral.
Multi-perspective scenario recording (the caller's audio + the operator's audio + the brain's audio saved per-run as WAV for review)	later rung (rung-2 escalation)	Recording per-call audio for supervisor review is a rung-2 capability (warm-handoff artifact). The harness measures; recording is a separate concern.
Distributed / multi-binary fleet latency sweep	rung 3 (post-escalation)	ADR-0010's sweep targets a single binary at low concurrency. Fleet-scale (N binaries, NAT, autoscaling) lands after the trunk is real (step 5) + escalation is real (rung 2).
Adaptive VAD_RMS_THRESHOLD tuning	post-spearhead	Slice-4 §1.2 defers; 4½ inherits the const threshold (500.0). Real-world noise-floor learning is a later rung.
Concurrency > 50 calls	later rung	The spearhead's scale envelope caps at "dozens of PSTN calls on one box" (slice-4 §6.3); 50 is the upper claim. Beyond 50 ⇒ fleet (above), not single-binary.
Latency under degraded brain conditions (mock brain stalls)	later refinement	The MVP measures against MockRealtimeBrain (deterministic). The real latency risk is "what if the brain takes 800 ms instead of 300 ms?" — addressed by a "brain slow" scenario ADDED LATER, not in the MVP.
Hooking cargo bench / criterion into the CI artifact publishing	later	False precision: the harness is cargo test --features=sim-bench, not cargo bench. We are asserting threshold-shape gates, not micro-bench-diff regressions (which are noisy in CI). criterion is a different epistemology.
Browser-based e2e (Playwright/Selenium)	post-spearhead	Unchanged from prior slices' deferral. The synthetic-peer harness is the test vehicle.

---

2. Architecture delta

2.1 What changes vs slice-4 + slice-5/seams

Slice 4½ adds ONE crate + one CI matrix entry. The fused vertical's existing hot path (loop_driver.rs + rtc_session.rs + MediaThread + Reflex<P> + TapAudioPipe) is untouched. The harness drives the same media-leg ingress path a real caller uses — so what it measures is what the customer experiences, by construction.

                       ┌─ crates/rutster-sim/ (NEW, this slice) ──────────┐
                       │                                                   │
                       │  Scenario(.toml)  ──►  SimAudioPipe: AudioPipe    │
                       │                            │                      │
                       │  LatencyProbe  ◄── timestamps on frames passing  │
                       │                            through the pipe       │
                       │                                                   │
                       │  ConcurrencyRunner (1/10/50 SimCalls)             │
                       │       │                                           │
                       │       └─► MediaCmd::Stats (slice-5/seams already) │
                       │              reads tick_overruns / last_tick_micros│
                       └──────────────────────┬────────────────────────────┘
                                              │ drives via the existing
                                              │ media-leg ingress path
                                              ▼
┌────────────────────────────────────────────── Rutster trust boundary (FOB) ────────────┐
│  WebRTC ingress / Trunk ingress (slice-5)  →  RtcSession  →  Reflex<TapAudioPipe>      │
│      ↑                          ↑                            ↑        ↑              │
│      └── existing, untouched ───┘                            │        │              │
│                                                                │        │              │
│              loop_driver.rs + rtc_session.rs (byte-identical) ┘        │              │
│                                                                         │              │
│  MediaThread (slice-4) ─── MediaCmd::{Register, AcceptOffer, Delete,                   │
│      Shutdown, Stats, Drain, RegisterTrunk[slice-5]}                                   │
│                                                                                                       │
│  Tick-lag gauge: MediaStats { tick_overruns, last_tick_micros } (slice-5)              │
└───────────────────────────────────────────────────────────────────────────────────────────┘

2.2 The measurement boundary (load-bearing — gets it wrong, the harness lies)

A latency measurement is honest only if the timestamp is captured at the same point the customer experiences the audio. The harness employs TWO clocks:

• Caller-side onset timestamp (t_onset): captured inside SimAudioPipe::on_pcm_frame when a scenario-step directive says "speak now." This is the wall-clock the caller started speaking.
• Caller-side receipt timestamp (t_receipt): captured inside SimAudioPipe::next_pcm_frame when the brain's response PCM frame is returned to the "caller's ear" (the SimAudioPipe's source path returns Some(frame)).

Latency = t_receipt - t_onset. Both timestamps come from Instant::now() (monotonic, unaffected by NTP) within the same SimAudioPipe instance — so wall-clock skew between nodes is structurally eliminated. The harness measures the system's response to a synthetic caller, not "what time it is on the operator's box vs the caller's box."

This is the design choice that distinguishes the harness from "just add some Instant::now() calls in tracing." The SimAudioPipe IS the caller — it owns both the onset timestamp (it decided when to "speak") and the receipt timestamp (it observed when the system "replied"). The harness can't lie about latency because the only clock it uses is the caller's clock.

2.3 Why SimAudioPipe instead of a tracing-only approach

The alternative: instrument loop_driver.rs / rtc_session.rs / the tap with tracing::Span timestamps and aggregate them post-hoc. We reject this design because:

1. It violates the seam. loop_driver.rs + rtc_session.rs are byte-identical seams; timing instrumentation would either change those files (slice-1 §8.5 #6 violated) or layer on top via decorators (complex, flaky under load).
2. It measures wall-clock at multiple points, not customer experience. A tracing::Span that says "encode took 4 ms" tells us nothing about whether the brain's reply actually reached the caller's ear in ≤60 ms — there's queueing + jitter + WS transport between "encode" and "heard."
3. It produces observability dashboards, not CI gates. Operators need dashboards (separate concern — the EventSink from slice-5/seams emits per-call CDR-anchored fields); the spearhead needs regression-failing thresholds stated as test assertions, which is the harness's job.

The SimAudioPipe is the right architectural choice for CI-suite measurements. Tracing dashboards land with ADR-0005 Valkey wiring (a later rung) — both necessary, both different.

2.4 Why 1 / 10 / 50 (and not other values)

• 1 call isolates the baseline — the cold-path latency with zero concurrency pressure. If this regresses, the bug is in the loop itself, not in contention. Slice-4's §5.1 ≤60 ms kill budget is asserted here.
• 10 calls is the warm working set. Approximately the peak spearhead-scale; the std thread's 10 ms meta-tick comfortably fits 10 sessions per tick (each session costs <100 µs in loop_driver::drive), so this asserts budget at warm-but-uncontended conditions.
• 50 calls is the saturation point. ADR-0010's "single-poll-task head-of-line-blocking debt" (review P2) lives here: 50 sessions per 10 ms tick = 200 µs per session in the meta-tick before contention squeezes; if last_tick_micros stays under 10 ms the single-thread design holds; if tick_overruns grows past some threshold the dedicated-threadpool-shard graduation (slice-4 §1.2 deferral) gets its data-driven case.

We do NOT test 100/500/5000 — that's fleet-scale (rung 3). 50 is the upper edge of the spearhead's "one binary, one city" claim.

2.5 The CI gate shape

A regression-failing threshold is a test assertion. The crate crates/rutster-sim/ ships:

• #[cfg(feature = "sim-bench")] modules + tests — default OFF, so cargo test --all (the routine CI gate on every PR) stays fast (assertions of correctness, not measurement).
• A cargo test --all --features=sim-bench invocation in a SEPARATE CI job. This job runs on every PR + nightly. Failure ⇒ red X ⇒ PR does not merge.

The thresholds are encoded as Rust assert! statements in crates/rutster-sim/src/thresholds.rs:

// crates/rutster-sim/src/thresholds.rs

/// Slice-4 spec §5.1 + §7 done-criteria #8: kill-time budget is
/// ≤60 ms (3 debounce frames × 20 ms tick) + 1 tick to drain + apply.
/// Observer slack to make CI deterministic-but-not-flaky on a slow runner:
///   effective CI assertion ≤80 ms (60 ms budget + 20 ms slack).
pub const BARGE_IN_KILL_TIME_P99_MS: f64 = 80.0;

/// Slice-1 + slice-3 mouth-to-ear budget: 200 ms (slice-1 notification) +
/// 250 ms mock brain + 100 ms playout buffer. CI assertion ceiling:
/// 700 ms (allowance for CI runner variance against dev machine).
pub const MOUTH_TO_EAR_P99_MS: f64 = 700.0;

/// Slice-5/seams tick-lag gauge: the meta-tick must stay under 10 ms
/// (the loop's nominal period). At 1 call: ≤2 ms. At 50 calls: ≤10 ms.
/// Tick overruns (count of ticks exceeding 10 ms) at p50 across the sweep:
/// ≤1% of total ticks.
pub const TICK_LAG_MAX_MS: f64 = 10.0;
pub const TICK_OVERRUN_PCT_MAX: f64 = 1.0;

/// Concurrency-sweep sample sizes.
pub const SWEEP_CONCURRENCIES: &[usize] = &[1, 10, 50];

These are constants for the MVP. They become env-var configurable in a post-spearhead tuning-framework (paired with slice-4's VAD threshold tuning deferral).

---

3. Component design

3.1 Scenario + ScenarioStep

// crates/rutster-sim/src/scenario.rs

/// A scripted caller scenario. Read from TOML (a scenario file under
/// `crates/rutster-sim/scenarios/*.toml`). Deterministic by construction —
/// the entire point is reproducible thresholds in CI.
///
/// # Why TOML (not YAML, not RON)
///
/// `serde` + `toml` is already a workspace member (cargo-deny licenses track).
/// TOML keeps the scenario file readable as a one-shot script (a sequence
/// of named steps + numbers); YAML would invite flow-mapping complexity
/// the scenario format doesn't need.
#[derive(Debug, Clone, serde::Deserialize)]
pub struct Scenario {
    /// Human-readable identifier; shows up in the CI failure message.
    pub name: String,
    /// The sequence of caller-side actions; played front-to-back.
    pub steps: Vec<ScenarioStep>,
}

/// One axis of caller behavior. A scenario is a time-ordered sequence of
/// these. The `SimAudioPipe` consumes them in order during `on_pcm_frame`.
#[derive(Debug, Clone, serde::Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum ScenarioStep {
    /// Send N loud PCM frames (sample value 1000, well above VAD_RMS_THRESHOLD=500.0).
    /// Triggers the local VAD via slice-4's `LocalVadReflex::on_pcm_frame`.
    SpeakLoud { frames: u32 },
    /// Send N zero frames (sample value 0, well below VAD_RMS_THRESHOLD).
    /// Used by the quiet-caller advisory scenario: drives mock-brain advisory path.
    SpeakQuiet { frames: u32 },
    /// Insert N zero frames before the next step (silence pacing).
    Pause { frames: u32 },
    /// Wait until the harness receives M "ear" frames before advancing.
    /// Used as a barrier: the brain's reply must arrive before the next caller action.
    AwaitReply { frames: u32 },
    /// End the scenario. The `SimAudioPipe` returns None from next_pcm_frame thereafter.
    End,
}

Scenario file example (crates/rutster-sim/scenarios/loud-barge.toml):

# Drives the PRIMARY barge-in path (slice-4 §5.1). The caller says one loud
# burst of audio; the local VAD trips; playout dies; no brain advisory needed.
# Asserts the wedge-#1 path: "VAD killing TTS the instant the caller speaks,
# without the brain."

name = "loud-barge"

[[steps]]
kind = "speak_loud"
frames = 20  # 20 frames @ 20ms = 400 ms of speech; comfortably past the 60 ms debounce

[[steps]]
kind = "await_reply"
frames = 0  # barrier: the FOB should be muted at this point (Reflex::muted == true)

[[steps]]
kind = "end"

3.2 SimAudioPipe: AudioPipe

// crates/rutster-sim/src/sim_audio_pipe.rs

/// A test-double `AudioPipe` that simulates a caller. Drives a `Scenario`
/// on `on_pcm_frame` (the sink path: caller speaks); receives brain
/// response frames on `next_pcm_frame` (the source path: caller hears).
/// Captures `Instant::now()` at every meaningful event for the
/// `LatencyProbe` to consume.
///
/// # Why this is THE measurement boundary
///
/// Both clocks live INSIDE this pipe. The wall-clock the *caller* started
/// speaking is captured here (we decided when to "speak"); the wall-clock
/// the *caller* heard the reply is captured here (we observed the system's
/// reply). See §2.2 — the harness can't lie about latency because the only
/// clock it uses is the caller's.
pub struct SimAudioPipe {
    /// The scenario under playback.
    scenario: Scenario,
    /// Cursor into `scenario.steps`.
    step_idx: usize,
    /// Frames remaining in the current step (decrements per `on_pcm_frame`
    /// for `SpeakLoud`/`SpeakQuiet`/`Pause`; freezes for `AwaitReply`).
    step_frames_remaining: u32,
    /// Frames received from `next_pcm_frame` while in `AwaitReply`.
    /// When this reaches the step's `frames` target, advance.
    reply_frames_received: u32,
    /// Captured timestamps (the `LatencyProbe` consumes this via
    /// `take_captures()` after the run). Discarded on every `on_pcm_frame`
    /// call once the capture buffer is at capacity (bounded; hot-path).
    captures: Vec<Capture>,
    /// A pre-allocated frame returned from `next_pcm_frame` when we have a
    /// pending reply (the harness intercepts frames routed back through the
    /// existing media loop and returns them here). See §3.4 for how the
    /// `SimCall` wires this pipe to a real `MediaThread`.
    reply_ring: std::collections::VecDeque<PcmFrame>,
}

/// A timestamped event captured by the `SimAudioPipe`. Read by the
/// `LatencyProbe` post-run to compute p50/p99 latencies.
#[derive(Debug, Clone, Copy)]
pub enum Capture {
    /// The caller started speaking loudly (a `SpeakLoud` step began).
    CallerLoudOnset { at: Instant },
    /// The FOB killed playout (a `next_pcm_frame` call returned None
    /// immediately after a barge event). The wall-clock this slice cares
    /// about for kill-time.
    BargeKillObserved { at: Instant },
    /// The caller heard a brain reply (a `next_pcm_frame` returned Some
    /// after the barge cleared). The wall-clock this slice cares about
    /// for mouth-to-ear.
    CallerHeardReply { at: Instant },
}

3.2.1 The two key methods

// crates/rutster-sim/src/sim_audio_pipe.rs (continued)

impl AudioSource for SimAudioPipe {
    fn next_pcm_frame(&mut self) -> Option<PcmFrame> {
        // The source path: the caller's "ear." Brain replies (frames the
        // system produced on the egress side and routed back to us via the
        // SimCall's mpsc wiring — see §3.4) land here.
        match self.reply_ring.pop_front() {
            Some(frame) => {
                if self.is_in_await_reply_step() {
                    self.reply_frames_received += 1;
                    if self.reply_frames_received >= self.current_step_target() {
                        self.advance_step();
                    }
                }
                // Capture: this is the "caller heard" wall-clock.
                self.captures.push(Capture::CallerHeardReply { at: Instant::now() });
                Some(frame)
            }
            None => {
                // The reflex muted us (slice-4's Reflex<P>::muted == true).
                // Capture: this is the "barge kill observed" wall-clock.
                // (Only capture if we are mid-AwaitReply post-barge; see the
                // LatencyProbe's classification §3.3 for the dedup logic.)
                self.captures.push(Capture::BargeKillObserved { at: Instant::now() });
                None
            }
        }
    }
}

impl AudioSink for SimAudioPipe {
    fn on_pcm_frame(&mut self, _frame: PcmFrame) {
        // The sink path: the caller "speaks." The scenario drives here.
        // Decode the current step + emit the appropriate PCM signal.
        self.dispatch_step_action();
        // (We discard the inbound frame — the caller doesn't hear itself;
        // the SimCall's wiring pushes the caller-side frame into the
        // `tx_pcm_in` channel for the tap to forward to the brain.)
    }
}

3.3 LatencyProbe

// crates/rutster-sim/src/latency.rs

/// Computes the two metrics slice-4's design sets budgets for, from a
/// captured-stream of `Capture` events produced by a `SimAudioPipe`.
///
/// # Verification discipline
///
/// The probe is the single source of truth for "did latency regress?"
/// Assertions are made against `LatencyProbe::p99_kill_ms()` and
/// `LatencyProbe::p99_mouth_to_ear_ms()`. A failure here is the build-red
/// signal ADR-0010 demands.
pub struct LatencyProbe {
    captures: Vec<Capture>,
}

impl LatencyProbe {
    pub fn from_captures(captures: Vec<Capture>) -> Self { Self { captures } }

    /// Barge-in kill-time: caller-speech-onset → first `BargeKillObserved`.
    /// Returns Duration per call (we capture one onset+kill pair per barge).
    pub fn kill_times(&self) -> Vec<Duration> { /* pair captures */ }

    /// Mouth-to-ear: caller-speech-onset → next `CallerHeardReply`.
    pub fn mouth_to_ear_times(&self) -> Vec<Duration> { /* pair captures */ }

    pub fn p50_kill_ms(&self) -> f64 { percentile(&self.kill_times(), 50) }
    pub fn p99_kill_ms(&self) -> f64 { percentile(&self.kill_times(), 99) }
    pub fn p50_mouth_to_ear_ms(&self) -> f64 { percentile(&self.mouth_to_ear_times(), 50) }
    pub fn p99_mouth_to_ear_ms(&self) -> f64 { percentile(&self.mouth_to_ear_times(), 99) }
}

fn percentile(durations: &[Duration], pct: u8) -> f64 {
    if durations.is_empty() { return f64::NAN; }
    let mut sorted: Vec<u128> = durations.iter().map(|d| d.as_millis()).collect();
    sorted.sort_unstable();
    let idx = ((sorted.len() as f64 - 1.0) * (pct as f64 / 100.0)).round() as usize;
    sorted[idx] as f64
}

The LatencyProbe is post-hoc: a single SimCall runs to completion, the SimAudioPipe's captures are drained, the probe computes the metrics. No per-tick instrumentation cost during the hot path itself — just the Instant::now() calls inside SimAudioPipe::next_pcm_frame (which isn't the hot path anyway; it's the simulated-caller epilogue).

3.4 SimCall (the wiring) + ConcurrencyRunner

// crates/rutster-sim/src/runner.rs

/// One simulated call: a `SimAudioPipe` + the wiring to drive it against the
/// existing `MediaThread`. Single binary; no separate process.
pub struct SimCall {
    /// The scenario-driven caller pipe.
    pipe: SimAudioPipe,
    /// The `MediaThread` cmd_tx — we register a session, drive the pipe
    /// via `tx_pcm_in`, capture frame receipts in `next_pcm_frame`.
    media_cmd_tx: mpsc::Sender<MediaCmd>,
    /// Latency probe populated post-run.
    probe: Option<LatencyProbe>,
}

impl SimCall {
    pub async fn run(mut self) -> LatencyProbe {
        // 1. Register a session with the MediaThread.
        // 2. Wire self.pipe as the session's AudioPipe (MediaCmd::RegisterSim).
        // 3. Drive the scenario: each scenario step emits `on_pcm_frame`
        //    calls against the SimAudioPipe; the MediaThread's loop_driver
        //    echoes frames back via next_pcm_frame.
        // 4. On End: drain captures + return the probe.
        todo!("see §4 data flow")
    }
}

/// The concurrency sweep runner. Spawns N `SimCall`s in parallel (tokio),
/// awaits all, aggregates per-call latencies into the sweep report.
pub struct ConcurrencyRunner {
    /// Target binary in-process MediaThread cmd_tx. Passed in by the test fixture.
    media_cmd_tx: mpsc::Sender<MediaCmd>,
    /// Concurrency levels to sweep (slice-4½ hardcoded [1, 10, 50]).
    concurrencies: Vec<usize>,
}

impl ConcurrencyRunner {
    /// Run the full sweep; return the per-concurrency-level report.
    pub async fn run(&self, scenario: Scenario) -> SweepReport { /* ... */ }
}

/// The artifact feeding the CI assertions. The thresholds.rs asserts
/// `report.barge_kill_p99_ms <= BARGE_IN_KILL_TIME_P99_MS` etc.
#[derive(Debug)]
pub struct SweepReport {
    pub per_concurrency: Vec<PerConcurrencyReport>,
}

#[derive(Debug)]
pub struct PerConcurrencyReport {
    pub concurrency: usize,
    pub p50_kill_ms: f64,
    pub p99_kill_ms: f64,
    pub p50_mouth_to_ear_ms: f64,
    pub p99_mouth_to_ear_ms: f64,
    /// From slice-5/seams MediaCmd::Stats. The "doctrine-drift detector"
    /// for the timing-thread debt — ADR-0010's debt-pairing readout.
    pub max_tick_lag_micros: u64,
    pub tick_overruns: u64,
    pub total_ticks: u64,
    pub tick_overrun_pct: f64,
}

3.5 MediaCmd::RegisterSim (the seam — one new enum variant)

Slice-5/seams already exists with MediaCmd::Register, AcceptOffer, Delete, Shutdown, Stats, Drain. Slice 4½ adds ONE variant: RegisterSim, which lets a SimCall register a session whose AudioPipe is a SimAudioPipe instead of a WebRTC-backed RtcSession. This is the minimum extension to drive the harness without needing to spin up a real WebRTC peer.

// crates/rutster/src/media_thread.rs (extended this slice)

pub enum MediaCmd {
    // ... existing variants (unchanged from slice-5)
    /// slice-4½: harness-side session registration. The
    /// `SimAudioPipe` lives entirely on the binary side; no WebRTC
    /// handshake needed. The `tx_pcm_in` channel is the existing
    /// sink-input seam (the harness emits `on_pcm_frame(frame)` directly
    /// rather than the loop_driver pulling RTP + decoding first).
    RegisterSim {
        pipe: Box<dyn AudioPipe>,
        reply: oneshot::Sender<ChannelId>,
    },
}

The std thread's run_media_thread handles RegisterSim by inserting a synthetic "session" entry that drives the harness's SimAudioPipe through the same loop_driver::drive calls as real WebRTC sessions — the seam holds (loop_driver.rs + rtc_session.rs byte-identical). The harness measures what real callers experience because it drives the same code path.

3.6 Tick-lag gauge integration (the doctrine-drift detector)

Slice-5/seams added MediaStats { tick_overruns, last_tick_micros } to MediaCmd::Stats. Slice 4½ consumes both. The ConcurrencyRunner polls MediaCmd::Stats once per second during the sweep; the per-concurrency-level report carries:

• max_tick_lag_micros — the maximum observed last_tick_micros during the sweep at this concurrency level. Indicates "the worst tick the loop experienced."
• tick_overruns / total_ticks — cumulative count of ticks where last_tick_micros > 10_000 (10 ms); converts to percent. Indicates "what fraction of ticks overflowed."

The thresholds (TICK_LAG_MAX_MS = 10.0, TICK_OVERRUN_PCT_MAX = 1.0) are the answer to ADR-0010's "if the concurrency sweep shows the shared-tokio poll loop breaching budget at realistic call counts, the dedicated-timing-thread work gets scheduled on data, not vibes." If slice 4½'s sweep shows tick_overrun_pct > 1.0 at 50 calls, the FOB reflex loop's single-thread debt is real and graduates from doctrine to data — that's the slice's load-bearing finding regardless of whether the latency thresholds pass.

---

4. Data flow

4.1 Single SimCall (the unit of measurement)

1. SimCall::run() starts.
2. Send MediaCmd::RegisterSim { pipe: Box<SimAudioPipe>, reply } to media_cmd_tx.
3. The MediaThread handles RegisterSim:
   - Constructs a "synthetic session" entry in its HashMap.
   - The session's `pipe` field is the SimAudioPipe.
   - Subsequent loop_driver::drive(now) calls touch this session identically
     to a WebRTC session (the seam holds).
4. The harness drives the scenario:
   - For each SpeakLoud/SpeakQuiet step: the SimCall emits `pipe.on_pcm_frame(frame)` calls
     at the 20 ms tick cadence. loop_driver::drive's encode path is bypassed for the SimPipe
     (we wrote the frame directly into the pipe).
   - loop_driver::drive immediately treats the next_pcm_frame call as the source path: it
     pulls from the SimPipe's reply_ring (where brain replies populate when the SimCall
     sees them via the tap/engine path).
5. Brain replies (from MockRealtimeBrain or the in-process tap):
   - Routed back into the SimPipe's reply_ring via an mpsc the SimCall holds.
   - loop_driver::drive picks them up on next_pcm_frame, encodes if real-WebRTC, but for sim
     we just observe — captures CallerHeardReply timestamp.
6. Reflex barge-in (slice-4 already merged):
   - If the SimPipe emitted SpeakLoud frames, the LocalVadReflex<P> (which wraps the pipe
     in the session_map/MediaThread composition site, slice-4 Task 6) trips the SpeechStarted
     advisory → Reflex<P>::muted = true → next_pcm_frame returns None → capture
     BargeKillObserved timestamp.
7. On End step: harness stops driving, returns LatencyProbe.

4.2 Concurrency sweep (the doctrine detector)

1. ConcurrencyRunner::run(scenario) launches N Tokio tasks, each running SimCall::run
   against the SAME media_cmd_tx (shared media thread).
2. The MediaThread drives ALL N sessions per 10 ms meta-tick (slice-4 §2.2 unchanged shape).
3. Per second during the sweep: ConcurrencyRunner fires MediaCmd::Stats; accumulates
   tick_overruns + last_tick_micros samples.
4. On all SimCalls completing: aggregate per-call LatencyProbes → p50/p99 vector.
5. Build SweepReport with per-concurrency rows, asserting thresholds.rs constants.

4.3 Why in-process (not client-server)

The harness DOES NOT stand up the binary as a server + a separate sim-client process. Why:

1. Determinism. Loopback within the same process eliminates websocket / socket / TCP jitter as a confounder. The thresholds are assertions about the FOB reflex loop itself — not assertions about NTPD variance on the CI runner.
2. CI simplicity. cargo test --features=sim-bench runs in-process; no port binding, no test-orchestration container, no race against epoll initialization.
3. Direct seam access. The harness can construct a SimAudioPipe and ship it via MediaCmd::RegisterSim directly — same path the production binary would use if it had a "synthetic caller" feature, no client-server glue needed.

A separate client-server mode (true loopback against the binary's HTTP/WebSocket surface) IS deferred — it's needed when the harness gains LLM-driven callers (post-spearhead refinement) and needs network realism.

---

5. Measurement plan + thresholds

5.1 The budgets (concrete numbers)

Metric	Budget (slice-4 design)	CI assertion (slice 4½)	Rationale for slack
Barge-in kill-time, p99	≤60 ms (3 debounce × 20 ms + 1 drain tick)	≤80 ms	CI runner has known variance against dev; the budget is 60 ms; the assertion is 80 to avoid flakiness.
Mouth-to-ear round-trip, p99	≤200 ms slice-1 + ≤300 ms mock brain + ≤100 ms playout = ~600 ms	≤700 ms	Same logic; the mock brain is deterministic but the harness adds observer cost.
Tick-lag (max prev-poll duration)	unspecified	≤10 ms	The slice-5/seams META_TICK const; the invariant the assertion makes explicit.
Tick overruns (fraction of ticks > 10 ms)	unspecified	≤1%	At 50 calls × 1000 ticks each = ≥99% need to be ≤10 ms. Allows for one scheduling hiccup per ~99 well-behaved ticks.

5.2 Per-concurrency swept assertions

For each N ∈ [1, 10, 50]:
  run scenario loud-barge.toml against N concurrent SimCalls.
  assert p99_kill_ms <= 80 ms;
  assert p99_mouth_to_ear_ms <= 700 ms;
  assert max_tick_lag_micros <= 10_000;  // 10 ms
  assert tick_overrun_pct <= 1.0;

5.3 The scenarios (3 shipped)

Scenario	Path	What it asserts
loud-barge.toml	Caller speaks 20 loud frames → awaits reply → end.	The PRIMARY barge-in path (slice-4 §5.1): local VAD fires, kill within ≤80 ms at p99, NO brain advisory required.
quiet-advisory.toml	Caller speaks 20 quiet frames (sub-VAD-threshold) → awaits reply → end.	The SECONDARY barge-in path (slice-4 §5.2): brain advisory fires from MockRealtimeBrain, kill flows through slice-3 plumbing + slice-4 advisory_tx → Reflex.
sustained-call.toml	Caller speaks 10 loud → 10 quiet → 10 loud → 10 quiet → 10 loud → end (5 minutes of talk). The fatigue / sustained-load check.	Multi-barge: 3 SpeechStarted advisories should fire in sequence; the Reflex::barge_epoch increments 3×; latency is asserted across all three bars (the second + third bar shouldn't drift > 1.5× the first).

5.4 CI integration

The CI workflow gains a new job:

# .github/workflows/ci.yml (additive)

  sim-bench:
    name: sim-bench (stable)
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - uses: Swatinem/rust-cache@v2
      - name: Install libopus
        run: sudo apt-get update && sudo apt-get install -y libopus-dev
      - name: Run sim-bench threshold sweep
        run: cargo test --all --features=sim-bench -- --test-threads=1

Notes:

• --test-threads=1 — concurrent sim-bench tests would pollute each other's MediaStats polling (the tick-lag gauge measures the SHARED media thread; concurrent runs of the sweep would contaminate each other).
• Run on stable only — the matrix already runs cargo test --all on stable + 1.85; the bench feature lives on stable.

5.5 The thresholds-as-test contract

// crates/rutster-sim/src/thresholds.rs (continuation — the CI test entry)

#[cfg(test)]
mod threshold_assertions {
    use super::*;

    #[tokio::test]
    #[cfg(feature = "sim-bench")]
    async fn loud_barge_at_each_concurrency_passes_thresholds() {
        let scenario = Scenario::load("scenarios/loud-barge.toml").unwrap();
        for &n in SWEEP_CONCURRENCIES {
            let report = ConcurrencyRunner::in_process(n).run(scenario.clone()).await;
            let row = report.per_concurrency.iter()
                .find(|r| r.concurrency == n).expect("concurrency row");

            assert!(
                row.p99_kill_ms <= BARGE_IN_KILL_TIME_P99_MS,
                "p99 kill-time at N={}: {}ms > {}ms (budget overflow)",
                n, row.p99_kill_ms, BARGE_IN_KILL_TIME_P99_MS,
            );
            assert!(
                row.p99_mouth_to_ear_ms <= MOUTH_TO_EAR_P99_MS,
                "p99 mouth-to-ear at N={}: {}ms > {}ms",
                n, row.p99_mouth_to_ear_ms, MOUTH_TO_EAR_P99_MS,
            );
            assert!(
                (row.max_tick_lag_micros as f64) / 1000.0 <= TICK_LAG_MAX_MS,
                "max tick-lag at N={}: {}us > {}ms",
                n, row.max_tick_lag_micros, TICK_LAG_MAX_MS,
            );
            assert!(
                row.tick_overrun_pct <= TICK_OVERRUN_PCT_MAX,
                "tick overrun % at N={}: {}% > {}%",
                n, row.tick_overrun_pct, TICK_OVERRUN_PCT_MAX,
            );
        }
    }

    #[tokio::test]
    #[cfg(feature = "sim-bench")]
    async fn quiet_advisory_at_1_concurrency_passes_thresholds() { /* ... */ }

    #[tokio::test]
    #[cfg(feature = "sim-bench")]
    async fn sustained_call_multibarge_does_not_drift() { /* ... */ }
}

---

6. Why these decisions

6.1 Why a new crate (crates/rutster-sim/) instead of in-tree tests

A test under crates/rutster/tests/sim_*.rs would absorb the harness into the binary crate — encouraging the harness to depend on binary internals. The separate crate keeps the dependency direction: rutster-sim → rutster-media + rutster-call-model + rutster (for MediaCmd). The sim crate is a FOB member per ADR-0008: hot-path-adjacent (drives the loop), differentiating (the proof artifact). It earns cratehood the same way rutster-tap did.

6.2 Why scripted scenarios, not LLM-driven callers (deferred)

• Reproducibility is the entire point. A CI gate that asserts "p99 ≤ 80 ms" can't be flaky on LLM variance — the LLM might take 2 s to respond, the threshold fails, the build is red, the dev attributes it to the LLM provider. The signal is corrupted.
• Isolation of what's being measured: scripted scenarios measure the FOB + the loop + the reflex; LLM callers introduce brain-side variance that isn't a FOB property.
• LLM-driven callers land in a post-spearhead refinement tier (ADR-0010 explicit deferral). They measure a different question ("how does it feel to talk to my brain?") than the MVP harness asks ("does the FOB reflex loop meet its budget under load?").

6.3 Why in-process, not client-server (the false-precision risk)

A client-server test harness gives Network-Realism™ but takes away the measurement discipline: a TCP retransmit on the CI runner would inflate p99 unattributably, and a "drift over threshold" failure becomes a triage time-sink ("is the FOB regressing, or did the CI runner's network blip?"). The in-process mode gives single-process determinism at the cost of network realism — which the spearhead explicitly doesn't need (it's measuring the FOB loop, not the deployed system). A future post-spearhead tier adds a client-server mode for integration realism + a different set of thresholds.

6.4 Why tick-lag as a primary readout (the doctrine-drift contract)

The single-thread media loop (slice-4 §6.3) is a spearhead-scale decision explicitly deferred to a "load demands it" trigger. Without the harness measuring it under load, the deferral has no data-driven graduation criterion — it'd be doctrine ("faster per-session threads later") instead of evidence ("measured tick_overrun_pct = 12% at 50 calls, the threadpool shard lands now"). ADR-0010 explicitly pairs the debt — MediaStats.tick_overruns exists because of slice-5/seams' seam work; slice 4½ makes the gauge OBSERVED. Both pass and the single-thread design is validated; both breach and the graduation is scheduled with data.

6.5 Why --features=sim-bench (default off)

cargo test --all runs in 12 s on the CI runner today (fmt+clippy+test sweep, no bench). Turning the threshold sweep on by default would either (a) slow down every PR's turnaround to the sim sweep duration (estimated 30–60 s, dominated by the 50-concurrency sweep), or (b) make the threshold sweep's failure mode just-another-failing-test that gets ignored. The opt-in feature makes the threshold sweep a separate concern with its own CI job, surfacing its results prominently in the PR status checks, and small enough to not block the routine cargo test --all gate. This mirrors how performance-sensitive open-source projects gate cargo bench PRs.

6.6 Why p99 (not p50) as the load-bearing assertion

p50 = "the typical experience." p99 = "the worst acceptable case." The contact-center wedge (README §"Why it exists") rests on tight-tail-latency: a p50 of 60 ms with a p99 of 1.5 s is INDISTINGUISHABLE from a cloud CCaaS provider that averaged down to 60 ms but had bad tails. The assertion HAS to fire on the tail to be meaningful.

p999 (or max) IS deferred — too noisy for CI gate assertions (one scheduler hiccup blowing past the ceiling would block every PR for unrelated reasons). p99 is the empirical sweet spot: sensitive enough to catch real regressions, lenient enough to survive CI runner variance.

---

7. Done-criteria

1. cargo test --all passes (stable + 1.85) — the routine gate, UNCHANGED. The sim-bench feature is opt-in; default cargo test --all does NOT run the threshold sweep.
2. cargo fmt --check + cargo clippy -- -D warnings clean on the new crate.
3. cargo test --all --features=sim-bench passes — the new gate, on stable. CI runs this in a separate sim-bench job per PR + nightly.
4. cargo deny check passes — no new dep conflicts (toml is already a workspace member; serde already workspace member).
5. cargo doc --no-deps renders the new crates/rutster-sim/ cleanly with learner-facing comments per AGENTS.md code style.
6. Loop driver + rtc_session seam STILL holds: loop_driver.rs + rtc_session.rs byte-identical to slice-3 (CI pinned-blob gate from slice-4 Task 10 unchanged). The new MediaCmd::RegisterSim variant lives in media_thread.rs, NOT in the seam files.
7. The lance loud-barge.toml scenario passes the threshold sweep at all of [1, 10, 50] concurrency.
8. The quiet-advisory.toml scenario passes at 1 concurrency (the secondary-path focus).
9. The sustained-call.toml scenario's 3-barge sequence shows performance drift ≤ 1.5× across barges (anti-fatigue assertion).
10. Tick-lag gauge reads a MediaStats.{tick_overruns, last_tick_micros} value during the sweep and surfaces it in the SweepReport.
11. SweepReport's per-concurrency rows are logged to stderr in a structured format (CI failure messages are readable; "p99 kill-time at N=50: 84ms > 80ms" not "test_sim_thresholds failed").
12. The single-thread-vs-threadpool question has a data-point answer documented ("slice 4½ found p99 tick-lag = Xms at 50 calls; the threadpool shard remains deferred / the threadpool shard should land now"). Even if the answer is "data confirms the deferral is fine for now," the decision is no longer vibe-based.

---

8. Open decisions

8.1 Should the harness also assert against the HTTP/WebSocket out-of-process surface?

Decision (slice 4½): no. In-process measurement only. The out-of-process mode is a future post-spearhead refinement (paired with LLM-driven callers). Reasoning in §6.3.

8.2 Should the sim-bench CI job also run on the 1.85 toolchain?

Decision (slice 4½): no. The matrix already runs cargo test --all on both. The sim-bench job is a stable-only opt-in feature; 1.85 doesn't get a second tier of benches. Rationale: the sim-bench feature is cfg(feature) — feature-gated code paths need their own gates, not a toolchain proliferation.

8.3 Threshold values: hardcodedconsts vs env-overridable for the CI runner operator?

Decision (slice 4½): hardcoded consts in thresholds.rs. Post-spearhead, with the per-environment tuning framework (paired with slice-4 §1.2 VAD-threshold tuning deferral), they become env-driven. Hardcoded now makes the budget-vs-assertion-slack reasoning (§5.1) explicit in source — not subject to runner-env drift.

8.4 Should MediaCmd::RegisterSim carry a pipe: Box<dyn AudioPipe> or a more structured

"sim descriptor" the media thread materializes?

Decision (slice 4½): Box<dyn AudioPipe> — same shape as the existing RtcSession's pipe construction (which has pipe: Box<dyn AudioPipe>). A "sim descriptor" would add a layer of indirection the harness doesn't benefit from; the harness already constructs the SimAudioPipe and is fully prepared to ship it across the channel. The single variant is the minimal seam.

8.5 Concurrency-sweep sample size / iteration count per concurrency level

Decision (slice 4½): 1000 ticks per concurrency level (the meta-tick count for the sweep durations). At 10 ms per tick = 10 seconds of sweep per concurrency × 3 levels = 30s sweep total. Sufficient sample size for a stable p99 (n = ~1000 ticks); bounded enough to keep CI fast.

8.6 Should the harness record per-call audio (WAV capture of the SimAudioPipe's frames) for

supervisor review?

Decision (slice 4½): no. That's a rung-2 escalation feature (warm-handoff artifact). The harness measures; recording is a separate concern. The SimAudioPipe exposes a take_captures() API for the LatencyProbe only; raw frame capture is out of scope.

---

9. Cross-references

• ADR-0010 — centralized rationale: why 4½ exists, what it should produce, what it should defer (LLM callers), how it pairs the timing-thread debt.
• slice-4 spec §5.1 — the ≤60 ms kill budget + the latency-arithmetic this slice asserts against.
• slice-1 spec §8.5 #6 — the seam gate (loop_driver.rs
  • rtc_session.rs byte-identical) slice 4½ re-affirms (NO changes to those files).
• slice-5/seams plan (the infrastructure pre-paving this slice consumes) — MediaCmd::Stats exposes MediaStats { tick_overruns, last_tick_micros }, the readout this slice's concurrency sweep surfaces.
• ADR-0002 — the fused vertical; the in-process measurement IS the fused-vertical seam (no gRPC hop between the harness + the loop).
• ADR-0008 — FOB/green-zone doctrine; the harness is a FOB member (hot-path-adjacent + differentiating). No green-zone dep added.
• PORT_PLAN.md §Phasing — step 4½ = sim harness (per ADR-0010 insertion).