Aaron D. Lee
c31e8a5069
Co-authored-by: Aaron D. Lee <himself@adlee.work> Co-committed-by: Aaron D. Lee <himself@adlee.work>
890 lines
47 KiB
Markdown
890 lines
47 KiB
Markdown
# Rutster slice 4½ — Benchmark + simulation harness (the `rutster-sim` seed)
|
||
|
||
- **Status:** Draft (pending review)
|
||
- **Date:** 2026-07-05
|
||
- **Spearhead step:** 4½ of 6 (inserted by [ADR-0010](../../adr/0010-spearhead-benchmark-sim-harness.md) — "after barge-in (4), before rented transport (5)")
|
||
- **Origin:** [2026-07-03 adversarial review](../../reviews/2026-07-03-adversarial-review.md) (recommendations R1+R2) + [2026-07-03 market feature scan](../../reviews/2026-07-03-market-feature-scan.md) (F1 — simulation as the standout engine-fit feature)
|
||
- **Depends on (already merged):**
|
||
- [slice 1 — WebRTC media loopback](2026-06-28-slice-1-webrtc-loopback-design.md) — the media core + the `AudioPipe` trait slice 4½ simulates against
|
||
- [slice 2 — The agent tap](2026-06-28-slice-2-agent-tap-design.md) — the `TapAudioPipe` slice 4½ measures end-to-end
|
||
- 2026-06-30 slice-3 realtime brain (merged `c30a452`) — `MockRealtimeBrain` slice 4½ drives the harness against
|
||
- [slice 4 — Barge-in / VAD-driven playout kill](2026-07-01-slice-4-barge-in-design.md) — the ≤60 ms kill budget this slice MEASURES (slice-4 §5.1)
|
||
- 2026-07-04 slice-5/seams — `MediaCmd::Stats` exposes `MediaStats { tick_overruns, last_tick_micros }` (the "tick-lag gauge" readout this harness surfaces as the primary concurrency-sweep result)
|
||
- **Related:** [ADR-0002](../../adr/0002-north-star-and-fused-core.md) (the wedge: "local real-time reflexes that don't need the brain" — currently an arithmetic claim, this slice proves it), [ADR-0008](../../adr/0008-fob-and-green-zone.md) (the harness is FOB: hot-path-adjacent + differentiating), [ARCHITECTURE.md §"Biggest technical risk"](../../ARCHITECTURE.md) (the reflex loop *is* the long pole; the harness is its measurement surface)
|
||
|
||
---
|
||
|
||
## TL;DR
|
||
|
||
Stand up spearhead step 4½: a self-hostable **benchmark + simulation harness** in a new
|
||
`crates/rutster-sim/` crate. It drives synthetic callers through the SAME media-leg path real
|
||
callers use, measures **p50/p99 mouth-to-ear latency** and **barge-in kill-time** against slice-4's
|
||
≤60 ms kill budget, and runs the same measurements at **1 / 10 / 50 concurrent synthetic calls**.
|
||
A separate CI job (opt-in via `--features=sim-bench`) asserts the per-budget thresholds every
|
||
commit. **A latency regression fails the build the same way a broken test does** — ADR-0010's
|
||
central demand.
|
||
|
||
The harness is also the **doctrine-drift detector** for the timing-thread debt (slice-4 §1.2
|
||
deferred per-session threads; slice-5/seams landed a single shared std::thread + a tick-lag gauge
|
||
without data). 4½'s concurrency sweep turns `MediaStats.tick_overruns` from a counter into a
|
||
**decision artifact**: "does single-thread poll loop breach budget at realistic concurrency?" gets
|
||
answered with data, not vibes. If yes, the dedicated threadpool-shard graduation (slice-4 deferral
|
||
#2) gets scheduled on evidence.
|
||
|
||
Three things this slice deliberately is NOT:
|
||
- **Not an LLM-driven caller** (the scenario pack is scripted PCM segments; LLM-driven callers are
|
||
post-spearhead). The harness ships with three deterministic scenarios that exercise the wedge.
|
||
- **Not a fuzz target** (the harness exercises real latency under load; it is not designed to find
|
||
panics in the PCM path — that's the deferred `cargo-fuzz` harness).
|
||
- **Not a load test for production capacity** (p99 under 50 synthetic calls is the sperhead-scale
|
||
limit; load testing for horizontally-scaled fleet deployments lands with the rung-2 escalation /
|
||
step-6 spend-cap work).
|
||
|
||
---
|
||
|
||
## 1. Scope
|
||
|
||
### 1.1 In scope
|
||
|
||
- A new `crates/rutster-sim/` crate (currently non-existent; this slice creates it from `Cargo.toml`
|
||
+ `lib.rs`).
|
||
- A **`Scenario` data type** + a TOML scenario file format (`crates/rutster-sim/scenarios/*.toml`)
|
||
describing scripted caller PCM playback with timing directives ("speak N frames, pause M frames,
|
||
interrupt at T"). At least three shipped scenarios: `loud-barge.toml`, `quiet-advisory.toml`,
|
||
`sustained-call.toml`.
|
||
- A **`SimAudioPipe: AudioPipe`** implementation that drives a `Scenario` on `next_pcm_frame`
|
||
(plays back scripted PCM frames) and captures received frames on `on_pcm_frame` (timestamps
|
||
each — the latency-measurement boundary).
|
||
- A **`LatencyProbe`** that computes the two metrics slice-4's design sets targets for:
|
||
- **Mouth-to-ear round-trip** — caller-speech-onset → first frame returned at the caller's "ear"
|
||
(the `SimAudioPipe`'s source path).
|
||
- **Barge-in kill-time** — caller-speech-onset → reflex-barge-fire (the moment `next_pcm_frame`
|
||
first returns `None` post-barge).
|
||
- A **`ConcurrencyRunner`** that spawns N concurrent `SimCall`s against a target binary URL
|
||
(default: in-process via `MediaThread` for determinism) and computes per-call latencies + the
|
||
p50/p99 aggregates across the sample.
|
||
- **Tick-lag gauge integration**: the runner reads `MediaStats { tick_overruns, last_tick_micros }`
|
||
via `MediaCmd::Stats` (already exposed by slice-5/seams) and surfaces both as primary readouts in
|
||
the sweep report — both the per-call `last_tick_micros` distribution and the cumulative
|
||
`tick_overruns` count.
|
||
- **CI-regressed thresholds**: a separate CI job runs `cargo test --all --features=sim-bench` per
|
||
PR + nightly; thresholds asserted; a regression fails the build.
|
||
- New learner-facing comments explaining the measurement discipline (the test corpus *is* the
|
||
code-reading curriculum for "how do you measure a real-time system without lying to yourself?" —
|
||
the slice-1 §7 verbosity standard carries over).
|
||
|
||
### 1.2 Out of scope (with scheduled return)
|
||
|
||
| Deferred item | Returns in | Why deferred |
|
||
|---|---|---|
|
||
| LLM-driven synthetic callers | post-spearhead refinement | ADR-0010 says scripted scenarios are 4½; LLM callers are an extension. Scripted scenarios are deterministic (reproducibility is the point); LLM callers add variance that complicates threshold-assertion CI gates. |
|
||
| Per-environment calibration of threshold values | post-spearhead | The MVP ships thresholds tuned for the dev loopback (smoke) + CI runner (deterministic box). Real-world noise calibration (per-CPU, per-RAM) is a tuning-framework problem — paired with slice-4's VAD-threshold tuning deferral. |
|
||
| Multi-perspective scenario recording (the caller's audio + the operator's audio + the brain's audio saved per-run as WAV for review) | later rung (rung-2 escalation) | Recording per-call audio for supervisor review is a rung-2 capability (warm-handoff artifact). The harness measures; recording is a separate concern. |
|
||
| Distributed / multi-binary fleet latency sweep | rung 3 (post-escalation) | ADR-0010's sweep targets a single binary at low concurrency. Fleet-scale (N binaries, NAT, autoscaling) lands after the trunk is real (step 5) + escalation is real (rung 2). |
|
||
| Adaptive `VAD_RMS_THRESHOLD` tuning | post-spearhead | Slice-4 §1.2 defers; 4½ inherits the const threshold (500.0). Real-world noise-floor learning is a later rung. |
|
||
| Concurrency > 50 calls | later rung | The spearhead's scale envelope caps at "dozens of PSTN calls on one box" (slice-4 §6.3); 50 is the upper claim. Beyond 50 ⇒ fleet (above), not single-binary. |
|
||
| Latency under degraded brain conditions (mock brain stalls) | later refinement | The MVP measures against `MockRealtimeBrain` (deterministic). The real latency risk is "what if the brain takes 800 ms instead of 300 ms?" — addressed by a "brain slow" scenario ADDED LATER, not in the MVP. |
|
||
| Hooking `cargo bench` / criterion into the CI artifact publishing | later | False precision: the harness is `cargo test --features=sim-bench`, not `cargo bench`. We are asserting threshold-shape gates, not micro-bench-diff regressions (which are noisy in CI). `criterion` is a different epistemology. |
|
||
| Browser-based e2e (Playwright/Selenium) | post-spearhead | Unchanged from prior slices' deferral. The synthetic-peer harness is the test vehicle. |
|
||
|
||
---
|
||
|
||
## 2. Architecture delta
|
||
|
||
### 2.1 What changes vs slice-4 + slice-5/seams
|
||
|
||
Slice 4½ adds ONE crate + one CI matrix entry. The fused vertical's existing hot path
|
||
(`loop_driver.rs` + `rtc_session.rs` + `MediaThread` + `Reflex<P>` + `TapAudioPipe`) is **untouched**.
|
||
The harness drives the same media-leg ingress path a real caller uses — so what it measures is what
|
||
the customer experiences, by construction.
|
||
|
||
```
|
||
┌─ crates/rutster-sim/ (NEW, this slice) ──────────┐
|
||
│ │
|
||
│ Scenario(.toml) ──► SimAudioPipe: AudioPipe │
|
||
│ │ │
|
||
│ LatencyProbe ◄── timestamps on frames passing │
|
||
│ through the pipe │
|
||
│ │
|
||
│ ConcurrencyRunner (1/10/50 SimCalls) │
|
||
│ │ │
|
||
│ └─► MediaCmd::Stats (slice-5/seams already) │
|
||
│ reads tick_overruns / last_tick_micros│
|
||
└──────────────────────┬────────────────────────────┘
|
||
│ drives via the existing
|
||
│ media-leg ingress path
|
||
▼
|
||
┌────────────────────────────────────────────── Rutster trust boundary (FOB) ────────────┐
|
||
│ WebRTC ingress / Trunk ingress (slice-5) → RtcSession → Reflex<TapAudioPipe> │
|
||
│ ↑ ↑ ↑ ↑ │
|
||
│ └── existing, untouched ───┘ │ │ │
|
||
│ │ │ │
|
||
│ loop_driver.rs + rtc_session.rs (byte-identical) ┘ │ │
|
||
│ │ │
|
||
│ MediaThread (slice-4) ─── MediaCmd::{Register, AcceptOffer, Delete, │
|
||
│ Shutdown, Stats, Drain, RegisterTrunk[slice-5]} │
|
||
│ │
|
||
│ Tick-lag gauge: MediaStats { tick_overruns, last_tick_micros } (slice-5) │
|
||
└───────────────────────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 2.2 The measurement boundary (load-bearing — gets it wrong, the harness lies)
|
||
|
||
A latency measurement is honest only if the timestamp is captured at the same point the customer
|
||
experiences the audio. The harness employs TWO clocks:
|
||
|
||
- **Caller-side onset timestamp** (`t_onset`): captured inside `SimAudioPipe::on_pcm_frame` when a
|
||
scenario-step directive says "speak now." This is the wall-clock the *caller* started speaking.
|
||
- **Caller-side receipt timestamp** (`t_receipt`): captured inside `SimAudioPipe::next_pcm_frame`
|
||
when the brain's response PCM frame is returned to the "caller's ear" (the `SimAudioPipe`'s source
|
||
path returns `Some(frame)`).
|
||
|
||
Latency = `t_receipt - t_onset`. Both timestamps come from `Instant::now()` (monotonic, unaffected by
|
||
NTP) within the same `SimAudioPipe` instance — so wall-clock skew between nodes is structurally
|
||
eliminated. The harness measures *the system's response to a synthetic caller*, not "what time it
|
||
is on the operator's box vs the caller's box."
|
||
|
||
This is the design choice that distinguishes the harness from "just add some `Instant::now()` calls
|
||
in tracing." The `SimAudioPipe` IS the caller — it owns both the onset timestamp (it decided when
|
||
to "speak") and the receipt timestamp (it observed when the system "replied"). The harness can't
|
||
lie about latency because the only clock it uses is the caller's clock.
|
||
|
||
### 2.3 Why `SimAudioPipe` instead of a tracing-only approach
|
||
|
||
The alternative: instrument `loop_driver.rs` / `rtc_session.rs` / the tap with `tracing::Span`
|
||
timestamps and aggregate them post-hoc. We reject this design because:
|
||
|
||
1. **It violates the seam.** `loop_driver.rs` + `rtc_session.rs` are byte-identical seams; timing
|
||
instrumentation would either change those files (slice-1 §8.5 #6 violated) or layer on top via
|
||
decorators (complex, flaky under load).
|
||
2. **It measures wall-clock at multiple points, not customer experience.** A `tracing::Span` that
|
||
says "encode took 4 ms" tells us nothing about whether the brain's reply actually reached the
|
||
caller's ear in ≤60 ms — there's queueing + jitter + WS transport between "encode" and "heard."
|
||
3. **It produces observability dashboards, not CI gates.** Operators need dashboards (separate
|
||
concern — the `EventSink` from slice-5/seams emits per-call CDR-anchored fields); the spearhead
|
||
needs *regression-failing thresholds* stated as test assertions, which is the harness's job.
|
||
|
||
The `SimAudioPipe` is the right architectural choice for **CI-suite measurements**. Tracing
|
||
dashboards land with ADR-0005 Valkey wiring (a later rung) — both necessary, both different.
|
||
|
||
### 2.4 Why 1 / 10 / 50 (and not other values)
|
||
|
||
- **1 call** isolates the *baseline* — the cold-path latency with zero concurrency pressure. If
|
||
this regresses, the bug is in the loop itself, not in contention. Slice-4's §5.1 ≤60 ms kill budget
|
||
is asserted here.
|
||
- **10 calls** is the *warm working set.* Approximately the peak spearhead-scale; the std thread's
|
||
10 ms meta-tick comfortably fits 10 sessions per tick (each session costs <100 µs in
|
||
`loop_driver::drive`), so this asserts budget at warm-but-uncontended conditions.
|
||
- **50 calls** is the *saturation point.* ADR-0010's "single-poll-task head-of-line-blocking debt"
|
||
(review P2) lives here: 50 sessions per 10 ms tick = 200 µs per session in the meta-tick before
|
||
contention squeezes; if `last_tick_micros` stays under 10 ms the single-thread design holds; if
|
||
`tick_overruns` grows past some threshold the dedicated-threadpool-shard graduation (slice-4 §1.2
|
||
deferral) gets its data-driven case.
|
||
|
||
We do NOT test 100/500/5000 — that's fleet-scale (rung 3). 50 is the upper edge of the spearhead's
|
||
"one binary, one city" claim.
|
||
|
||
### 2.5 The CI gate shape
|
||
|
||
A regression-failing threshold is a test assertion. The crate `crates/rutster-sim/` ships:
|
||
|
||
- `#[cfg(feature = "sim-bench")]` modules + tests — default OFF, so `cargo test --all` (the routine
|
||
CI gate on every PR) stays fast (assertions of *correctness*, not measurement).
|
||
- A `cargo test --all --features=sim-bench` invocation in a SEPARATE CI job. This job runs on every
|
||
PR + nightly. Failure ⇒ red X ⇒ PR does not merge.
|
||
|
||
The thresholds are encoded as Rust `assert!` statements in `crates/rutster-sim/src/thresholds.rs`:
|
||
|
||
```rust
|
||
// crates/rutster-sim/src/thresholds.rs
|
||
|
||
/// Slice-4 spec §5.1 + §7 done-criteria #8: kill-time budget is
|
||
/// ≤60 ms (3 debounce frames × 20 ms tick) + 1 tick to drain + apply.
|
||
/// Observer slack to make CI deterministic-but-not-flaky on a slow runner:
|
||
/// effective CI assertion ≤80 ms (60 ms budget + 20 ms slack).
|
||
pub const BARGE_IN_KILL_TIME_P99_MS: f64 = 80.0;
|
||
|
||
/// Slice-1 + slice-3 mouth-to-ear budget: 200 ms (slice-1 notification) +
|
||
/// 250 ms mock brain + 100 ms playout buffer. CI assertion ceiling:
|
||
/// 700 ms (allowance for CI runner variance against dev machine).
|
||
pub const MOUTH_TO_EAR_P99_MS: f64 = 700.0;
|
||
|
||
/// Slice-5/seams tick-lag gauge: the meta-tick must stay under 10 ms
|
||
/// (the loop's nominal period). At 1 call: ≤2 ms. At 50 calls: ≤10 ms.
|
||
/// Tick overruns (count of ticks exceeding 10 ms) at p50 across the sweep:
|
||
/// ≤1% of total ticks.
|
||
pub const TICK_LAG_MAX_MS: f64 = 10.0;
|
||
pub const TICK_OVERRUN_PCT_MAX: f64 = 1.0;
|
||
|
||
/// Concurrency-sweep sample sizes.
|
||
pub const SWEEP_CONCURRENCIES: &[usize] = &[1, 10, 50];
|
||
```
|
||
|
||
These are *constants* for the MVP. They become *env-var configurable* in a post-spearhead
|
||
tuning-framework (paired with slice-4's VAD threshold tuning deferral).
|
||
|
||
---
|
||
|
||
## 3. Component design
|
||
|
||
### 3.1 `Scenario` + `ScenarioStep`
|
||
|
||
```rust
|
||
// crates/rutster-sim/src/scenario.rs
|
||
|
||
/// A scripted caller scenario. Read from TOML (a scenario file under
|
||
/// `crates/rutster-sim/scenarios/*.toml`). Deterministic by construction —
|
||
/// the entire point is reproducible thresholds in CI.
|
||
///
|
||
/// # Why TOML (not YAML, not RON)
|
||
///
|
||
/// `serde` + `toml` is already a workspace member (cargo-deny licenses track).
|
||
/// TOML keeps the scenario file readable as a one-shot script (a sequence
|
||
/// of named steps + numbers); YAML would invite flow-mapping complexity
|
||
/// the scenario format doesn't need.
|
||
#[derive(Debug, Clone, serde::Deserialize)]
|
||
pub struct Scenario {
|
||
/// Human-readable identifier; shows up in the CI failure message.
|
||
pub name: String,
|
||
/// The sequence of caller-side actions; played front-to-back.
|
||
pub steps: Vec<ScenarioStep>,
|
||
}
|
||
|
||
/// One axis of caller behavior. A scenario is a time-ordered sequence of
|
||
/// these. The `SimAudioPipe` consumes them in order during `on_pcm_frame`.
|
||
#[derive(Debug, Clone, serde::Deserialize)]
|
||
#[serde(tag = "kind", rename_all = "snake_case")]
|
||
pub enum ScenarioStep {
|
||
/// Send N loud PCM frames (sample value 1000, well above VAD_RMS_THRESHOLD=500.0).
|
||
/// Triggers the local VAD via slice-4's `LocalVadReflex::on_pcm_frame`.
|
||
SpeakLoud { frames: u32 },
|
||
/// Send N zero frames (sample value 0, well below VAD_RMS_THRESHOLD).
|
||
/// Used by the quiet-caller advisory scenario: drives mock-brain advisory path.
|
||
SpeakQuiet { frames: u32 },
|
||
/// Insert N zero frames before the next step (silence pacing).
|
||
Pause { frames: u32 },
|
||
/// Wait until the harness receives M "ear" frames before advancing.
|
||
/// Used as a barrier: the brain's reply must arrive before the next caller action.
|
||
AwaitReply { frames: u32 },
|
||
/// End the scenario. The `SimAudioPipe` returns None from next_pcm_frame thereafter.
|
||
End,
|
||
}
|
||
```
|
||
|
||
Scenario file example (`crates/rutster-sim/scenarios/loud-barge.toml`):
|
||
|
||
```toml
|
||
# Drives the PRIMARY barge-in path (slice-4 §5.1). The caller says one loud
|
||
# burst of audio; the local VAD trips; playout dies; no brain advisory needed.
|
||
# Asserts the wedge-#1 path: "VAD killing TTS the instant the caller speaks,
|
||
# without the brain."
|
||
|
||
name = "loud-barge"
|
||
|
||
[[steps]]
|
||
kind = "speak_loud"
|
||
frames = 20 # 20 frames @ 20ms = 400 ms of speech; comfortably past the 60 ms debounce
|
||
|
||
[[steps]]
|
||
kind = "await_reply"
|
||
frames = 0 # barrier: the FOB should be muted at this point (Reflex::muted == true)
|
||
|
||
[[steps]]
|
||
kind = "end"
|
||
```
|
||
|
||
### 3.2 `SimAudioPipe: AudioPipe`
|
||
|
||
```rust
|
||
// crates/rutster-sim/src/sim_audio_pipe.rs
|
||
|
||
/// A test-double `AudioPipe` that simulates a caller. Drives a `Scenario`
|
||
/// on `on_pcm_frame` (the sink path: caller speaks); receives brain
|
||
/// response frames on `next_pcm_frame` (the source path: caller hears).
|
||
/// Captures `Instant::now()` at every meaningful event for the
|
||
/// `LatencyProbe` to consume.
|
||
///
|
||
/// # Why this is THE measurement boundary
|
||
///
|
||
/// Both clocks live INSIDE this pipe. The wall-clock the *caller* started
|
||
/// speaking is captured here (we decided when to "speak"); the wall-clock
|
||
/// the *caller* heard the reply is captured here (we observed the system's
|
||
/// reply). See §2.2 — the harness can't lie about latency because the only
|
||
/// clock it uses is the caller's.
|
||
pub struct SimAudioPipe {
|
||
/// The scenario under playback.
|
||
scenario: Scenario,
|
||
/// Cursor into `scenario.steps`.
|
||
step_idx: usize,
|
||
/// Frames remaining in the current step (decrements per `on_pcm_frame`
|
||
/// for `SpeakLoud`/`SpeakQuiet`/`Pause`; freezes for `AwaitReply`).
|
||
step_frames_remaining: u32,
|
||
/// Frames received from `next_pcm_frame` while in `AwaitReply`.
|
||
/// When this reaches the step's `frames` target, advance.
|
||
reply_frames_received: u32,
|
||
/// Captured timestamps (the `LatencyProbe` consumes this via
|
||
/// `take_captures()` after the run). Discarded on every `on_pcm_frame`
|
||
/// call once the capture buffer is at capacity (bounded; hot-path).
|
||
captures: Vec<Capture>,
|
||
/// A pre-allocated frame returned from `next_pcm_frame` when we have a
|
||
/// pending reply (the harness intercepts frames routed back through the
|
||
/// existing media loop and returns them here). See §3.4 for how the
|
||
/// `SimCall` wires this pipe to a real `MediaThread`.
|
||
reply_ring: std::collections::VecDeque<PcmFrame>,
|
||
}
|
||
|
||
/// A timestamped event captured by the `SimAudioPipe`. Read by the
|
||
/// `LatencyProbe` post-run to compute p50/p99 latencies.
|
||
#[derive(Debug, Clone, Copy)]
|
||
pub enum Capture {
|
||
/// The caller started speaking loudly (a `SpeakLoud` step began).
|
||
CallerLoudOnset { at: Instant },
|
||
/// The FOB killed playout (a `next_pcm_frame` call returned None
|
||
/// immediately after a barge event). The wall-clock this slice cares
|
||
/// about for kill-time.
|
||
BargeKillObserved { at: Instant },
|
||
/// The caller heard a brain reply (a `next_pcm_frame` returned Some
|
||
/// after the barge cleared). The wall-clock this slice cares about
|
||
/// for mouth-to-ear.
|
||
CallerHeardReply { at: Instant },
|
||
}
|
||
```
|
||
|
||
#### 3.2.1 The two key methods
|
||
|
||
```rust
|
||
// crates/rutster-sim/src/sim_audio_pipe.rs (continued)
|
||
|
||
impl AudioSource for SimAudioPipe {
|
||
fn next_pcm_frame(&mut self) -> Option<PcmFrame> {
|
||
// The source path: the caller's "ear." Brain replies (frames the
|
||
// system produced on the egress side and routed back to us via the
|
||
// SimCall's mpsc wiring — see §3.4) land here.
|
||
match self.reply_ring.pop_front() {
|
||
Some(frame) => {
|
||
if self.is_in_await_reply_step() {
|
||
self.reply_frames_received += 1;
|
||
if self.reply_frames_received >= self.current_step_target() {
|
||
self.advance_step();
|
||
}
|
||
}
|
||
// Capture: this is the "caller heard" wall-clock.
|
||
self.captures.push(Capture::CallerHeardReply { at: Instant::now() });
|
||
Some(frame)
|
||
}
|
||
None => {
|
||
// The reflex muted us (slice-4's Reflex<P>::muted == true).
|
||
// Capture: this is the "barge kill observed" wall-clock.
|
||
// (Only capture if we are mid-AwaitReply post-barge; see the
|
||
// LatencyProbe's classification §3.3 for the dedup logic.)
|
||
self.captures.push(Capture::BargeKillObserved { at: Instant::now() });
|
||
None
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
impl AudioSink for SimAudioPipe {
|
||
fn on_pcm_frame(&mut self, _frame: PcmFrame) {
|
||
// The sink path: the caller "speaks." The scenario drives here.
|
||
// Decode the current step + emit the appropriate PCM signal.
|
||
self.dispatch_step_action();
|
||
// (We discard the inbound frame — the caller doesn't hear itself;
|
||
// the SimCall's wiring pushes the caller-side frame into the
|
||
// `tx_pcm_in` channel for the tap to forward to the brain.)
|
||
}
|
||
}
|
||
```
|
||
|
||
### 3.3 `LatencyProbe`
|
||
|
||
```rust
|
||
// crates/rutster-sim/src/latency.rs
|
||
|
||
/// Computes the two metrics slice-4's design sets budgets for, from a
|
||
/// captured-stream of `Capture` events produced by a `SimAudioPipe`.
|
||
///
|
||
/// # Verification discipline
|
||
///
|
||
/// The probe is the single source of truth for "did latency regress?"
|
||
/// Assertions are made against `LatencyProbe::p99_kill_ms()` and
|
||
/// `LatencyProbe::p99_mouth_to_ear_ms()`. A failure here is the build-red
|
||
/// signal ADR-0010 demands.
|
||
pub struct LatencyProbe {
|
||
captures: Vec<Capture>,
|
||
}
|
||
|
||
impl LatencyProbe {
|
||
pub fn from_captures(captures: Vec<Capture>) -> Self { Self { captures } }
|
||
|
||
/// Barge-in kill-time: caller-speech-onset → first `BargeKillObserved`.
|
||
/// Returns Duration per call (we capture one onset+kill pair per barge).
|
||
pub fn kill_times(&self) -> Vec<Duration> { /* pair captures */ }
|
||
|
||
/// Mouth-to-ear: caller-speech-onset → next `CallerHeardReply`.
|
||
pub fn mouth_to_ear_times(&self) -> Vec<Duration> { /* pair captures */ }
|
||
|
||
pub fn p50_kill_ms(&self) -> f64 { percentile(&self.kill_times(), 50) }
|
||
pub fn p99_kill_ms(&self) -> f64 { percentile(&self.kill_times(), 99) }
|
||
pub fn p50_mouth_to_ear_ms(&self) -> f64 { percentile(&self.mouth_to_ear_times(), 50) }
|
||
pub fn p99_mouth_to_ear_ms(&self) -> f64 { percentile(&self.mouth_to_ear_times(), 99) }
|
||
}
|
||
|
||
fn percentile(durations: &[Duration], pct: u8) -> f64 {
|
||
if durations.is_empty() { return f64::NAN; }
|
||
let mut sorted: Vec<u128> = durations.iter().map(|d| d.as_millis()).collect();
|
||
sorted.sort_unstable();
|
||
let idx = ((sorted.len() as f64 - 1.0) * (pct as f64 / 100.0)).round() as usize;
|
||
sorted[idx] as f64
|
||
}
|
||
```
|
||
|
||
The `LatencyProbe` is **post-hoc**: a single `SimCall` runs to completion, the `SimAudioPipe`'s
|
||
captures are drained, the probe computes the metrics. No per-tick instrumentation cost during the
|
||
hot path itself — just the `Instant::now()` calls inside `SimAudioPipe::next_pcm_frame` (which
|
||
isn't the hot path anyway; it's the simulated-caller epilogue).
|
||
|
||
### 3.4 `SimCall` (the wiring) + `ConcurrencyRunner`
|
||
|
||
```rust
|
||
// crates/rutster-sim/src/runner.rs
|
||
|
||
/// One simulated call: a `SimAudioPipe` + the wiring to drive it against the
|
||
/// existing `MediaThread`. Single binary; no separate process.
|
||
pub struct SimCall {
|
||
/// The scenario-driven caller pipe.
|
||
pipe: SimAudioPipe,
|
||
/// The `MediaThread` cmd_tx — we register a session, drive the pipe
|
||
/// via `tx_pcm_in`, capture frame receipts in `next_pcm_frame`.
|
||
media_cmd_tx: mpsc::Sender<MediaCmd>,
|
||
/// Latency probe populated post-run.
|
||
probe: Option<LatencyProbe>,
|
||
}
|
||
|
||
impl SimCall {
|
||
pub async fn run(mut self) -> LatencyProbe {
|
||
// 1. Register a session with the MediaThread.
|
||
// 2. Wire self.pipe as the session's AudioPipe (MediaCmd::RegisterSim).
|
||
// 3. Drive the scenario: each scenario step emits `on_pcm_frame`
|
||
// calls against the SimAudioPipe; the MediaThread's loop_driver
|
||
// echoes frames back via next_pcm_frame.
|
||
// 4. On End: drain captures + return the probe.
|
||
todo!("see §4 data flow")
|
||
}
|
||
}
|
||
|
||
/// The concurrency sweep runner. Spawns N `SimCall`s in parallel (tokio),
|
||
/// awaits all, aggregates per-call latencies into the sweep report.
|
||
pub struct ConcurrencyRunner {
|
||
/// Target binary in-process MediaThread cmd_tx. Passed in by the test fixture.
|
||
media_cmd_tx: mpsc::Sender<MediaCmd>,
|
||
/// Concurrency levels to sweep (slice-4½ hardcoded [1, 10, 50]).
|
||
concurrencies: Vec<usize>,
|
||
}
|
||
|
||
impl ConcurrencyRunner {
|
||
/// Run the full sweep; return the per-concurrency-level report.
|
||
pub async fn run(&self, scenario: Scenario) -> SweepReport { /* ... */ }
|
||
}
|
||
|
||
/// The artifact feeding the CI assertions. The thresholds.rs asserts
|
||
/// `report.barge_kill_p99_ms <= BARGE_IN_KILL_TIME_P99_MS` etc.
|
||
#[derive(Debug)]
|
||
pub struct SweepReport {
|
||
pub per_concurrency: Vec<PerConcurrencyReport>,
|
||
}
|
||
|
||
#[derive(Debug)]
|
||
pub struct PerConcurrencyReport {
|
||
pub concurrency: usize,
|
||
pub p50_kill_ms: f64,
|
||
pub p99_kill_ms: f64,
|
||
pub p50_mouth_to_ear_ms: f64,
|
||
pub p99_mouth_to_ear_ms: f64,
|
||
/// From slice-5/seams MediaCmd::Stats. The "doctrine-drift detector"
|
||
/// for the timing-thread debt — ADR-0010's debt-pairing readout.
|
||
pub max_tick_lag_micros: u64,
|
||
pub tick_overruns: u64,
|
||
pub total_ticks: u64,
|
||
pub tick_overrun_pct: f64,
|
||
}
|
||
```
|
||
|
||
### 3.5 `MediaCmd::RegisterSim` (the seam — one new enum variant)
|
||
|
||
Slice-5/seams already exists with `MediaCmd::Register`, `AcceptOffer`, `Delete`, `Shutdown`,
|
||
`Stats`, `Drain`. Slice 4½ adds ONE variant: `RegisterSim`, which lets a `SimCall` register a
|
||
session whose `AudioPipe` is a `SimAudioPipe` instead of a WebRTC-backed `RtcSession`. This is the
|
||
minimum extension to drive the harness without needing to spin up a real WebRTC peer.
|
||
|
||
```rust
|
||
// crates/rutster/src/media_thread.rs (extended this slice)
|
||
|
||
pub enum MediaCmd {
|
||
// ... existing variants (unchanged from slice-5)
|
||
/// slice-4½: harness-side session registration. The
|
||
/// `SimAudioPipe` lives entirely on the binary side; no WebRTC
|
||
/// handshake needed. The `tx_pcm_in` channel is the existing
|
||
/// sink-input seam (the harness emits `on_pcm_frame(frame)` directly
|
||
/// rather than the loop_driver pulling RTP + decoding first).
|
||
RegisterSim {
|
||
pipe: Box<dyn AudioPipe>,
|
||
reply: oneshot::Sender<ChannelId>,
|
||
},
|
||
}
|
||
```
|
||
|
||
The std thread's `run_media_thread` handles `RegisterSim` by inserting a synthetic "session" entry
|
||
that drives the harness's `SimAudioPipe` through the same `loop_driver::drive` calls as real
|
||
WebRTC sessions — the seam holds (`loop_driver.rs` + `rtc_session.rs` byte-identical). The
|
||
harness measures what real callers experience because **it drives the same code path**.
|
||
|
||
### 3.6 Tick-lag gauge integration (the doctrine-drift detector)
|
||
|
||
Slice-5/seams added `MediaStats { tick_overruns, last_tick_micros }` to `MediaCmd::Stats`.
|
||
Slice 4½ consumes both. The `ConcurrencyRunner` polls `MediaCmd::Stats` once per second during
|
||
the sweep; the per-concurrency-level report carries:
|
||
|
||
- `max_tick_lag_micros` — the maximum observed `last_tick_micros` during the sweep at this
|
||
concurrency level. Indicates "the worst tick the loop experienced."
|
||
- `tick_overruns` / `total_ticks` — cumulative count of ticks where `last_tick_micros > 10_000`
|
||
(10 ms); converts to percent. Indicates "what fraction of ticks overflowed."
|
||
|
||
The thresholds (`TICK_LAG_MAX_MS = 10.0`, `TICK_OVERRUN_PCT_MAX = 1.0`) are the answer to ADR-0010's
|
||
"if the concurrency sweep shows the shared-tokio poll loop breaching budget at realistic call
|
||
counts, the dedicated-timing-thread work gets scheduled on data, not vibes." If slice 4½'s sweep
|
||
shows `tick_overrun_pct > 1.0` at 50 calls, **the FOB reflex loop's single-thread debt is real and
|
||
graduates from doctrine to data** — that's the slice's load-bearing finding regardless of whether
|
||
the latency thresholds pass.
|
||
|
||
---
|
||
|
||
## 4. Data flow
|
||
|
||
### 4.1 Single `SimCall` (the unit of measurement)
|
||
|
||
```
|
||
1. SimCall::run() starts.
|
||
2. Send MediaCmd::RegisterSim { pipe: Box<SimAudioPipe>, reply } to media_cmd_tx.
|
||
3. The MediaThread handles RegisterSim:
|
||
- Constructs a "synthetic session" entry in its HashMap.
|
||
- The session's `pipe` field is the SimAudioPipe.
|
||
- Subsequent loop_driver::drive(now) calls touch this session identically
|
||
to a WebRTC session (the seam holds).
|
||
4. The harness drives the scenario:
|
||
- For each SpeakLoud/SpeakQuiet step: the SimCall emits `pipe.on_pcm_frame(frame)` calls
|
||
at the 20 ms tick cadence. loop_driver::drive's encode path is bypassed for the SimPipe
|
||
(we wrote the frame directly into the pipe).
|
||
- loop_driver::drive immediately treats the next_pcm_frame call as the source path: it
|
||
pulls from the SimPipe's reply_ring (where brain replies populate when the SimCall
|
||
sees them via the tap/engine path).
|
||
5. Brain replies (from MockRealtimeBrain or the in-process tap):
|
||
- Routed back into the SimPipe's reply_ring via an mpsc the SimCall holds.
|
||
- loop_driver::drive picks them up on next_pcm_frame, encodes if real-WebRTC, but for sim
|
||
we just observe — captures CallerHeardReply timestamp.
|
||
6. Reflex barge-in (slice-4 already merged):
|
||
- If the SimPipe emitted SpeakLoud frames, the LocalVadReflex<P> (which wraps the pipe
|
||
in the session_map/MediaThread composition site, slice-4 Task 6) trips the SpeechStarted
|
||
advisory → Reflex<P>::muted = true → next_pcm_frame returns None → capture
|
||
BargeKillObserved timestamp.
|
||
7. On End step: harness stops driving, returns LatencyProbe.
|
||
```
|
||
|
||
### 4.2 Concurrency sweep (the doctrine detector)
|
||
|
||
```
|
||
1. ConcurrencyRunner::run(scenario) launches N Tokio tasks, each running SimCall::run
|
||
against the SAME media_cmd_tx (shared media thread).
|
||
2. The MediaThread drives ALL N sessions per 10 ms meta-tick (slice-4 §2.2 unchanged shape).
|
||
3. Per second during the sweep: ConcurrencyRunner fires MediaCmd::Stats; accumulates
|
||
tick_overruns + last_tick_micros samples.
|
||
4. On all SimCalls completing: aggregate per-call LatencyProbes → p50/p99 vector.
|
||
5. Build SweepReport with per-concurrency rows, asserting thresholds.rs constants.
|
||
```
|
||
|
||
### 4.3 Why in-process (not client-server)
|
||
|
||
The harness DOES NOT stand up the binary as a server + a separate sim-client process. Why:
|
||
|
||
1. **Determinism.** Loopback within the same process eliminates websocket / socket / TCP jitter as
|
||
a confounder. The thresholds are assertions about the FOB reflex loop itself — not assertions
|
||
about NTPD variance on the CI runner.
|
||
2. **CI simplicity.** `cargo test --features=sim-bench` runs in-process; no port binding, no
|
||
test-orchestration container, no race against `epoll` initialization.
|
||
3. **Direct seam access.** The harness can construct a `SimAudioPipe` and ship it via
|
||
`MediaCmd::RegisterSim` directly — same path the production binary would use if it had a
|
||
"synthetic caller" feature, no client-server glue needed.
|
||
|
||
A separate client-server mode (true loopback against the binary's HTTP/WebSocket surface) IS
|
||
deferred — it's needed when the harness gains LLM-driven callers (post-spearhead refinement) and
|
||
needs network realism.
|
||
|
||
---
|
||
|
||
## 5. Measurement plan + thresholds
|
||
|
||
### 5.1 The budgets (concrete numbers)
|
||
|
||
| Metric | Budget (slice-4 design) | CI assertion (slice 4½) | Rationale for slack |
|
||
|---|---|---|---|
|
||
| Barge-in kill-time, p99 | ≤60 ms (3 debounce × 20 ms + 1 drain tick) | ≤80 ms | CI runner has known variance against dev; the budget is 60 ms; the assertion is 80 to avoid flakiness. |
|
||
| Mouth-to-ear round-trip, p99 | ≤200 ms slice-1 + ≤300 ms mock brain + ≤100 ms playout = ~600 ms | ≤700 ms | Same logic; the mock brain is deterministic but the harness adds observer cost. |
|
||
| Tick-lag (max prev-poll duration) | unspecified | ≤10 ms | The slice-5/seams META_TICK const; the invariant the assertion makes explicit. |
|
||
| Tick overruns (fraction of ticks > 10 ms) | unspecified | ≤1% | At 50 calls × 1000 ticks each = ≥99% need to be ≤10 ms. Allows for one scheduling hiccup per ~99 well-behaved ticks. |
|
||
|
||
### 5.2 Per-concurrency swept assertions
|
||
|
||
```
|
||
For each N ∈ [1, 10, 50]:
|
||
run scenario loud-barge.toml against N concurrent SimCalls.
|
||
assert p99_kill_ms <= 80 ms;
|
||
assert p99_mouth_to_ear_ms <= 700 ms;
|
||
assert max_tick_lag_micros <= 10_000; // 10 ms
|
||
assert tick_overrun_pct <= 1.0;
|
||
```
|
||
|
||
### 5.3 The scenarios (3 shipped)
|
||
|
||
| Scenario | Path | What it asserts |
|
||
|---|---|---|
|
||
| `loud-barge.toml` | Caller speaks 20 loud frames → awaits reply → end. | The PRIMARY barge-in path (slice-4 §5.1): local VAD fires, kill within ≤80 ms at p99, NO brain advisory required. |
|
||
| `quiet-advisory.toml` | Caller speaks 20 quiet frames (sub-VAD-threshold) → awaits reply → end. | The SECONDARY barge-in path (slice-4 §5.2): brain advisory fires from `MockRealtimeBrain`, kill flows through slice-3 plumbing + slice-4 `advisory_tx` → `Reflex`. |
|
||
| `sustained-call.toml` | Caller speaks 10 loud → 10 quiet → 10 loud → 10 quiet → 10 loud → end (5 minutes of talk). The fatigue / sustained-load check. | Multi-barge: 3 `SpeechStarted` advisories should fire in sequence; the `Reflex::barge_epoch` increments 3×; latency is asserted across all three bars (the second + third bar shouldn't drift > 1.5× the first). |
|
||
|
||
### 5.4 CI integration
|
||
|
||
The CI workflow gains a new job:
|
||
|
||
```yaml
|
||
# .github/workflows/ci.yml (additive)
|
||
|
||
sim-bench:
|
||
name: sim-bench (stable)
|
||
runs-on: ubuntu-latest
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
- uses: dtolnay/rust-toolchain@stable
|
||
- uses: Swatinem/rust-cache@v2
|
||
- name: Install libopus
|
||
run: sudo apt-get update && sudo apt-get install -y libopus-dev
|
||
- name: Run sim-bench threshold sweep
|
||
run: cargo test --all --features=sim-bench -- --test-threads=1
|
||
```
|
||
|
||
Notes:
|
||
- `--test-threads=1` — concurrent sim-bench tests would pollute each other's `MediaStats` polling
|
||
(the tick-lag gauge measures the SHARED media thread; concurrent runs of the sweep would
|
||
contaminate each other).
|
||
- Run on `stable` only — the matrix already runs `cargo test --all` on stable + 1.85; the bench
|
||
feature lives on stable.
|
||
|
||
### 5.5 The thresholds-as-test contract
|
||
|
||
```rust
|
||
// crates/rutster-sim/src/thresholds.rs (continuation — the CI test entry)
|
||
|
||
#[cfg(test)]
|
||
mod threshold_assertions {
|
||
use super::*;
|
||
|
||
#[tokio::test]
|
||
#[cfg(feature = "sim-bench")]
|
||
async fn loud_barge_at_each_concurrency_passes_thresholds() {
|
||
let scenario = Scenario::load("scenarios/loud-barge.toml").unwrap();
|
||
for &n in SWEEP_CONCURRENCIES {
|
||
let report = ConcurrencyRunner::in_process(n).run(scenario.clone()).await;
|
||
let row = report.per_concurrency.iter()
|
||
.find(|r| r.concurrency == n).expect("concurrency row");
|
||
|
||
assert!(
|
||
row.p99_kill_ms <= BARGE_IN_KILL_TIME_P99_MS,
|
||
"p99 kill-time at N={}: {}ms > {}ms (budget overflow)",
|
||
n, row.p99_kill_ms, BARGE_IN_KILL_TIME_P99_MS,
|
||
);
|
||
assert!(
|
||
row.p99_mouth_to_ear_ms <= MOUTH_TO_EAR_P99_MS,
|
||
"p99 mouth-to-ear at N={}: {}ms > {}ms",
|
||
n, row.p99_mouth_to_ear_ms, MOUTH_TO_EAR_P99_MS,
|
||
);
|
||
assert!(
|
||
(row.max_tick_lag_micros as f64) / 1000.0 <= TICK_LAG_MAX_MS,
|
||
"max tick-lag at N={}: {}us > {}ms",
|
||
n, row.max_tick_lag_micros, TICK_LAG_MAX_MS,
|
||
);
|
||
assert!(
|
||
row.tick_overrun_pct <= TICK_OVERRUN_PCT_MAX,
|
||
"tick overrun % at N={}: {}% > {}%",
|
||
n, row.tick_overrun_pct, TICK_OVERRUN_PCT_MAX,
|
||
);
|
||
}
|
||
}
|
||
|
||
#[tokio::test]
|
||
#[cfg(feature = "sim-bench")]
|
||
async fn quiet_advisory_at_1_concurrency_passes_thresholds() { /* ... */ }
|
||
|
||
#[tokio::test]
|
||
#[cfg(feature = "sim-bench")]
|
||
async fn sustained_call_multibarge_does_not_drift() { /* ... */ }
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Why these decisions
|
||
|
||
### 6.1 Why a new crate (`crates/rutster-sim/`) instead of in-tree tests
|
||
|
||
A test under `crates/rutster/tests/sim_*.rs` would absorb the harness into the binary crate —
|
||
encouraging the harness to depend on binary internals. The separate crate keeps the dependency
|
||
direction: `rutster-sim` → `rutster-media` + `rutster-call-model` + `rutster` (for `MediaCmd`).
|
||
The sim crate is a **FOB member** per ADR-0008: hot-path-adjacent (drives the loop), differentiating
|
||
(the proof artifact). It earns cratehood the same way `rutster-tap` did.
|
||
|
||
### 6.2 Why scripted scenarios, not LLM-driven callers (deferred)
|
||
|
||
- **Reproducibility** is the entire point. A CI gate that asserts "p99 ≤ 80 ms" can't be flaky on
|
||
LLM variance — the LLM might take 2 s to respond, the threshold fails, the build is red, the dev
|
||
attributes it to the LLM provider. The signal is corrupted.
|
||
- **Isolation** of what's being measured: scripted scenarios measure the FOB + the loop + the
|
||
reflex; LLM callers introduce brain-side variance that isn't a FOB property.
|
||
- LLM-driven callers land in a post-spearhead refinement tier (ADR-0010 explicit deferral). They
|
||
measure a different question ("how does it feel to talk to my brain?") than the MVP harness asks
|
||
("does the FOB reflex loop meet its budget under load?").
|
||
|
||
### 6.3 Why in-process, not client-server (the false-precision risk)
|
||
|
||
A client-server test harness gives Network-Realism™ but takes away the **measurement discipline**: a
|
||
TCP retransmit on the CI runner would inflate p99 unattributably, and a "drift over threshold"
|
||
failure becomes a triage time-sink ("is the FOB regressing, or did the CI runner's network blip?").
|
||
The in-process mode gives **single-process determinism** at the cost of network realism — which the
|
||
spearhead explicitly doesn't need (it's measuring the FOB loop, not the deployed system). A
|
||
future post-spearhead tier adds a client-server mode for integration realism + a different set of
|
||
thresholds.
|
||
|
||
### 6.4 Why tick-lag as a primary readout (the doctrine-drift contract)
|
||
|
||
The single-thread media loop (slice-4 §6.3) is a spearhead-scale decision explicitly deferred to a
|
||
"load demands it" trigger. Without the harness measuring it under load, the deferral has no
|
||
data-driven graduation criterion — it'd be doctrine ("faster per-session threads later") instead of
|
||
evidence ("measured `tick_overrun_pct = 12%` at 50 calls, the threadpool shard lands now"). ADR-0010
|
||
explicitly pairs the debt — `MediaStats.tick_overruns` exists because of slice-5/seams' seam
|
||
work; slice 4½ makes the gauge OBSERVED. Both pass and the single-thread design is validated; both
|
||
breach and the graduation is scheduled with data.
|
||
|
||
### 6.5 Why `--features=sim-bench` (default off)
|
||
|
||
`cargo test --all` runs in 12 s on the CI runner today (fmt+clippy+test sweep, no bench).
|
||
Turning the threshold sweep on by default would either (a) slow down every PR's turnaround to the
|
||
sim sweep duration (estimated 30–60 s, dominated by the 50-concurrency sweep), or (b) make the
|
||
threshold sweep's failure mode just-another-failing-test that gets ignored. The opt-in feature
|
||
makes the threshold sweep a **separate concern with its own CI job**, surfacing its results
|
||
prominently in the PR status checks, and small enough to not block the routine `cargo test --all`
|
||
gate. This mirrors how performance-sensitive open-source projects gate `cargo bench` PRs.
|
||
|
||
### 6.6 Why p99 (not p50) as the load-bearing assertion
|
||
|
||
p50 = "the typical experience." p99 = "the worst acceptable case." The contact-center wedge
|
||
(README §"Why it exists") rests on tight-tail-latency: a p50 of 60 ms with a p99 of 1.5 s is
|
||
INDISTINGUISHABLE from a cloud CCaaS provider that averaged down to 60 ms but had bad tails. The
|
||
assertion HAS to fire on the tail to be meaningful.
|
||
|
||
p999 (or max) IS deferred — too noisy for CI gate assertions (one scheduler hiccup blowing past the
|
||
ceiling would block every PR for unrelated reasons). p99 is the empirical sweet spot: sensitive
|
||
enough to catch real regressions, lenient enough to survive CI runner variance.
|
||
|
||
---
|
||
|
||
## 7. Done-criteria
|
||
|
||
1. `cargo test --all` passes (stable + 1.85) — the routine gate, UNCHANGED. The sim-bench feature
|
||
is opt-in; default `cargo test --all` does NOT run the threshold sweep.
|
||
2. `cargo fmt --check` + `cargo clippy -- -D warnings` clean on the new crate.
|
||
3. `cargo test --all --features=sim-bench` passes — the new gate, on stable. CI runs this in a
|
||
separate `sim-bench` job per PR + nightly.
|
||
4. `cargo deny check` passes — no new dep conflicts (`toml` is already a workspace member;
|
||
`serde` already workspace member).
|
||
5. `cargo doc --no-deps` renders the new `crates/rutster-sim/` cleanly with learner-facing
|
||
comments per AGENTS.md code style.
|
||
6. Loop driver + rtc_session seam STILL holds: `loop_driver.rs` + `rtc_session.rs` byte-identical to
|
||
slice-3 (CI pinned-blob gate from slice-4 Task 10 unchanged). The new `MediaCmd::RegisterSim`
|
||
variant lives in `media_thread.rs`, NOT in the seam files.
|
||
7. The lance `loud-barge.toml` scenario passes the threshold sweep at all of [1, 10, 50]
|
||
concurrency.
|
||
8. The `quiet-advisory.toml` scenario passes at 1 concurrency (the secondary-path focus).
|
||
9. The `sustained-call.toml` scenario's 3-barge sequence shows performance drift ≤ 1.5× across
|
||
barges (anti-fatigue assertion).
|
||
10. Tick-lag gauge reads a `MediaStats.{tick_overruns, last_tick_micros}` value during the sweep
|
||
and surfaces it in the SweepReport.
|
||
11. SweepReport's per-concurrency rows are logged to stderr in a structured format (CI failure
|
||
messages are readable; "p99 kill-time at N=50: 84ms > 80ms" not "test_sim_thresholds
|
||
failed").
|
||
12. The single-thread-vs-threadpool question has a data-point answer documented ("slice 4½ found p99
|
||
tick-lag = Xms at 50 calls; the threadpool shard remains deferred / the threadpool shard
|
||
should land now"). Even if the answer is "data confirms the deferral is fine for now," the
|
||
decision is no longer vibe-based.
|
||
|
||
---
|
||
|
||
## 8. Open decisions
|
||
|
||
### 8.1 Should the harness also assert against the HTTP/WebSocket out-of-process surface?
|
||
|
||
**Decision (slice 4½):** no. In-process measurement only. The out-of-process mode is a future
|
||
post-spearhead refinement (paired with LLM-driven callers). Reasoning in §6.3.
|
||
|
||
### 8.2 Should the sim-bench CI job also run on the 1.85 toolchain?
|
||
|
||
**Decision (slice 4½):** no. The matrix already runs `cargo test --all` on both. The sim-bench
|
||
job is a stable-only opt-in feature; 1.85 doesn't get a second tier of benches. Rationale:
|
||
the sim-bench feature is `cfg(feature)` — feature-gated code paths need their own gates, not a
|
||
toolchain proliferation.
|
||
|
||
### 8.3 Threshold values: hardcodedconsts vs env-overridable for the CI runner operator?
|
||
|
||
**Decision (slice 4½):** hardcoded consts in `thresholds.rs`. Post-spearhead, with the
|
||
per-environment tuning framework (paired with slice-4 §1.2 VAD-threshold tuning deferral), they
|
||
become env-driven. Hardcoded now makes the budget-vs-assertion-slack reasoning (§5.1) explicit in
|
||
source — not subject to runner-env drift.
|
||
|
||
### 8.4 Should `MediaCmd::RegisterSim` carry a `pipe: Box<dyn AudioPipe>` or a more structured
|
||
"sim descriptor" the media thread materializes?
|
||
|
||
**Decision (slice 4½):** `Box<dyn AudioPipe>` — same shape as the existing `RtcSession`'s pipe
|
||
construction (which has `pipe: Box<dyn AudioPipe>`). A "sim descriptor" would add a layer of
|
||
indirection the harness doesn't benefit from; the harness already constructs the `SimAudioPipe`
|
||
and is fully prepared to ship it across the channel. The single variant is the minimal seam.
|
||
|
||
### 8.5 Concurrency-sweep sample size / iteration count per concurrency level
|
||
|
||
**Decision (slice 4½):** 1000 ticks per concurrency level (the meta-tick count for the sweep
|
||
durations). At 10 ms per tick = 10 seconds of sweep per concurrency × 3 levels = 30s sweep total.
|
||
Sufficient sample size for a stable p99 (n = ~1000 ticks); bounded enough to keep CI fast.
|
||
|
||
### 8.6 Should the harness record per-call audio (WAV capture of the `SimAudioPipe`'s frames) for
|
||
supervisor review?
|
||
|
||
**Decision (slice 4½):** no. That's a rung-2 escalation feature (warm-handoff artifact). The
|
||
harness measures; recording is a separate concern. The `SimAudioPipe` exposes a `take_captures()`
|
||
API for the `LatencyProbe` only; raw frame capture is out of scope.
|
||
|
||
---
|
||
|
||
## 9. Cross-references
|
||
|
||
- [ADR-0010](../../adr/0010-spearhead-benchmark-sim-harness.md) — centralized rationale: why 4½
|
||
exists, what it should produce, what it should defer (LLM callers), how it pairs the
|
||
timing-thread debt.
|
||
- [slice-4 spec §5.1](2026-07-01-slice-4-barge-in-design.md) — the ≤60 ms kill budget + the
|
||
latency-arithmetic this slice asserts against.
|
||
- [slice-1 spec §8.5 #6](2026-06-28-slice-1-webrtc-loopback-design.md) — the seam gate (`loop_driver.rs`
|
||
+ `rtc_session.rs` byte-identical) slice 4½ re-affirms (NO changes to those files).
|
||
- [slice-5/seams plan](../plans/2026-07-04-slice-5-scalability-seams.md) (the infrastructure
|
||
pre-paving this slice consumes) — `MediaCmd::Stats` exposes `MediaStats { tick_overruns,
|
||
last_tick_micros }`, the readout this slice's concurrency sweep surfaces.
|
||
- [ADR-0002](../../adr/0002-north-star-and-fused-core.md) — the fused vertical; the in-process
|
||
measurement IS the fused-vertical seam (no gRPC hop between the harness + the loop).
|
||
- [ADR-0008](../../adr/0008-fob-and-green-zone.md) — FOB/green-zone doctrine; the harness is a FOB
|
||
member (hot-path-adjacent + differentiating). No green-zone dep added.
|
||
- [PORT_PLAN.md §Phasing](../../PORT_PLAN.md) — step 4½ = sim harness (per ADR-0010 insertion).
|