rutster/docs/adr/0010-spearhead-benchmark-sim-harness.md

ADR-0010 — Spearhead re-sequencing: benchmark + simulation harness at step 4½; escalation pulled forward

• Status: Accepted (maintainer-ratified 2026-07-03)
• Date: 2026-07-03
• Origin: 2026-07-03 adversarial review (recommendations R1, R2) + market feature scan (F1 — simulation testing as the standout engine-fit feature).
• Amends: README.md (spearhead list), docs/PORT_PLAN.md (phasing section), the vision-revision spearhead sequence (design record; unmodified — this ADR is the amendment)

Context

Three findings converged on the spearhead's back half:

1. The wedge is unmeasured (review P3/R2). The no-GC determinism pillar — the only claim justifying Rust over a weekend of Pipecat — has no proof artifact. README latency figures are arithmetic estimates; TapMetrics counts drops, not latency. Slice 4's design sets a ≤60 ms barge-in kill budget but ships only a synthetic e2e test, not a measurement harness.
2. The step-5 demo was commoditized (review D2). OpenAI Realtime SIP is GA: a trunk pointed at sip.api.openai.com reproduces "I called my box and an AI answered" with zero infrastructure. The momentum value of rushing to step 5 has collapsed; the differentiating value of measured reflex quality and of escalation (rung 2 — which no framework or brain vendor ships) has risen.
3. Simulation is the standout engine-fit feature (scan F1). Parloa sells pre-deployment simulation + evals as an enterprise phase; Cekura/Coval sell it standalone. Nobody ships it self-hosted. rutster's tap symmetry (a synthetic caller is just another media-leg ingress) and the slice-3 mock brain mean the substrate is half-built.

One build serves all three: the benchmark is a simulation harness pointed at latency metrics.

Decision

Insert step 4½ into the spearhead — the benchmark + simulation harness — after barge-in (4), before rented transport (5). Numbered 4½ deliberately: existing "step 5"/"step 6" references (stub-crate docs, PORT_PLAN rows, fuzz plan) stay valid.

Step 4½ scope (the rutster-sim seed):

• Synthetic-caller ingress — scripted scenario playback (WAV/PCM segments with timing directives: speak, pause, interrupt-at-T) driving the same media-leg ingress path a real caller uses. LLM-driven callers are a later extension, not this step.
• Latency measurement — p50/p99 mouth-to-ear round-trip and barge-in kill-time (caller-speech-onset → playout-ring-flushed), measured against slice-4's ≤60 ms budget.
• Concurrency sweep — the same measurements at 1 / 10 / 50 concurrent synthetic calls, which also surfaces the single-poll-task head-of-line-blocking debt (review P2) with data instead of doctrine.
• CI-regressed — thresholds asserted per commit; a latency regression fails the build the same way a broken test does.

Sequencing after 4½: rung-2 escalation (human takeover) is pulled forward ahead of steps 5–6. It is the white space no competitor ships (warm-handoff artifact + whisper — scan F2), it works entirely on WebRTC ingress (no PSTN dependency), and it is the capability that answers "why not point the trunk at OpenAI directly?" before step 5 makes that question live. Steps 5 (rented transport) and 6 (spend cap) follow, unchanged in content.

Consequences

• Positive: the wedge gains its proof artifact (and the only differentiating demo available this year: reproducible p99 kill-latency vs. the GC'd frameworks); the doctrine-drift detector for the timing-thread debt exists before scale makes it urgent; the eval-infrastructure foundation (scenario packs, rung-3/4 regression evals) is laid as a side effect; the spearhead's next public moment is a capability no one else has, not a commoditized one.
• Negative: steps 5–6 slip by two slices; "a real phone number" — still the most visceral demo — waits. Accepted deliberately: the demo it enables is no longer scarce (D2), and the reflex claims stay unsubstantiated until measured.
• Debt paired: if the concurrency sweep shows the shared-tokio poll loop breaching budget at realistic call counts, the dedicated-timing-thread work (ARCHITECTURE.md doctrine, review P2) gets scheduled on data, not vibes.

References

• Adversarial review 2026-07-03 — findings D1, D2, P2, P3; recommendations R1, R2
• Market feature scan 2026-07-03 — F1 (simulation harness), F2 (handoff protocol), A8 (scenario packs)
• ADR-0007 — the trunk posture step 5 implements
• Slice-4 design (docs/superpowers/specs/2026-07-01-slice-4-barge-in-design.md) — the ≤60 ms kill budget this harness measures