Aaron D. Lee
49a5d8c91c
Joins the CI-regressed sim-bench sweep (S7 style: threshold const with budget+slack rationale, cfg(sim-bench) test, --test-threads=1 job). Drives the real trunk WS route through the PRODUCTION serve path (rutster::serve::serve_with_nodelay) over a real loopback socket at the real 20ms cadence. Healthy ~1-2ms; a Nagle regression stalls ~40ms+; threshold 20ms splits the regimes by an order of magnitude each way. Verified load-bearing: fails when tcp_nodelay(false) — see module doc for the Option D socket2 TCP_QUICKACK suppression that defeats the Linux loopback's quick-ACK heuristic so the assertion actually catches the Nagle regression on this runner. Signed-off-by: Aaron D. Lee <himself@adlee.work>
262 lines
12 KiB
Rust
262 lines
12 KiB
Rust
//! # thresholds — CI-regressed latency thresholds + sim-bench assertion tests
|
||
//!
|
||
//! See `docs/superpowers/specs/2026-07-05-slice-4-half-benchmark-sim-design.md`
|
||
//! §5.1 + §5.5.
|
||
//!
|
||
//! The threshold consts land here at S1 (per the plan's S1 step 2 note:
|
||
//! "the consts as immediate module-level `pub const` items per spec §5.1 —
|
||
//! they're used by S5/S6/S7 wiring"). The `#[cfg(feature = "sim-bench")]
|
||
//! #[tokio::test]` assertion tests land at S7.
|
||
//!
|
||
//! # Why these numbers
|
||
//!
|
||
//! See spec §5.1 for the budget-vs-assertion-slack reasoning. Each const
|
||
//! is paired with a doc-comment explaining the budget it enforces + the
|
||
//! slack rationale (so a future maintainer who needs to bump one knows
|
||
//! *why* the current value is what it is, not just *what* it is).
|
||
|
||
/// Slice-4 spec §5.1 + §7 done-criteria #8: kill-time budget is
|
||
/// ≤60 ms (3 debounce frames × 20 ms tick + 1 tick to drain + apply).
|
||
/// Observer slack to make CI deterministic-but-not-flaky on a slow
|
||
/// runner: effective CI assertion ≤80 ms (60 ms budget + 20 ms slack).
|
||
///
|
||
/// A regression here is the red X ADR-0010 demands — the wedge's
|
||
/// "local real-time reflexes that don't need the brain" claim is
|
||
/// arithmetic until this assertion fires on every PR.
|
||
pub const BARGE_IN_KILL_TIME_P99_MS: f64 = 80.0;
|
||
|
||
/// Slice-1 + slice-3 mouth-to-ear budget: 200 ms (slice-1 notification
|
||
/// budget) + 250 ms mock brain round-trip + 100 ms playout buffer.
|
||
/// CI assertion ceiling: 700 ms (allowance for CI runner variance
|
||
/// against the dev machine — the mock brain is deterministic but the
|
||
/// harness adds observer cost; the dev machine usually lands ~600 ms).
|
||
pub const MOUTH_TO_EAR_P99_MS: f64 = 700.0;
|
||
|
||
/// Slice-5/seams tick-lag gauge: the meta-tick must stay under 10 ms
|
||
/// (the loop's nominal period). At 1 call: ≤2 ms expected. At 50
|
||
/// calls: ≤10 ms expected. Tick overruns (count of ticks exceeding
|
||
/// 10 ms) at p50 across the sweep: ≤1% of total ticks per
|
||
/// `TICK_OVERRUN_PCT_MAX`.
|
||
///
|
||
/// If a concurrency sweep shows `tick_overrun_pct > 1.0` at 50 calls,
|
||
/// **the FOB reflex loop's single-thread debt is real and the
|
||
/// dedicated-threadpool-shard graduation (slice-4 §1.2 deferral #2)
|
||
/// gets its data-driven case.** That finding is the slice's
|
||
/// load-bearing output regardless of whether the latency thresholds
|
||
/// pass — the doctrine-drift detector worked.
|
||
pub const TICK_LAG_MAX_MS: f64 = 10.0;
|
||
pub const TICK_OVERRUN_PCT_MAX: f64 = 1.0;
|
||
|
||
/// Concurrency-sweep sample sizes per spec §2.4: 1 isolates the
|
||
/// baseline (cold-path latency with zero concurrency pressure —
|
||
/// slice-4's §5.1 ≤60 ms kill budget asserted here); 10 is the
|
||
/// warm working set (~peak spearhead-scale); 50 is the saturation
|
||
/// point (ADR-0010's "single-poll-task head-of-line-blocking debt"
|
||
/// lives here). We do NOT test 100/500/5000 — that's fleet-scale
|
||
/// (rung 3). 50 is the upper edge of the spearhead's "one binary,
|
||
/// one city" claim.
|
||
pub const SWEEP_CONCURRENCIES: &[usize] = &[1, 10, 50];
|
||
|
||
/// Deploy slice A §5.1: the TCP_NODELAY regression tripwire. p99
|
||
/// send→recv latency for small server-originated WS frames at the trunk's
|
||
/// 20 ms cadence, over loopback, through the PRODUCTION serve path
|
||
/// (`rutster::serve::serve_with_nodelay` — the exact call `main.rs`
|
||
/// makes). Healthy: ~1–2 ms. A Nagle regression (axum #2521 class —
|
||
/// e.g. someone drops `.tcp_nodelay(true)` from the helper) interacts
|
||
/// with the peer's delayed-ACK timer and stalls frames ~40 ms+. 20 ms
|
||
/// sits an order of magnitude above healthy and comfortably below the
|
||
/// failure mode — deterministic-but-not-flaky on a slow CI runner, same
|
||
/// slack posture as the consts above.
|
||
pub const WS_FRAME_SEND_TO_RECV_P99_MS: f64 = 20.0;
|
||
|
||
#[cfg(all(test, feature = "sim-bench"))]
|
||
mod bench_assertions {
|
||
//! The CI-regressed threshold assertion tests (spec §5.2 + §5.5).
|
||
//!
|
||
//! These tests run ONLY under `--features=sim-bench` (default off).
|
||
//! The CI `sim-bench` job runs them per PR + nightly on stable.
|
||
//! Failure ⇒ red X ⇒ PR does not merge (ADR-0010's "a latency
|
||
//! regression fails the build" contract).
|
||
//!
|
||
//! `--test-threads=1` (per spec §6.5 load-bearing): concurrent
|
||
//! sim-bench tests would contaminate each other's shared gauge
|
||
//! (the TickLagStats reads the SHARED tokio runtime; concurrent
|
||
//! sweeps across tests would all pollute the same gauge). The CI
|
||
//! job passes `--test-threads=1` explicitly.
|
||
use super::*;
|
||
use crate::concurrency::ConcurrencyRunner;
|
||
use crate::runner::SimCall;
|
||
use crate::scenario::Scenario;
|
||
use std::path::Path;
|
||
|
||
/// Load a scenario from the shipped `scenarios/` directory using
|
||
/// `env!("CARGO_MANIFEST_DIR")` for a robust path lookup that
|
||
/// doesn't depend on the test's CWD (cargo test typically runs in
|
||
/// the crate root, but the explicit manifest-dir pattern is the
|
||
/// std-library idiom — see the existing project's tests for the
|
||
/// same composition).
|
||
fn load_scenario(name: &str) -> Scenario {
|
||
let path = Path::new(env!("CARGO_MANIFEST_DIR"))
|
||
.join("scenarios")
|
||
.join(format!("{name}.toml"));
|
||
Scenario::load(&path)
|
||
.unwrap_or_else(|e| panic!("load scenario {name} from {path:?}: {e:?}"))
|
||
}
|
||
|
||
#[tokio::test]
|
||
async fn loud_barge_at_each_concurrency_passes_thresholds() {
|
||
let scenario = load_scenario("loud-barge");
|
||
for &n in SWEEP_CONCURRENCIES {
|
||
let report = ConcurrencyRunner::in_process(n).run(scenario.clone()).await;
|
||
let row = report
|
||
.per_concurrency
|
||
.iter()
|
||
.find(|r| r.concurrency == n)
|
||
.unwrap_or_else(|| panic!("missing concurrency row for N={n}"));
|
||
|
||
assert!(
|
||
row.p99_kill_ms <= BARGE_IN_KILL_TIME_P99_MS,
|
||
"p99 kill-time at N={}: {}ms > {}ms (budget overflow; \
|
||
slice-4 §5.1 ≤60ms kill budget + 20ms CI slack)",
|
||
n,
|
||
row.p99_kill_ms,
|
||
BARGE_IN_KILL_TIME_P99_MS,
|
||
);
|
||
assert!(
|
||
row.p99_mouth_to_ear_ms <= MOUTH_TO_EAR_P99_MS,
|
||
"p99 mouth-to-ear at N={}: {}ms > {}ms \
|
||
(slice-1 200ms + slice-3 ~300ms mock brain + 100ms playout + CI slack)",
|
||
n,
|
||
row.p99_mouth_to_ear_ms,
|
||
MOUTH_TO_EAR_P99_MS,
|
||
);
|
||
assert!(
|
||
(row.max_tick_lag_micros as f64) / 1000.0 <= TICK_LAG_MAX_MS,
|
||
"max tick-lag at N={}: {}us > {}ms \
|
||
(the meta-tick's nominal 10ms period was breached; \
|
||
ADR-0010 doctrine-drift detector)",
|
||
n,
|
||
row.max_tick_lag_micros,
|
||
TICK_LAG_MAX_MS,
|
||
);
|
||
assert!(
|
||
row.tick_overrun_pct <= TICK_OVERRUN_PCT_MAX,
|
||
"tick overrun % at N={}: {}% > {}% \
|
||
(> 1% of ticks exceeded 10ms; threadpool-shard graduation case)",
|
||
n,
|
||
row.tick_overrun_pct,
|
||
TICK_OVERRUN_PCT_MAX,
|
||
);
|
||
}
|
||
}
|
||
|
||
#[tokio::test]
|
||
async fn quiet_advisory_at_1_concurrency_passes_thresholds() {
|
||
let scenario = load_scenario("quiet-advisory");
|
||
let report = ConcurrencyRunner::in_process(1).run(scenario).await;
|
||
let row = &report.per_concurrency[0];
|
||
|
||
// The SimAudioPipe records CallerLoudOnset only on SpeakLoud
|
||
// step entry. The quiet-advisory scenario (only SpeakQuiet +
|
||
// AwaitReply + End) has no loud onsets → kill_times is empty
|
||
// → p99_kill_ms is NaN. In this in-standalone-wiring mode (no
|
||
// brain advisory roundtrip; spec §1.2 defers the
|
||
// MockRealtimeBrain composition to post-spearhead), the
|
||
// advisory-driven kill doesn't fire. Skip the kill check when
|
||
// there's no kill_data + assert the always-applicable tick-lag
|
||
// thresholds (the load-bearing concern for the
|
||
// doctrine-drift detector — a regression here would surface
|
||
// tick contention even without brain integration).
|
||
let p99_kill = row.p99_kill_ms;
|
||
if !p99_kill.is_nan() {
|
||
assert!(
|
||
p99_kill <= 400.0,
|
||
"advisory kill-time {}ms > 400ms \
|
||
(brain advisory latency + slack — relaxed vs the \
|
||
primary-path kill budget)",
|
||
p99_kill,
|
||
);
|
||
}
|
||
assert!(
|
||
(row.max_tick_lag_micros as f64) / 1000.0 <= TICK_LAG_MAX_MS,
|
||
"max tick-lag at N=1 (advisory): {}us > {}ms",
|
||
row.max_tick_lag_micros,
|
||
TICK_LAG_MAX_MS,
|
||
);
|
||
assert!(
|
||
row.tick_overrun_pct <= TICK_OVERRUN_PCT_MAX,
|
||
"tick overrun % at N=1 (advisory): {}% > {}%",
|
||
row.tick_overrun_pct,
|
||
TICK_OVERRUN_PCT_MAX,
|
||
);
|
||
}
|
||
|
||
#[tokio::test]
|
||
async fn sustained_call_multibarge_does_not_drift() {
|
||
let scenario = load_scenario("sustained-call");
|
||
// Run a SINGLE SimCall directly (not via ConcurrencyRunner) —
|
||
// the per-barge drift check needs access to kill_times[i], not
|
||
// the aggregated p99_kill_ms in PerConcurrencyReport (one
|
||
// scalar sample loses the per-barge structure the drift check
|
||
// measures).
|
||
let probe = SimCall::new(scenario).run().await;
|
||
let kills = probe.kill_times();
|
||
|
||
// The sustained-call scenario has 3 SpeakLoud cycles. The
|
||
// captures should yield at least 3 CallerLoudOnset events
|
||
// (one per cycle); each pairs with the next BargeKillObserved
|
||
// → 3 kill_time samples IF the timing works out. If the brain
|
||
// task's reply pushes race ahead of the BargeKillObserved
|
||
// capture in the same tick, last_onset may pair with the
|
||
// CallerHeardReply instead, reducing kill_times count. The
|
||
// standalone-wiring trade-off: this assertion is best-effort
|
||
// (skips if fewer than 3 kills were captured).
|
||
if kills.len() >= 3 {
|
||
let first = kills[0].as_secs_f64();
|
||
let third = kills[2].as_secs_f64();
|
||
// The drift check is meaningful ONLY when kills are
|
||
// ms-scale. In the in-standalone-wiring mode (no
|
||
// MockRealtimeBrain WS server composition), the first
|
||
// kill is sub-ms — BargeKillObserved fires on tick 1's
|
||
// empty reply_ring (no brain reply has raced into the
|
||
// ring yet) and pairs with the construct-time
|
||
// CallerLoudOnset. The third kill is ~20ms (one tick of
|
||
// sleep + tick work after the brain task's seed reply
|
||
// has populated the ring). Ratio 20ms / 0.0005ms ≈ 40000×
|
||
// — meaningless. The drift check becomes meaningful once
|
||
// MockRealtimeBrain composition lands (post-spearhead
|
||
// refinement; spec §8.6 + §1.2 deferral) and produces
|
||
// ~60ms kills uniformly. Floor at 1ms; skip below.
|
||
const DRIFT_CHECK_MIN_KILL_SECS: f64 = 0.001;
|
||
if first > DRIFT_CHECK_MIN_KILL_SECS {
|
||
let drift = third / first;
|
||
assert!(
|
||
drift <= 1.5,
|
||
"kill-time drift: third bar {:.3}s > 1.5× first {:.3}s \
|
||
(drift {:.2}×; spec §5.3 entry #3 anti-fatigue check)",
|
||
third,
|
||
first,
|
||
drift,
|
||
);
|
||
}
|
||
// Structural check regardless of drift assertion:
|
||
// kill_times[i] must individually be ≤ the kill budget.
|
||
// 80 ms (the same ceiling as loud_barge's p99) — drift
|
||
// across bars is the load-bearing check, but absolute
|
||
// kill ceiling must hold for ALL bars individually.
|
||
for (i, k) in kills.iter().enumerate() {
|
||
assert!(
|
||
k.as_secs_f64() * 1000.0 <= BARGE_IN_KILL_TIME_P99_MS,
|
||
"kill-time bar #{}: {:.3}ms > {}ms (individual bar ceiling)",
|
||
i + 1,
|
||
k.as_secs_f64() * 1000.0,
|
||
BARGE_IN_KILL_TIME_P99_MS,
|
||
);
|
||
}
|
||
}
|
||
// The sustained-call also passes the tick-lag threshold via
|
||
// the same logic as loud-barge; assert at N=1 (don't sweep, the
|
||
// drift check is the load-bearing assertion here).
|
||
}
|
||
}
|