rutster/docs/reviews/2026-07-04-scalability-infra-review.md

# 2026-07-04 — Scalability & infra-fit review

**Scope:** what single-process assumptions are baked into the code today that will fight
horizontal scaling, autoscaling, and load balancing later — and how rutster's shape maps onto
infra overall. **Not in scope:** the fused per-call vertical itself (deliberate, ADR-0002) or
documented later-rung deferrals with clean seams.

**Method:** 7-lens multi-agent survey over every crate (43 raw findings), dedup (17), adversarial
per-finding verification (each verifier read the cited code with a default-skeptical stance),
plus first-hand re-verification of 5 findings whose verifiers were lost to a session limit.
2 findings were rejected outright (one factually wrong, one a documented deferral); several were
downgraded. Severity: **blocker** = correctness breaks or scaling impossible at N>1 (or even N=1
in cloud); **major** = missing seam that calcifies as code accretes; **minor** = trivial when the
time comes.

---

## The infra model this system actually implies

Before the findings: the frame. rutster's fused vertical means **a call is pinned to a process
for its whole life** — DTLS/ICE state, jitter buffer, reflex state, the playout ring. Calls are
non-migratable, same as they were on Asterisk. That is not a defect; it defines the scaling model:

1. **Placement, not load balancing.** New calls get *placed* on a node (least-loaded /
   bin-packed); every subsequent operation on that call must reach the *owning* node. A
   round-robin L7 LB over the REST surface is the wrong mental model and breaks at N=2 today.
   The proven patterns: (a) a shared session→node directory (Valkey — already ratified,
   ADR-0005) behind a thin routing/API tier, or (b) the create-response carries a node-addressed
   URL (the LiveKit/Janus pattern). The CPaaS trunk path (ADR-0007 layer 1) converges on the
   same decision point: Twilio/Telnyx hand the media-stream URL out **per call at answer time**,
   so trunk dispatch and API routing are the *same* placement problem with the same answer.

2. **Capacity is tick-budget, and it must be measured to be scheduled.** The calls-per-node
   bound is the media thread exhausting its 10ms meta-tick. CPU% is the wrong autoscaling
   signal for a real-time engine (it lags and it lies); the right signals are
   **calls-in-flight** and **tick-overrun rate** (deadline misses). Neither exists today.
   Convergence worth exploiting: ADR-0010's pulled-forward benchmark/sim harness needs exactly
   this gauge — its primary readout should be "max sessions at <X% tick overrun," which then
   becomes the default admission cap.

3. **Scale-in = drain, and drain is slow.** Calls last minutes. Scale-in, rolling deploys, and
   spot reclamation all deliver SIGTERM; the only viable pattern is
   stop-accepting → bleed-out (bounded by a deadline) → terminate. Consequences: autoscaler
   scale-in protection / lifecycle hooks are mandatory for the engine tier, spot instances are
   effectively off the table for it, and deploys imply version-skew tolerance across the fleet
   (the tap protocol being versioned already is the right instinct).

4. **Three tiers with different scaling laws.** The decomposition the ADRs already imply:
   - **API/routing tier** — stateless once a session directory exists; scales on RPS; can live
     inside the engine binary at first and split out later without design change.
   - **Engine fleet** — scales on call capacity; drain-based lifecycle; per-node advertised
     media address; this is the tier every finding below is about.
   - **Brain fleet** — external by design; scales independently; the tap protocol is the seam,
     and wire semantics (resume/terminal-bye) are the part that calcifies.
   Substrate: Valkey (directory + bus + presence), object storage (CDR/recordings), the CPaaS
   or SBC at the edge. This is exactly "fused where fusion buys the wedge; composable where
   independent scaling matters" — the code just hasn't grown the horizontal seams yet.

5. **The silent N>1 correctness traps** are worse than the loud ones: a DELETE that 204s on the
   wrong node while the call (and its spend) keeps running; per-instance spend caps that
   quietly become N× the fleet cap; CDRs that evaporate with an OOM-kill. These bite after
   things *appear* to work.

---

## Blockers

### B1. Media addressing: loopback bind, no advertised-address concept, no port range
`crates/rutster-media/src/rtc_session.rs:179` binds `127.0.0.1:0`; `accept_offer`
(rtc_session.rs:236) advertises the raw local socket as the **sole** ICE host candidate. No
srflx/STUN, no NAT 1:1 advertised-IP config, no port-range allocator; `RtcSession::new()` is
zero-arg, and nothing in the chain main → `MediaThread::spawn` → `MediaCmd::Register` can carry
addressing config. The file is CI-seam-frozen (ci.yml:51–64 — a deliberate, updatable speed
bump, but it means the seam change is a "loud" PR by design).
**Why blocker:** RTP cannot ride the HTTP LB; even a single cloud instance behind NAT is
unreachable — the SDP answer says `127.0.0.1:<ephemeral>`. Fleet firewalling needs a bounded
port range. *(Adversarially verified: CONFIRMED, structurally baked in — str0m rejects 0.0.0.0
candidates, so this needs a bind-vs-advertised split, a new concept, not a constant swap.)*
**Fix shape:** `MediaAddressConfig { bind_interface, advertised_address, udp_port_range }`
threaded through the construction chain; advertised addr feeds `Candidate::host`, bind addr
feeds the socket; update the seam-gate hashes in the same PR.

### B2. Session placement: ownership is process-local, with a silent-failure DELETE
The only registry of live calls is the media thread's private
`HashMap<ChannelId, ThreadSession>` (media_thread.rs:164). Routes reach it over a
process-local mpsc. ADR-0005's Valkey has **zero code presence** — no dep, no crate, no trait.
At N=2 behind any LB:
- `POST /v1/sessions/:id/offer` on a non-owning node → "not found" → **404** (loud failure);
- `DELETE /v1/sessions/:id` on a non-owning node → `sessions.remove` finds nothing, reply is
  `()` regardless, route returns **204 unconditionally** (routes.rs:131–138) — the hangup
  *reports success while the call keeps running*, burning trunk minutes and brain tokens. Once
  the spend gate exists, "kill this runaway call" failing silently is a security-posture hole,
  not just a routing bug.
*(Verified first-hand after the workflow's verifier was lost to a session limit; every line
re-checked.)*
**Fix shape:** session directory (ChannelId → node) written at Register/Delete — Valkey per
ADR-0005 is the obvious backend — plus one of: routing tier, consistent-hash LB, or
node-addressed URLs returned at create. The `MediaCmd` seam gives the writes a clean landing
spot; the missing piece is the *concept*, which also appears nowhere in ARCHITECTURE.md's
horizontal-platform list.

## Major

### M1. No graceful drain — SIGTERM hard-drops every in-flight call
`MediaCmd::Shutdown` is `sessions.clear(); return` (media_thread.rs:224–232) — it even skips
the 750ms per-session tap teardown that `Delete` performs. axum's graceful shutdown drains
HTTP requests; calls are not HTTP requests. main.rs:55–59 says it outright: "No in-flight call
preservation story in the dev loop." *(CONFIRMED; not yet structurally baked — the fix is
contained plumbing today — but it must land before the planned threadpool-shard graduation of
this exact thread, or the two-state running/dead lifecycle gets baked into the shards, the
readiness surface, and the trunk routing.)*
**Fix shape:** a `Drain` lifecycle state: reject new Registers (503), flip readiness off, keep
ticking until the map empties or a deadline, then Shutdown.

### M2. No admission control, no capacity signal
Register inserts unconditionally; the fixed `sleep(META_TICK)` (media_thread.rs:303) ignores
how long the tick took, so saturation silently stretches the tick and degrades **every** call's
20ms pacing at once — the worst overload mode for a real-time engine, and invisible: the
session count surfaces exactly once, in the shutdown log. A node can never say "full" and an
autoscaler has nothing to read. *(CONFIRMED. Notably: this repo documents its deferrals
meticulously, and this one is nowhere documented — a genuinely silent gap. ADR-0010's benchmark
needs the tick-lag gauge to produce its headline number.)*
**Fix shape:** configurable max-sessions in Register (Err → 503), per-tick elapsed-vs-budget
gauge, `MediaCmd::Stats { reply }` for the readiness/metrics surface.

### M3. ADR-0005 exists only on paper — lifecycle events die with the process
No bus, no `EventSink` trait, nothing durable: register/Connected/evict/Shutdown are tracing
lines; the `Delete` handler drops `TapConn` without ever reading `conn.metrics`, breaking the
in-code promise at tap_engine.rs:82–84 ("the eventual CDR/ringbus emitter"); `Channel` carries
only a **monotonic `Instant`** — there is no wall-clock timestamp in the entire production
workspace from which a CDR could even be built. An OOM-killed node erases all evidence of its
calls. *(CONFIRMED, leaning baked-in: the "tracing is the record" assumption is reproducing
slice-over-slice, and thread-shard graduation will multiply the emission points.)*
**Fix shape:** minimal `EventSink` trait owned by the media thread (log-backed now, Valkey
streams later), CDR-shaped started/ended events with wall-clock time + metrics snapshots.

### M4. Tap protocol has no resume or terminal semantics — the calcifying one
Brain bye, error, and stream-end all funnel into infinite re-dial (5s cap, forever); every
reconnect restarts `seq_egress = 0` with a hello carrying no epoch/resume token; the reference
brain acks any hello and opens a fresh OpenAI session. Mid-call reconnect = silently amnesiac
brain **today, even at N=1**; a brain that deliberately ends a session gets re-dialed for the
rest of the call. At brain-fleet scale: reconnects land on different instances with no
protocol-level way to detect lost context, plus re-dial storms. *(CONFIRMED, baked in — this is
a **wire-protocol** change, v2 + brain-side changes, the most expensive kind of retrofit. The
protocol being versioned is the one mercy.)*
**Fix shape:** resume token/connection epoch in hello + resume-ack/reject from the brain +
terminal-vs-transient bye reason that exits the retry loop.

### M5. Spend-gate accounting locality is undefined — and one ADR sentence steers it wrong
The crate is a stub (nothing baked yet), but ADR-0009 prescribes where the *check* sits without
distinguishing enforcement point from **accounting ledger**. Per-instance counters on a fleet =
N× every cap, and toll-fraud thresholds that never trip because attempts spread across nodes.
The subtle trap found in verification: ADR-0005's "the bus is NOT the source of truth for
billing-critical state" is the sentence most likely to push the step-6 implementer toward local
counters — but enforcement counters (rate/spend state) are not billing truth (durable CDR);
they're different things with different stores. *(WEAK/major-borderline — nothing accretes yet;
the fix today is one sentence.)*
**Fix shape:** amend ADR-0009 now: "in-process enforcement, **shared accounting**" — gate built
against a ledger trait with atomic check-and-reserve (in-memory impl for N=1, Valkey for N>1).

### M6. Metrics are write-only — nothing aggregates, nothing exports
`TapMetrics`/`ReflexMetrics` are per-session atomics with a snapshot method that production
code never calls; no process-level registry, no calls-in-flight gauge, no Prometheus/OTel
export. An LB health check and an autoscaler would both be reading a static HTML page today.
*(Verified first-hand. The atomics+snapshot shape is export-friendly — the missing piece is
the aggregation registry and one scrape endpoint, medium plumbing.)*

### M7. Session teardown stalls every live call on the node
`MediaCmd::Delete` runs `tokio_handle.block_on(timeout(750ms, &mut conn.join))`
(media_thread.rs:205–217) **on the media thread, inside the tick loop**. One teardown with a
slow/unresponsive brain freezes the 20ms loop for every other call on the node — up to ~37
missed frames each; several Deletes drained in one batch stack sequentially. Under fleet churn
(hangups are constant in a call center) this is a per-node isolation failure that worsens with
density. *(Verified first-hand.)*
**Fix shape:** hand teardown to a tokio task (fire close_tx, spawn the bounded join await);
the tick loop should never block on brain I/O — same discipline the tap pipe already follows.

## Minor

- **Readiness/liveness absent** (routes.rs:151): only probe-able route is `GET /` static HTML —
  returns 200 while the media thread is dead (its panic isn't monitored; sessions just 500).
  Downgraded to minor by verification: the `MediaCmd` seam makes `/healthz` + `/readyz` trivial
  additions, nothing accretes around their absence. Lands naturally with M1/M2.
- **Trunk dispatch** (rutster-trunk stub): the "which instance gets this call's media fork"
  question. Downgraded: Twilio/Telnyx hand out the stream URL per call by design, so answer-time
  binding is the *default* usage — this needs one design constraint written into the step-5
  spec, not code today. It is, however, the same placement concept as B2 — solve once.
- **Tap URL loopback-only + wss:// not compiled** (routes.rs:87, workspace Cargo.toml:48):
  documented step-6 deferral with the per-session override seam already in place. Non-obvious
  detail from verification: no crate enables a TLS feature on tokio-tungstenite, so relaxing the
  validator alone wouldn't help — the dial capability itself is compiled out, and the env
  default goes through the same validator (no escape hatch).
- **HTTP bind hardcoded** `0.0.0.0:8080` (main.rs:44); QUICKSTART literally says "edit main.rs
  to change the port." `RUTSTER_TAP_BIND` in the brain binary is the pattern to copy.
- **Brain endpoint static per-call URL** (tap_engine.rs:271): downgraded on verification —
  `connect_async` re-resolves DNS each dial, so a brain fleet behind DNS/VIP works today; the
  real calcification is M4's missing resume semantics, not URL indirection.

## Rejected in verification (for the record)

- *"EnvFilter `rutster=info` silences library-crate warnings"* — factually wrong;
  tracing-subscriber matches by string prefix, so `rutster` covers `rutster_media::*` et al.
  (verified empirically against the locked 0.3.23).
- *"No packaging/compose artifact"* — documented later rung (DEVELOPMENT.md:172), purely
  additive, nothing accretes around its absence.

## What's genuinely clean (don't break these)

- **`ChannelId` = UUIDv4** — globally unique, fleet-safe, doubles as the wire session id.
- **The `MediaCmd` command channel** — the load-bearing seam. Drain, stats, admission,
  directory writes, and event emission all have a natural landing spot because of it.
- **Bounded backpressure with drop-counting on both tap directions** (32-frame mpscs, 5-frame
  playout ring, drop-oldest, every drop counted) — a slow brain cannot eat unbounded memory.
- **Per-call `tap_url` override** — per-call brain routing already exists.
- **Versioned tap protocol** (`v:1` enforced at decode) — M4 has an evolution path.
- **Zero production global statics** in the workspace.
- **Tap is outbound from the call-owning node** — connection follows call ownership; exactly
  right for multi-node.

## Sequencing — what to bake in now vs. later

The point is not to build the fleet now. It's that a handful of *concepts* are cheap to plant
today and expensive to retrofit once slices accrete around their absence:

**Now / next slice (cheap seams, prevents calcification):**
1. `MediaAddressConfig` (B1) — also unblocks the first cloud demo at N=1.
2. Admission cap + tick-lag gauge + `MediaCmd::Stats` (M2) — double-billed to ADR-0010's
   benchmark harness, which needs the same instrumentation.
3. Drain vocabulary (M1) + `/healthz` + `/readyz` — before thread-sharding.
4. `EventSink` trait + wall-clock timestamps on `Channel` (M3).
5. One-sentence ADR-0009 amendment: in-process enforcement, shared accounting (M5); clarify
   ADR-0005's "not source of truth" ≠ "no enforcement counters in Valkey."
6. Reserve resume-token + terminal-bye in the tap protocol's v2 plan (M4) — wire semantics
   are the most calcifying surface in the whole system.
7. Move Delete teardown off the tick loop (M7). Trivial, and it's a latency bug today.
8. `RUTSTER_HTTP_BIND` (copy the existing pattern).

**When N>1 actually lands (design then, not now — but write the concepts down):**
- Valkey session directory + placement/routing tier (B2) — add "call placement" to
  ARCHITECTURE.md's horizontal-platform list so the concept exists on paper.
- Trunk step-5 spec constraint: media-stream URL handed out per call at answer time (placement
  decision shared with B2).
- Metrics aggregation registry + scrape endpoint (M6).
- TLS on tap dial + validator policy (documented step 6).

---
*Method note: multi-agent review (7 survey lenses → dedup → adversarial verify → severity
calibration), ~1M tokens of subagent reading; 5 of 17 findings re-verified by hand after their
verifiers hit a session limit. All file:line citations checked against working tree @ d696536.*