rutster/docs/superpowers/specs/2026-07-05-deployment-topology-design.md

Rutster deployment topology — one binary, three blessed shapes

• Status: Draft (pending review)
• Date: 2026-07-05
• Origin: ADR-0008 explicitly reserved a deployment topology ADR "once that design closes." This spec closes it. Brainstormed 2026-07-05; the TLS/edge fork was researched via a six-family survey — see the companion TLS/edge decision brief.
• Produces: ADR-0011 (deployment topology) — drafted from §2–§4 of this spec, Proposed, ratified per repo process — plus the artifacts and code slices in §5–§6.
• Depends on (already merged): slice-5/seams (drain lifecycle, /readyz, admission cap, EventSink, MediaAddressConfig, config.rs env-parser pattern); slice-4½ benchmark/sim (S1–S8, CI-regressed — the future source of the real per-node admission cap).
• Related: ADR-0002 (fused per-call vertical), ADR-0005 (Valkey = bus + KV + presence; media never rides the bus; bus is never billing source-of-truth), ADR-0006 (two inbound protocols only; tap is egress-only), ADR-0007 (rented transport ⇒ public HTTPS/WSS reachability), ADR-0009 (ledger trait: in-memory single-node, Valkey-backed fleets), ADR-0010 (sequencing this spec must not reorder), README pillar 6 ("one binary, one bus, one deploy").

---

TL;DR

The FOB never decomposes; topology is a configuration property, not a code property. The same binary runs in every blessed shape; only RUTSTER_* env and what is deployed alongside it differ. "Microservices" in rutster means green-zone services get their own processes/containers — never that the per-call vertical splits (ADR-0002).

Three blessed topologies:

	Shape	Status after this work
T1 — Solo	One all-in-one container: s6 supervises Caddy + FOB + brain + valkey-server (EventSink consumer day one; ledger/directory later). docker run with a domain + Twilio creds → taking calls.	Shipped artifact
T2 — Modular	Compose stack: caddy, engine, brain (shared netns with engine), valkey as separate services. compose up is the pillar.	Shipped artifact (reference deployment)
T3 — Fleet	N symmetric nodes + shared green zone (one Valkey, object storage). Node-addressed URLs, answer-time placement via Valkey presence.	ADR chapter only (paper)

Riders shipping with the artifacts: TCP_NODELAY + app-level WS-ping hygiene, trunk config fixes, in-process TLS Phase 1 (BYO-cert, hot-reload), a hand-rolled Prometheus /metrics endpoint, ValkeyEventSink (first Valkey consumer), image-publish CI, and a docs/deploy/ documentation tree with a copy-paste-to-first-call acceptance bar.

Six things this design deliberately is NOT:

• Not a decomposition of the FOB into services. Media/reflex/tap/spend stay one process, one trust domain, per ADR-0002. Rejected permanently, not deferred.
• Not serverless. Lambdas are categorically unfit for the engine: non-migratable multi-hour calls, per-session UDP sockets, in-process media state. Webhook-on-Lambda buys nothing because placement needs fleet state anyway. Stated as a rejected shape in ADR-0011.
• Not k8s manifests/Helm. Compose-first (pillar 6). k8s notes live in docs only (drain vs terminationGracePeriodSeconds); manifests are a later rung if demand appears.
• Not the durable CDR pipeline. ValkeyEventSink is evidence preservation on a capped stream — explicitly NOT the billing ledger. CDR → object storage stays deferred (ADR-0005 source-of-truth rule).
• Not the zero-egress / air-gap profile. T1 is about one-command simplicity, not isolation. Zero-egress (SBC layer-2 ingress + self-hosted brain) is named as a defined-but-deferred profile in ADR-0011, one footnote section, no code.
• Not in-binary ACME (Phase 2). Deferred behind named triggers (§3.4). No turnkey crate fits the tree today (rustls-acme conflicts with the deny.toml rcgen/x509-parser posture); owning a renewal state machine in a 45-day-cert world is a deliberate, triggered decision.

---

1. Scope

1.1 In scope

• ADR-0011 (deployment topology): the three shapes, the edge doctrine, the fleet placement paradigm, the rejected shapes.
• Artifacts under deploy/: multi-stage Dockerfile (four first-party images), compose.yaml, Caddyfile, .env.example.
• Code: TCP_NODELAY, app-level WS pings, trunk config fixes (webhook base URL, credential wiring, trusted-proxy header posture), rustls Phase 1 (BYO-cert TLS), /metrics, ValkeyEventSink.
• CI: containerized smoke suite (incl. Caddy-reload-during-live-call), image publish on tag.
• Docs: docs/deploy/ tree (§10).

1.2 Out of scope (named in ADR-0011 as deferred, so the boundary is deliberate)

• T3 fleet implementation (presence heartbeats, directory redirects, placement code).
• Object-storage CDR durability; multi-tenancy (needs its own ADR before schemas ossify); wss:// tap + brain fleet + resume tokens (step 6); media-thread sharding graduation; in-binary ACME (Phase 2); SBC layer-2 wire contract; arm64 images.

---

2. Topologies

2.1 T1 — Solo (all-in-one container)

One image (rutster-allinone), tini/s6-overlay as PID 1 supervising four processes:

Process	Role	Binding
caddy	Edge: ACME, TLS termination, WS proxy to FOB	:443/:80 public
rutster (FOB)	Engine	127.0.0.1:8080 (behind Caddy); media UDP direct
rutster-brain-realtime	Brain	127.0.0.1:8082 loopback tap (as today)
valkey-server	Dark on day one except EventSink (§5.6)	127.0.0.1:6379

Volumes (both non-negotiable, documented loudly):

• /data — Caddy cert/ACME state. Loss risks Let's Encrypt duplicate-cert lockout (5/week) → total inbound outage class.
• /var/lib/valkey — stream/state persistence.

WebRTC media UDP goes direct to the FOB, never through Caddy: host networking recommended; otherwise published RUTSTER_MEDIA_PORT_RANGE + RUTSTER_MEDIA_ADVERTISED_IP (NAT 1:1 model, no STUN — unchanged from slice-5/seams).

Valkey ships bundled from v1 before most consumers exist (decision 2026-07-05): the operator contract (volumes, ports, upgrade shape) is stable from day one, so the spend ledger and fleet directory land later without changing the deployment shape. Cost accepted: ~15 MB and one mostly-idle process.

2.2 T2 — Modular (compose stack) — the reference deployment

Services: caddy (image rutster-edge), engine (rutster-engine), brain (rutster-brain), valkey (upstream valkey/valkey).

• Brain shares the engine's network namespace (network_mode: "service:engine") so the loopback-only tap posture (resolve_tap_url rejects non-loopback until step 6) survives unchanged. ADR-0011 notes the brain graduates to its own netns when the wss:// tap lands.
• Valkey is its own service on the compose network — network-near, satisfies ADR-0005's sub-ms rule.
• stop_grace_period: 660s paired with RUTSTER_DRAIN_DEADLINE_SECS=600 (§8).
• BYO-proxy supported: disable the caddy service; the engine keeps its plaintext :8080 mode; tuned-timeout snippets for nginx/HAProxy/Traefik ship in docs (§10).

2.3 T3 — Fleet (ADR chapter only; no code in this epoch)

N symmetric nodes, each running the T1 process set minus valkey, plus a shared green zone: one Valkey (presence + directory + ADR-0009 spend ledger), object storage (named as required, still deferred). Wildcard DNS *.pbx.domain; per-node certs (§3.3). Per-node admission cap = the ADR-0010 benchmark number (replacing the RUTSTER_MAX_SESSIONS=64 placeholder). Scale-in is drain-then-terminate only; spot instances rejected for the engine tier (2026-07-04 review). Placement paradigm: §4.

---

3. Edge doctrine

3.1 CPaaS invariants (the spec any edge must satisfy — research §2)

1. Publicly-CA-trusted cert; no self-signed path exists (Twilio error 31910). Auto-renewal is availability-critical.
2. wss:// on 443, open to the internet: Twilio publishes no egress IPs, offers no mTLS. Auth is application-layer only (HMAC signatures).
3. One WS per call, up to 24 h, 50 msg/s/direction at 20 ms cadence; zero tolerance for frame buffering, lifetime caps, or write-side idle timers. Neither Twilio nor Telnyx documents WS keepalive — keepalive is entirely our job (§5.2).
4. <Stream> URLs carry no query strings — wss routing keys on hostname/path only (constrains §4).
5. X-Twilio-Signature is HMAC over the URL as Twilio saw it: the edge must forward X-Forwarded-Proto/Host honestly and the FOB must reconstruct the public URL (§5.3).
6. Webhook budget: sub-second for UX, 15 s hard cap — no blocking issuance in the handshake path.
7. TLS 1.2+1.3, mainstream ECDHE, never 1.3-only; never pin Twilio certs.

Design for Twilio; Telnyx/Vonage come for free (they are strictly less demanding, plus optional source-IP allowlisting as extra hardening Twilio can't use).

3.2 Bundled edge: Caddy (both shipped artifacts, same build, same Caddyfile)

Custom xcaddy build (rutster-edge image): Caddy core (Apache-2.0) + curated DNS-01 plugin set — cloudflare, route53, porkbun, hetzner, desec (duckdns excluded: no license file). Zero-downtime in-memory cert renewal; verified protocol-unaware WS tunneling (no frame buffering). Config: ~6-line Caddyfile with tuned timeouts, honest X-Forwarded-*, stream_close_delay above max call duration — and a CI e2e case for config-reload-during-live-call, because the upstream mitigation has an open bug trail (caddy #6420/#7222) and is not trusted untested.

3.3 Fleet cert story (ADR chapter)

Per-node public TLS names are CPaaS-imposed (invariant 4 + node-addressed placement). Two blessed patterns: wildcard *.pbx.domain via DNS-01 with central issuance/distribution, or per-node distinct certs (node-N.domain) where the wildcard private key must not live on every node. Traps stated in the ADR: N nodes independently requesting the identical wildcard hit the LE duplicate-cert limit; Caddy on-demand TLS is rejected for node names (blocking first handshake vs the webhook budget).

3.4 In-process TLS: staged (the ratified end-state, not adopted wholesale now)

ADR-0008 claims the trust-domain edge as security-constitutive FOB territory, and str0m already terminates DTLS-SRTP in-process on the same aws-lc-rs provider — "keep TLS out of the FOB" is not the objection. The objection is economics: owning an ACME renewal state machine forever.

• Phase 1 (this epoch, §5.4): axum-server + rustls with operator-supplied certs, hot-reload without dropping live WS. Plaintext-behind-Caddy stays the default.
• Phase 2 (in-binary ACME) triggers, named in ADR-0011: (a) Let's Encrypt dns-persist-01 confirmed GA (closes the wildcard/CGNAT issuance gap; instant-acme already parses it); (b) a field incident where Caddy reload/stream_close_delay drops live calls; (c) fleet or VPS-forwarder topology making encrypted-to-the-binary a selling point; (d) maintenance fatigue with the curated xcaddy plugin build.

3.5 Tunnels and cloud LBs

• Tunnels are dev/demo-only, stated bluntly. Cloudflare Tunnel and ngrok see plaintext audio at the vendor edge (unconsented subprocessor — DPA/BAA/PCI failure), document mid-call WS kills, and cloudflared has an open Twilio handshake bug (#1465). ngrok is the blessed 5-minute demo path (only tunnel with a proven Media Streams record). Tailscale Funnel: privacy-clean single-user demo (TLS terminates on-node), unsizable beyond that.
• No tunnel carries inbound UDP → homelab behind CGNAT is PSTN-only, period. Production graduation path: cheap VPS + WireGuard as a dumb TCP forwarder with TLS terminating at home (forwarder physically cannot read audio), or run the engine on the VPS.
• AWS appendix (docs only, never a shipped artifact): ALB + wildcard ACM + host-header rules; mandatory attribute overrides idle_timeout=4000, client_keep_alive=604800, header forwarding. Never NLB-TLS (fixed 350 s silent drop, no HTTP context for signatures).

---

4. Fleet placement paradigm (ADR chapter; the trunk webhook handler and session API grow

toward this, no fleet code in this epoch)

Node-addressed URLs + Valkey presence; no router tier. (Decision 2026-07-05 over routing-tier-proxies-everything and distinguished-control-node.)

• Nodes heartbeat capacity into Valkey (TTL'd presence keys — the ADR-0005 presence layer).
• Placement happens at answer time by whichever node the request lands on (api.domain, DNS round-robin): read fleet capacity → pick owner → return TwiML whose Stream URL is wss://node-K.pbx.domain/.... The media WS dials the owning node directly — zero extra hops, consistent with connection-follows-ownership.
• WebRTC: create-session response carries the owner's node-addressed base URL + advertised media IP (UDP is already direct).
• Misdirected control ops (e.g. DELETE hitting a non-owner) consult the Valkey directory and 307-redirect to the owner.
• Degraded mode (Valkey dark): the answering node self-places if it has capacity; presence heals via TTL on recovery. Fail-safe degradation, consistent with ADR-0009's posture.
• Single-node is the degenerate case of the same code: a one-entry directory.

Rationale: Twilio's answer-time Stream URL is purpose-shaped for this (invariant 4 permits hostname routing); a proxying router tier would put every 20 ms frame through an extra userspace hop and re-create the SPOF the symmetric design avoids.

---

5. Code changes (workspace)

5.1 TCP_NODELAY (hygiene, unconditional)

Set on every accepted socket (axum #2521 — Nagle can coalesce 20 ms WS frames on the plaintext listener behind any proxy). Latency assertion joins the CI-regressed sim-bench.

5.2 App-level WS pings (hygiene, unconditional)

Engine-originated ping/pong on the trunk media-stream WS (and any future long-lived WS surface; WebRTC signaling is plain HTTP today, so the media stream is the only such socket), interval configurable (RUTSTER_WS_PING_SECS, default 20). Belt-and-braces against every idle timer between the FOB and the CPaaS (invariant 3).

5.3 Trunk config fixes + trusted-proxy posture

• TwiML Stream URL derived from RUTSTER_TWILIO_WEBHOOK_BASE (kills the hardcoded loopback at routes.rs:258).
• config::twilio_credentials wired into startup validation (parsed-but-never-called today).
• Trusted-proxy: X-Forwarded-Proto/Host honored only from the configured edge (RUTSTER_TRUSTED_PROXIES, CIDR list; empty = headers ignored); signature validation reconstructs the public URL from them (invariant 5).

5.4 rustls Phase 1 — BYO-cert in-process TLS

axum-server + rustls (already in-tree transitively): RUTSTER_TLS_CERT/RUTSTER_TLS_KEY enable a TLS listener; hot-reload on file change without dropping live WS (arc-swap resolver). Always compiled, runtime-gated; plaintext :8080 behind Caddy remains the artifact default.

5.5 /metrics — hand-rolled Prometheus text endpoint

Reads the existing atomics: active sessions, tick_overruns, last_tick_micros, admission rejects, EventSink drops (§5.6). Zero new deps; node-level cardinality only. Own internal bind RUTSTER_METRICS_BIND (default 127.0.0.1:9090) — never routed through Caddy.

5.6 ValkeyEventSink — first Valkey consumer

Implements the existing EventSink trait: XADD of current lifecycle events as JSON to a capped stream (XADD MAXLEN ~), strictly fire-and-forget — Valkey down ⇒ count-and-drop (counter surfaces in /metrics), never blocks the call path. RUTSTER_VALKEY_URL unset ⇒ today's TracingEventSink. Client crate: redis-rs (BSD-3; verify against deny.toml in the slice). Closes the "OOM-killed node erases all evidence of its calls" gap. Evidence preservation, not the billing ledger (ADR-0005 rule restated in code docs).

5.7 Env additions (all follow the config.rs fail-fast parser pattern)

Var	Default	Used by
RUTSTER_TLS_CERT / RUTSTER_TLS_KEY	unset (plaintext)	§5.4
RUTSTER_TRUSTED_PROXIES	empty (ignore forwarded headers)	§5.3
RUTSTER_WS_PING_SECS	20	§5.2
RUTSTER_METRICS_BIND	127.0.0.1:9090	§5.5
RUTSTER_VALKEY_URL	unset (tracing sink)	§5.6
RUTSTER_DRAIN_DEADLINE_SECS	unchanged code default 0; 600 in .env.example	§8

---

6. Artifacts

6.1 Images (four first-party, published to the git.adlee.work registry)

Image	Contents	Base
rutster-allinone	s6-overlay: Caddy + FOB + brain + valkey-server	debian-slim + libopus0
rutster-engine	FOB only	debian-slim + libopus0
rutster-brain	brain only	debian-slim
rutster-edge	custom xcaddy build (§3.2)	scratch/alpine

Multi-stage deploy/Dockerfile, pinned 1.85 toolchain per rust-toolchain.toml. Distro libopus0 over static-bundled (matches dev; static noted as future size work). linux/amd64 first; arm64 deferred. Bundled-binary licenses are aggregation-clean vs GPL-3 (Caddy Apache-2.0, valkey BSD-3, s6 ISC) — noted for the cargo-deny-adjacent image license scan in CI.

6.2 deploy/ tree

Dockerfile, compose.yaml, Caddyfile, .env.example (documented defaults incl. drain 600 s), plus the smoke-test harness consumed by CI (§9).

6.3 Publish CI

Gitea Actions job on tag push: build all four images, push to the registry. Runs after (and gated on) the existing fmt/clippy/test/deny/sim-bench pipeline.

---

7. Data flow (T1/T2)

PSTN: Twilio → HTTPS webhook → Caddy → FOB /v1/trunk/webhook (signature validated against URL reconstructed from trusted X-Forwarded-*) → TwiML with derived wss URL → Twilio dials media stream → Caddy (unbuffered WS tunnel) → FOB WS upgrade → MediaCmd::RegisterTrunk → media-thread tick (G.711 ↔ PCM) ↔ brain tap (loopback). WebRTC: signaling via Caddy; SDP answer advertises media IP; DTLS-SRTP UDP direct to FOB. Events: EventSink → Valkey stream (loopback T1 / compose network T2). Metrics: Prometheus scrapes the internal bind. T3 flow as §4.

---

8. Failure modes (worst first)

Failure	Handling
Caddy /data loss	Persistent volume mandatory; LE duplicate-cert lockout documented as total-inbound-outage class
Caddy reload vs live calls	stream_close_delay > max call duration + CI reload-during-call e2e (§9); documented reload procedure
Valkey down	EventSink counts-and-drops; calls unaffected. T3: degraded self-placement (§4)
Node/FOB death	Calls on it die (non-migratable — stated honestly); s6 restarts in T1; Valkey stream preserves evidence; T3 presence TTL stops new placements
Brain death	FOB reflexes continue (the FOB point); amnesiac reconnect documented; resume tokens are step 6
Drain vs orchestrator	.env.example drain 600 s; compose stop_grace_period 660 s (grace > drain); systemd TimeoutStopSec/k8s equivalents in docs
Quiet WS killed by any middle timer	§5.2 pings + tuned edge timeouts

---

9. Testing

• Code: NODELAY latency assertion in sim-bench; webhook URL derivation; signature validation behind forwarded headers (trusted and untrusted source cases); WS ping emission; TLS hot-reload (self-signed test certs); ValkeyEventSink XADD + drop-on-down (real valkey service container in CI).
• Artifacts (new CI muscle): build all images; all-in-one smoke — boots, Caddy terminates TLS via its internal CA (no ACME in CI), full sim call through the real edge→FOB WS path; compose smoke — all services healthy, a lifecycle event lands in the Valkey stream; Caddy config reload during a live simulated call, asserting zero drops.
• Existing seam-gates (loop_driver/rtc_session blob hashes), MSRV matrix, cargo-deny (now covering redis-rs), sim-bench single-threaded — all unchanged and must stay green.

10. Documentation (docs/deploy/)

topologies.md (three shapes + decision guide), quickstart-docker.md, homelab.md (CGNAT truth: PSTN-only behind tunnels; ngrok dev path; VPS+WireGuard graduation), aws.md (ALB appendix, three mandatory overrides, never-NLB-TLS), reverse-proxies.md (tuned-timeout snippets for BYO nginx/HAProxy/Traefik incl. the universal 60 s idle-timeout footgun), certificates.md (ACME paths, wildcard traps, duplicate-cert lockout).

Acceptance bar: every topology doc contains a copy-paste path from zero to first call — or the honest statement of what cannot work there (e.g. CGNAT WebRTC).

11. Sequencing — slices

Slice	Contents	Depends
A	Hygiene: §5.1 NODELAY + §5.2 pings + §5.3 trunk fixes/trusted-proxy	—
B	Images + compose + Caddyfile + smoke CI (§6, §9)	A
C	rustls Phase 1 (§5.4)	A
D	/metrics (§5.5)	—
E	ValkeyEventSink (§5.6)	—
F	Publish CI (§6.3)	B
G	docs/deploy/ tree (§10) + ADR-0011 drafting/ratification	B for accuracy

C–F order-independent after their deps. Additive to the ADR-0010 sequence (benchmark → escalation → trunk → spend) — not a reordering. The artifacts ship with WebRTC calling live now; PSTN lights up end-to-end the moment the remaining trunk work lands (trunk registration is a placeholder NACK today; the container wiring will be ready).

12. Open questions deliberately left open (tracked, not blocking)

dns-persist-01 GA timeline (gates rustls Phase 2 and dissolves the xcaddy plugin burden); object-storage CDR shape; multi-tenancy ADR (before CDR/config schemas ossify); brain fleet + tap auth/resume (step 6); media-thread sharding graduation; SBC layer-2 wire contract; the real admission-cap number from the ADR-0010 benchmark.