rutster/docs/adr/0011-deployment-topology.md

ADR-0011 — Deployment topology: one binary, three blessed shapes

• Status: Proposed
• Date: 2026-07-05
• Closes: the deployment-topology ADR reserved by ADR-0008's worked example ("the deployment topology itself may get its own ADR once that design closes").
• Origin: 2026-07-05 deployment-topology spec §2–§4. The TLS/edge fork was researched via a six-family survey — see the companion TLS/edge decision brief.
• Related: ADR-0002 (the fused per-call vertical no topology may split), ADR-0005 (Valkey = bus + KV + presence; media never rides the bus; the bus is never billing source-of-truth), ADR-0006 (two inbound protocols only; the tap is egress-only), ADR-0007 (rented transport ⇒ public HTTPS/WSS reachability is non-negotiable), ADR-0008 (the FOB/green-zone rule this ADR applies to deployment), ADR-0009 (ledger trait: in-memory single-node, Valkey-backed fleets), ADR-0010 (sequencing this ADR must not reorder), README pillar 6 ("one binary, one bus, one deploy").

Context

ADR-0008 made the build-vs-reuse doctrine physical in a worked example — the all-in-one container — and reserved the topology ADR for when the design closed. Two things have since closed it: the rented transport (ADR-0007) is merged, so every deployment must now present a public HTTPS/WSS edge to a CPaaS; and slice-5/seams landed the operational primitives a deployment story needs (drain lifecycle, /readyz, admission cap, MediaAddressConfig, EventSink).

The CPaaS side imposes a hard external spec on any edge rutster ships (researched in the decision brief; restated here because these invariants are load-bearing for every decision below):

1. Publicly-CA-trusted cert; no self-signed path exists (Twilio error 31910). Auto-renewal is availability-critical.
2. wss:// on 443, open to the whole internet: Twilio publishes no egress IPs and offers no mTLS. Auth is application-layer only (HMAC signatures).
3. One WS per call, up to 24 h, 50 msg/s/direction at 20 ms cadence; zero tolerance for frame buffering, connection-lifetime caps, or write-side idle timers. Neither Twilio nor Telnyx documents WS keepalive — keepalive is entirely our job.
4. <Stream> URLs carry no query strings — wss routing keys on hostname/path only.
5. X-Twilio-Signature is HMAC over the URL as Twilio saw it: the edge must forward X-Forwarded-Proto/Host honestly and the FOB must reconstruct the public URL.
6. Webhook budget: sub-second for UX, 15 s hard cap — no blocking issuance in the handshake path.
7. TLS 1.2+1.3, mainstream ECDHE, never 1.3-only; never pin Twilio certs.

Design for Twilio; Telnyx/Vonage come for free (strictly less demanding, plus optional source-IP allowlisting as extra hardening Twilio can't use).

On the other side, the FOB's own physics constrain every shape: per-call state is in-process and non-migratable (the fused vertical, ADR-0002), WebRTC media is per-session UDP direct to the process, and calls run for hours. Any topology that assumes migratable, short-lived, connectionless workloads is disqualified before it starts.

Decision

The FOB never decomposes; topology is a configuration property, not a code property. The same binary runs in every blessed shape; only RUTSTER_* env and what is deployed alongside it differ. "Microservices" in rutster means green-zone services get their own processes/containers — never that the per-call vertical splits (ADR-0002).

The three blessed shapes

T1 — Solo (all-in-one container) — shipped artifact

One image (rutster-allinone), s6-overlay as PID 1 supervising four processes:

Process	Role	Binding
caddy	Edge: ACME, TLS termination, WS proxy to FOB	:443/:80 public
rutster (FOB)	Engine	127.0.0.1:8080 (behind Caddy); media UDP direct
rutster-brain-realtime	Brain	127.0.0.1:8082 loopback tap (as today)
valkey-server	Dark on day one except EventSink	127.0.0.1:6379

Two volumes, both non-negotiable: /data (Caddy cert/ACME state — loss risks the Let's Encrypt duplicate-cert lockout, a total-inbound-outage class of failure) and /var/lib/valkey (stream/state persistence). WebRTC media UDP goes direct to the FOB, never through Caddy: host networking recommended; otherwise published RUTSTER_MEDIA_PORT_RANGE + RUTSTER_MEDIA_ADVERTISED_IP (NAT 1:1 model, no STUN — unchanged from slice-5/seams).

Valkey ships bundled from v1 before most consumers exist: the operator contract (volumes, ports, upgrade shape) is stable from day one, so the spend ledger (ADR-0009) and the fleet directory land later without changing the deployment shape. Cost accepted: ~15 MB and one mostly-idle process.

T2 — Modular (compose stack) — shipped artifact, the reference deployment

Services: caddy (image rutster-edge), engine (rutster-engine), brain (rutster-brain), valkey (upstream valkey/valkey).

• The brain shares the engine's network namespace (network_mode: "service:engine") so the loopback-only tap posture (resolve_tap_url rejects non-loopback until step 6) survives unchanged. The brain graduates to its own netns when the wss:// tap lands (step 6).
• Valkey is its own service on the compose network — network-near, satisfying ADR-0005's sub-ms rule.
• stop_grace_period: 660s paired with RUTSTER_DRAIN_DEADLINE_SECS=600 (grace > drain).
• BYO-proxy supported: disable the caddy service; the engine keeps its plaintext :8080 mode; tuned-timeout snippets for nginx/HAProxy/Traefik ship in docs/deploy/reverse-proxies.md.

T3 — Fleet — ADR chapter only (paper; no code in this epoch)

N symmetric nodes, each running the T1 process set minus valkey, plus a shared green zone: one Valkey (presence + directory + the ADR-0009 spend ledger) and object storage (named as required, still deferred). Wildcard DNS *.pbx.domain; per-node certs (see the fleet cert story below). The per-node admission cap is the ADR-0010 benchmark number (replacing the RUTSTER_MAX_SESSIONS=64 placeholder). Scale-in is drain-then-terminate only; spot instances are rejected for the engine tier (2026-07-04 review — a reclaimed node kills every non-migratable call on it).

Edge doctrine

Bundled edge: Caddy (both shipped artifacts, same build, same Caddyfile)

Custom xcaddy build (the rutster-edge image): Caddy core (Apache-2.0) + a curated DNS-01 plugin set — cloudflare, route53, porkbun, hetzner, desec (duckdns excluded: no license file). Zero-downtime in-memory cert renewal; verified protocol-unaware WS tunneling (no frame buffering). Config is a ~6-line Caddyfile with tuned timeouts, honest X-Forwarded-*, and stream_close_delay above max call duration — plus a CI e2e case for config-reload-during-live-call, because the upstream mitigation has an open bug trail (caddy #6420/#7222) and is not trusted untested.

In-process TLS: staged — the ratified end-state, not adopted wholesale now

ADR-0008 claims the trust-domain edge as security-constitutive FOB territory, and str0m already terminates DTLS-SRTP in-process on the same aws-lc-rs provider — "keep TLS out of the FOB" is not the objection. The objection is economics: owning an ACME renewal state machine forever, in a 45-day-cert world.

• Phase 1 (this epoch): axum-server + rustls with operator-supplied certs (RUTSTER_TLS_CERT/RUTSTER_TLS_KEY), hot-reload without dropping live WS. Plaintext-behind-Caddy stays the artifact default.
• Phase 2 (in-binary ACME) triggers, named: (a) Let's Encrypt dns-persist-01 confirmed GA (closes the wildcard/CGNAT issuance gap; instant-acme already parses it); (b) a field incident where Caddy reload/stream_close_delay drops live calls; (c) fleet or VPS-forwarder topology making encrypted-to-the-binary a selling point; (d) maintenance fatigue with the curated xcaddy plugin build. Until a trigger fires, Phase 2 is deliberately not built — no turnkey ACME crate fits the tree today (rustls-acme conflicts with the deny.toml rcgen/x509-parser posture).

Fleet cert story

Per-node public TLS names are CPaaS-imposed (invariant 4 + node-addressed placement). Two blessed patterns:

1. Wildcard *.pbx.domain via DNS-01 with central issuance/distribution.
2. Per-node distinct certs (node-N.domain) where the wildcard private key must not live on every node — one compromised node otherwise burns the whole namespace.

Traps, stated so nobody re-discovers them in production: N nodes independently requesting the identical wildcard hit the Let's Encrypt duplicate-cert limit (5/week) — distinct names or central distribution only. Caddy on-demand TLS is rejected for node names: the first handshake to a fresh name blocks seconds on issuance, colliding with the sub-second webhook budget (invariant 6), and its rate-limit knobs are deprecated. Node names known at provision time get pre-issued certs.

Fleet placement paradigm (the trunk webhook handler and session API grow toward this; no fleet code in this epoch)

Node-addressed URLs + Valkey presence; no router tier. (Decision 2026-07-05 over the two rejected alternatives below.)

• Nodes heartbeat capacity into Valkey (TTL'd presence keys — the ADR-0005 presence layer).
• Placement happens at answer time by whichever node the request lands on (api.domain, DNS round-robin): read fleet capacity → pick owner → return TwiML whose Stream URL is wss://node-K.pbx.domain/.... The media WS dials the owning node directly — zero extra hops, consistent with connection-follows-ownership. Twilio's answer-time Stream URL is purpose-shaped for this: invariant 4 permits hostname routing.
• WebRTC: the create-session response carries the owner's node-addressed base URL + advertised media IP (UDP is already direct).
• Misdirected control ops (e.g. a DELETE hitting a non-owner) consult the Valkey directory and 307-redirect to the owner.
• Degraded mode (Valkey dark): the answering node self-places if it has capacity; presence heals via TTL on recovery. Fail-safe degradation, consistent with ADR-0009's posture.
• Single-node is the degenerate case of the same code: a one-entry directory.

Rejected: a routing tier that proxies everything. It would put every 20 ms media frame through an extra userspace hop and re-create exactly the SPOF the symmetric design avoids — and it buys nothing, because the CPaaS already lets the answer-time URL do the routing.

Rejected: a distinguished control node. It breaks node symmetry (two artifacts, two failure stories, a promotion protocol) to solve a problem the presence directory already solves; placement needs fleet state either way, and Valkey is already the fleet-state home (ADR-0005).

Rejected shapes

• Decomposing the FOB into services — rejected permanently, not deferred. Media/reflex/tap/spend stay one process, one trust domain (ADR-0002).
• Serverless. Lambdas are categorically unfit for the engine: non-migratable multi-hour calls, per-session UDP sockets, in-process media state. Webhook-on-Lambda buys nothing because placement needs fleet state anyway.
• Tunnels in production. Cloudflare Tunnel and ngrok see plaintext audio at the vendor edge (an unconsented subprocessor — DPA/BAA/PCI failure), document mid-call WS kills, and cloudflared has an open Twilio handshake bug (cloudflared #1465). ngrok is the blessed 5-minute demo path (the only tunnel with a proven Media Streams record); Tailscale Funnel is the privacy-clean single-user demo (TLS terminates on-node), unsizable beyond that. No tunnel carries inbound UDP → homelab behind CGNAT is PSTN-only, period. The production graduation path is a cheap VPS + WireGuard as a dumb TCP forwarder with TLS terminating at home, or running the engine on the VPS — see docs/deploy/homelab.md.
• NLB TLS listeners — never. Fixed 350 s idle timeout that silently drops flows, and no HTTP context, so X-Forwarded-* headers for signature validation (invariant 5) don't exist. The ALB path is documented as a docs-only appendix (docs/deploy/aws.md), never a shipped artifact.
• k8s manifests/Helm. Compose-first (README pillar 6). k8s notes live in docs only (drain vs terminationGracePeriodSeconds); manifests are a later rung if demand appears.

Deferred, named (so the boundary is deliberate)

• T3 fleet implementation (presence heartbeats, directory redirects, placement code).
• The zero-egress / air-gap profile (SBC layer-2 ingress per ADR-0007 layer 2 + a self-hosted brain): a defined profile, no code this epoch. T1 is about one-command simplicity, not isolation.
• In-binary ACME (rustls Phase 2) — behind the four named triggers above.
• Durable CDR pipeline: ValkeyEventSink is evidence preservation on a capped stream — explicitly NOT the billing ledger (ADR-0005 source-of-truth rule). CDR → object storage stays deferred.
• Multi-tenancy (needs its own ADR before schemas ossify); wss:// tap + brain fleet + resume tokens (step 6); media-thread sharding graduation; arm64 images.

Consequences

• Positive: one binary and one mental model across all three shapes — an operator's knowledge transfers from docker run to a fleet; the CPaaS invariants are satisfied by a decade-hardened ACME loop (Caddy) rather than a DIY renewal state machine; the fleet paradigm needs zero new infrastructure beyond what ADR-0005 already bundles; the rejected shapes are written down, so nobody burns a week discovering that Lambda or NLB-TLS can't work.
• Negative: the curated xcaddy DNS-plugin set is a permanent maintenance surface until dns-persist-01 GA; Caddy config-reload behavior vs live calls is mitigated (stream_close_delay + CI e2e) but has an open upstream bug trail; homelab WebRTC behind CGNAT is unsolved by design — only the VPS graduation fixes it, and the docs must say so loudly or operators will read "homelab supported" as including browser calls.
• Mitigation: the Phase-2 triggers convert both negatives into a planned exit; the docs/deploy/ tree ships with a copy-paste-to-first-call acceptance bar so the honest limits are in the operator's face, not in a footnote.

References

• 2026-07-05 deployment-topology spec — §2–§4 are this ADR's source
• TLS/edge decision brief — six-family survey + the CPaaS ground truth (external citations live there)
• ADR-0002 · ADR-0005 · ADR-0006 · ADR-0007 · ADR-0008 · ADR-0009 · ADR-0010
• The docs/deploy/ tree: topologies · quickstart-docker · homelab · aws · reverse-proxies · certificates