diff --git a/docs/superpowers/specs/2026-07-05-deployment-topology-design.md b/docs/superpowers/specs/2026-07-05-deployment-topology-design.md new file mode 100644 index 0000000..35d72c1 --- /dev/null +++ b/docs/superpowers/specs/2026-07-05-deployment-topology-design.md @@ -0,0 +1,391 @@ +# Rutster deployment topology — one binary, three blessed shapes + +- **Status:** Draft (pending review) +- **Date:** 2026-07-05 +- **Origin:** [ADR-0008](../../adr/0008-fob-and-green-zone.md) explicitly reserved a deployment + topology ADR "once that design closes." This spec closes it. Brainstormed 2026-07-05; the + TLS/edge fork was researched via a six-family survey — see the companion + [TLS/edge decision brief](2026-07-05-tls-edge-decision-brief.md). +- **Produces:** **ADR-0011 (deployment topology)** — drafted from §2–§4 of this spec, Proposed, + ratified per repo process — plus the artifacts and code slices in §5–§6. +- **Depends on (already merged):** slice-5/seams (drain lifecycle, `/readyz`, admission cap, + `EventSink`, `MediaAddressConfig`, `config.rs` env-parser pattern); slice-4½ benchmark/sim + (S1–S8, CI-regressed — the future source of the real per-node admission cap). +- **Related:** [ADR-0002](../../adr/0002-north-star-and-fused-core.md) (fused per-call vertical), + [ADR-0005](../../adr/0005-event-bus.md) (Valkey = bus + KV + presence; media never rides the + bus; bus is never billing source-of-truth), [ADR-0006](../../adr/0006-ingress-posture.md) + (two inbound protocols only; tap is egress-only), [ADR-0007](../../adr/0007-trunk-rented-transport.md) + (rented transport ⇒ public HTTPS/WSS reachability), [ADR-0009](../../adr/0009-spend-gate-honest-rescope.md) + (ledger trait: in-memory single-node, Valkey-backed fleets), [ADR-0010](../../adr/0010-spearhead-benchmark-sim-harness.md) + (sequencing this spec must not reorder), README pillar 6 ("one binary, one bus, one deploy"). + +--- + +## TL;DR + +**The FOB never decomposes; topology is a configuration property, not a code property.** The same +binary runs in every blessed shape; only `RUTSTER_*` env and what is deployed alongside it differ. +"Microservices" in rutster means *green-zone services get their own processes/containers* — never +that the per-call vertical splits (ADR-0002). + +Three blessed topologies: + +| | Shape | Status after this work | +|---|---|---| +| **T1 — Solo** | One all-in-one container: s6 supervises Caddy + FOB + brain + valkey-server (EventSink consumer day one; ledger/directory later). `docker run` with a domain + Twilio creds → taking calls. | Shipped artifact | +| **T2 — Modular** | Compose stack: `caddy`, `engine`, `brain` (shared netns with engine), `valkey` as separate services. `compose up` is the pillar. | Shipped artifact (reference deployment) | +| **T3 — Fleet** | N symmetric nodes + shared green zone (one Valkey, object storage). Node-addressed URLs, answer-time placement via Valkey presence. | ADR chapter only (paper) | + +Riders shipping with the artifacts: TCP_NODELAY + app-level WS-ping hygiene, trunk config fixes, +in-process TLS **Phase 1** (BYO-cert, hot-reload), a hand-rolled Prometheus `/metrics` endpoint, +`ValkeyEventSink` (first Valkey consumer), image-publish CI, and a `docs/deploy/` documentation +tree with a copy-paste-to-first-call acceptance bar. + +**Six things this design deliberately is NOT:** + +- **Not a decomposition of the FOB into services.** Media/reflex/tap/spend stay one process, + one trust domain, per ADR-0002. Rejected permanently, not deferred. +- **Not serverless.** Lambdas are categorically unfit for the engine: non-migratable multi-hour + calls, per-session UDP sockets, in-process media state. Webhook-on-Lambda buys nothing because + placement needs fleet state anyway. Stated as a rejected shape in ADR-0011. +- **Not k8s manifests/Helm.** Compose-first (pillar 6). k8s notes live in docs only (drain vs + `terminationGracePeriodSeconds`); manifests are a later rung if demand appears. +- **Not the durable CDR pipeline.** `ValkeyEventSink` is evidence preservation on a capped + stream — explicitly NOT the billing ledger. CDR → object storage stays deferred + (ADR-0005 source-of-truth rule). +- **Not the zero-egress / air-gap profile.** T1 is about one-command simplicity, not isolation. + Zero-egress (SBC layer-2 ingress + self-hosted brain) is named as a defined-but-deferred + profile in ADR-0011, one footnote section, no code. +- **Not in-binary ACME (Phase 2).** Deferred behind named triggers (§3.4). No turnkey crate fits + the tree today (rustls-acme conflicts with the deny.toml rcgen/x509-parser posture); owning a + renewal state machine in a 45-day-cert world is a deliberate, triggered decision. + +--- + +## 1. Scope + +### 1.1 In scope + +- **ADR-0011** (deployment topology): the three shapes, the edge doctrine, the fleet placement + paradigm, the rejected shapes. +- **Artifacts** under `deploy/`: multi-stage `Dockerfile` (four first-party images), `compose.yaml`, + `Caddyfile`, `.env.example`. +- **Code:** TCP_NODELAY, app-level WS pings, trunk config fixes (webhook base URL, credential + wiring, trusted-proxy header posture), rustls Phase 1 (BYO-cert TLS), `/metrics`, + `ValkeyEventSink`. +- **CI:** containerized smoke suite (incl. Caddy-reload-during-live-call), image publish on tag. +- **Docs:** `docs/deploy/` tree (§10). + +### 1.2 Out of scope (named in ADR-0011 as deferred, so the boundary is deliberate) + +- T3 fleet **implementation** (presence heartbeats, directory redirects, placement code). +- Object-storage CDR durability; multi-tenancy (needs its own ADR before schemas ossify); + wss:// tap + brain fleet + resume tokens (step 6); media-thread sharding graduation; + in-binary ACME (Phase 2); SBC layer-2 wire contract; arm64 images. + +--- + +## 2. Topologies + +### 2.1 T1 — Solo (all-in-one container) + +One image (`rutster-allinone`), tini/s6-overlay as PID 1 supervising four processes: + +| Process | Role | Binding | +|---|---|---| +| `caddy` | Edge: ACME, TLS termination, WS proxy to FOB | :443/:80 public | +| `rutster` (FOB) | Engine | 127.0.0.1:8080 (behind Caddy); media UDP direct | +| `rutster-brain-realtime` | Brain | 127.0.0.1:8082 loopback tap (as today) | +| `valkey-server` | **Dark on day one** except EventSink (§5.6) | 127.0.0.1:6379 | + +Volumes (both non-negotiable, documented loudly): +- `/data` — Caddy cert/ACME state. Loss risks Let's Encrypt duplicate-cert lockout (5/week) + → total inbound outage class. +- `/var/lib/valkey` — stream/state persistence. + +WebRTC media UDP goes **direct to the FOB**, never through Caddy: host networking recommended; +otherwise published `RUTSTER_MEDIA_PORT_RANGE` + `RUTSTER_MEDIA_ADVERTISED_IP` (NAT 1:1 model, +no STUN — unchanged from slice-5/seams). + +Valkey ships bundled from v1 **before most consumers exist** (decision 2026-07-05): the operator +contract (volumes, ports, upgrade shape) is stable from day one, so the spend ledger and fleet +directory land later without changing the deployment shape. Cost accepted: ~15 MB and one +mostly-idle process. + +### 2.2 T2 — Modular (compose stack) — the reference deployment + +Services: `caddy` (image `rutster-edge`), `engine` (`rutster-engine`), `brain` (`rutster-brain`), +`valkey` (upstream `valkey/valkey`). + +- **Brain shares the engine's network namespace** (`network_mode: "service:engine"`) so the + loopback-only tap posture (`resolve_tap_url` rejects non-loopback until step 6) survives + unchanged. ADR-0011 notes the brain graduates to its own netns when the wss:// tap lands. +- Valkey is its own service on the compose network — network-near, satisfies ADR-0005's sub-ms + rule. +- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` (§8). +- BYO-proxy supported: disable the `caddy` service; the engine keeps its plaintext `:8080` mode; + tuned-timeout snippets for nginx/HAProxy/Traefik ship in docs (§10). + +### 2.3 T3 — Fleet (ADR chapter only; no code in this epoch) + +N **symmetric** nodes, each running the T1 process set *minus* valkey, plus a shared green zone: +one Valkey (presence + directory + ADR-0009 spend ledger), object storage (named as required, +still deferred). Wildcard DNS `*.pbx.domain`; per-node certs (§3.3). Per-node admission cap = +the ADR-0010 benchmark number (replacing the `RUTSTER_MAX_SESSIONS=64` placeholder). Scale-in is +drain-then-terminate only; spot instances rejected for the engine tier (2026-07-04 review). +Placement paradigm: §4. + +--- + +## 3. Edge doctrine + +### 3.1 CPaaS invariants (the spec any edge must satisfy — research §2) + +1. Publicly-CA-trusted cert; **no self-signed path exists** (Twilio error 31910). Auto-renewal is + availability-critical. +2. wss:// on 443, open to the internet: Twilio publishes no egress IPs, offers no mTLS. Auth is + application-layer only (HMAC signatures). +3. One WS per call, up to 24 h, 50 msg/s/direction at 20 ms cadence; zero tolerance for frame + buffering, lifetime caps, or write-side idle timers. Neither Twilio nor Telnyx documents WS + keepalive — **keepalive is entirely our job** (§5.2). +4. `` URLs carry **no query strings** — wss routing keys on hostname/path only + (constrains §4). +5. `X-Twilio-Signature` is HMAC over the URL *as Twilio saw it*: the edge must forward + `X-Forwarded-Proto/Host` honestly and the FOB must reconstruct the public URL (§5.3). +6. Webhook budget: sub-second for UX, 15 s hard cap — no blocking issuance in the handshake path. +7. TLS 1.2+1.3, mainstream ECDHE, never 1.3-only; never pin Twilio certs. + +Design for Twilio; Telnyx/Vonage come for free (they are strictly less demanding, plus optional +source-IP allowlisting as extra hardening Twilio can't use). + +### 3.2 Bundled edge: Caddy (both shipped artifacts, same build, same Caddyfile) + +Custom xcaddy build (`rutster-edge` image): Caddy core (Apache-2.0) + curated DNS-01 plugin set — +cloudflare, route53, porkbun, hetzner, desec (duckdns excluded: no license file). Zero-downtime +in-memory cert renewal; verified protocol-unaware WS tunneling (no frame buffering). Config: +~6-line Caddyfile with tuned timeouts, honest `X-Forwarded-*`, `stream_close_delay` above max +call duration — **and a CI e2e case for config-reload-during-live-call, because the upstream +mitigation has an open bug trail (caddy #6420/#7222) and is not trusted untested.** + +### 3.3 Fleet cert story (ADR chapter) + +Per-node public TLS names are CPaaS-imposed (invariant 4 + node-addressed placement). Two blessed +patterns: wildcard `*.pbx.domain` via DNS-01 with central issuance/distribution, or per-node +distinct certs (`node-N.domain`) where the wildcard private key must not live on every node. +Traps stated in the ADR: N nodes independently requesting the identical wildcard hit the LE +duplicate-cert limit; **Caddy on-demand TLS is rejected for node names** (blocking first +handshake vs the webhook budget). + +### 3.4 In-process TLS: staged (the ratified end-state, not adopted wholesale now) + +ADR-0008 claims the trust-domain edge as security-constitutive FOB territory, and str0m already +terminates DTLS-SRTP in-process on the same aws-lc-rs provider — "keep TLS out of the FOB" is not +the objection. The objection is economics: owning an ACME renewal state machine forever. + +- **Phase 1 (this epoch, §5.4):** `axum-server` + rustls with operator-supplied certs, hot-reload + without dropping live WS. Plaintext-behind-Caddy stays the default. +- **Phase 2 (in-binary ACME) triggers, named in ADR-0011:** (a) Let's Encrypt dns-persist-01 + confirmed GA (closes the wildcard/CGNAT issuance gap; instant-acme already parses it); + (b) a field incident where Caddy reload/`stream_close_delay` drops live calls; (c) fleet or + VPS-forwarder topology making encrypted-to-the-binary a selling point; (d) maintenance fatigue + with the curated xcaddy plugin build. + +### 3.5 Tunnels and cloud LBs + +- **Tunnels are dev/demo-only, stated bluntly.** Cloudflare Tunnel and ngrok see plaintext audio + at the vendor edge (unconsented subprocessor — DPA/BAA/PCI failure), document mid-call WS kills, + and cloudflared has an open Twilio handshake bug (#1465). ngrok is the blessed 5-minute demo + path (only tunnel with a proven Media Streams record). Tailscale Funnel: privacy-clean + single-user demo (TLS terminates on-node), unsizable beyond that. +- **No tunnel carries inbound UDP → homelab behind CGNAT is PSTN-only, period.** Production + graduation path: cheap VPS + WireGuard as a dumb TCP forwarder with TLS terminating at home + (forwarder physically cannot read audio), or run the engine on the VPS. +- **AWS appendix (docs only, never a shipped artifact):** ALB + wildcard ACM + host-header rules; + mandatory attribute overrides `idle_timeout=4000`, `client_keep_alive=604800`, header + forwarding. **Never NLB-TLS** (fixed 350 s silent drop, no HTTP context for signatures). + +--- + +## 4. Fleet placement paradigm (ADR chapter; the trunk webhook handler and session API grow +*toward* this, no fleet code in this epoch) + +**Node-addressed URLs + Valkey presence; no router tier.** (Decision 2026-07-05 over +routing-tier-proxies-everything and distinguished-control-node.) + +- Nodes heartbeat capacity into Valkey (TTL'd presence keys — the ADR-0005 presence layer). +- Placement happens at answer time by whichever node the request lands on (`api.domain`, + DNS round-robin): read fleet capacity → pick owner → return TwiML whose Stream URL is + `wss://node-K.pbx.domain/...`. The media WS dials the owning node directly — zero extra hops, + consistent with connection-follows-ownership. +- WebRTC: create-session response carries the owner's node-addressed base URL + advertised media + IP (UDP is already direct). +- Misdirected control ops (e.g. DELETE hitting a non-owner) consult the Valkey directory and + **307-redirect** to the owner. +- Degraded mode (Valkey dark): the answering node self-places if it has capacity; presence heals + via TTL on recovery. Fail-safe degradation, consistent with ADR-0009's posture. +- Single-node is the degenerate case of the same code: a one-entry directory. + +Rationale: Twilio's answer-time Stream URL is purpose-shaped for this (invariant 4 permits +hostname routing); a proxying router tier would put every 20 ms frame through an extra userspace +hop and re-create the SPOF the symmetric design avoids. + +--- + +## 5. Code changes (workspace) + +### 5.1 TCP_NODELAY (hygiene, unconditional) + +Set on every accepted socket (axum #2521 — Nagle can coalesce 20 ms WS frames on the plaintext +listener behind *any* proxy). Latency assertion joins the CI-regressed sim-bench. + +### 5.2 App-level WS pings (hygiene, unconditional) + +Engine-originated ping/pong on the trunk media-stream WS (and any future long-lived WS surface; +WebRTC signaling is plain HTTP today, so the media stream is the only such socket), interval +configurable (`RUTSTER_WS_PING_SECS`, default 20). Belt-and-braces against every idle timer +between the FOB and the CPaaS (invariant 3). + +### 5.3 Trunk config fixes + trusted-proxy posture + +- TwiML Stream URL derived from `RUTSTER_TWILIO_WEBHOOK_BASE` (kills the hardcoded loopback at + `routes.rs:258`). +- `config::twilio_credentials` wired into startup validation (parsed-but-never-called today). +- Trusted-proxy: `X-Forwarded-Proto/Host` honored **only** from the configured edge + (`RUTSTER_TRUSTED_PROXIES`, CIDR list; empty = headers ignored); signature validation + reconstructs the public URL from them (invariant 5). + +### 5.4 rustls Phase 1 — BYO-cert in-process TLS + +`axum-server` + rustls (already in-tree transitively): `RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY` +enable a TLS listener; hot-reload on file change without dropping live WS (arc-swap resolver). +Always compiled, runtime-gated; plaintext `:8080` behind Caddy remains the artifact default. + +### 5.5 `/metrics` — hand-rolled Prometheus text endpoint + +Reads the *existing* atomics: active sessions, `tick_overruns`, `last_tick_micros`, admission +rejects, EventSink drops (§5.6). Zero new deps; node-level cardinality only. Own internal bind +`RUTSTER_METRICS_BIND` (default `127.0.0.1:9090`) — never routed through Caddy. + +### 5.6 `ValkeyEventSink` — first Valkey consumer + +Implements the existing `EventSink` trait: `XADD` of current lifecycle events as JSON to a capped +stream (`XADD MAXLEN ~`), strictly fire-and-forget — Valkey down ⇒ count-and-drop (counter +surfaces in `/metrics`), never blocks the call path. `RUTSTER_VALKEY_URL` unset ⇒ today's +`TracingEventSink`. Client crate: redis-rs (BSD-3; verify against deny.toml in the slice). +Closes the "OOM-killed node erases all evidence of its calls" gap. **Evidence preservation, not +the billing ledger** (ADR-0005 rule restated in code docs). + +### 5.7 Env additions (all follow the `config.rs` fail-fast parser pattern) + +| Var | Default | Used by | +|---|---|---| +| `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY` | unset (plaintext) | §5.4 | +| `RUTSTER_TRUSTED_PROXIES` | empty (ignore forwarded headers) | §5.3 | +| `RUTSTER_WS_PING_SECS` | 20 | §5.2 | +| `RUTSTER_METRICS_BIND` | 127.0.0.1:9090 | §5.5 | +| `RUTSTER_VALKEY_URL` | unset (tracing sink) | §5.6 | +| `RUTSTER_DRAIN_DEADLINE_SECS` | unchanged code default 0; **600 in `.env.example`** | §8 | + +--- + +## 6. Artifacts + +### 6.1 Images (four first-party, published to the git.adlee.work registry) + +| Image | Contents | Base | +|---|---|---| +| `rutster-allinone` | s6-overlay: Caddy + FOB + brain + valkey-server | debian-slim + `libopus0` | +| `rutster-engine` | FOB only | debian-slim + `libopus0` | +| `rutster-brain` | brain only | debian-slim | +| `rutster-edge` | custom xcaddy build (§3.2) | scratch/alpine | + +Multi-stage `deploy/Dockerfile`, pinned 1.85 toolchain per `rust-toolchain.toml`. Distro +`libopus0` over static-bundled (matches dev; static noted as future size work). linux/amd64 +first; arm64 deferred. Bundled-binary licenses are aggregation-clean vs GPL-3 (Caddy Apache-2.0, +valkey BSD-3, s6 ISC) — noted for the cargo-deny-adjacent image license scan in CI. + +### 6.2 `deploy/` tree + +`Dockerfile`, `compose.yaml`, `Caddyfile`, `.env.example` (documented defaults incl. drain 600 s), +plus the smoke-test harness consumed by CI (§9). + +### 6.3 Publish CI + +Gitea Actions job on tag push: build all four images, push to the registry. Runs after (and +gated on) the existing fmt/clippy/test/deny/sim-bench pipeline. + +--- + +## 7. Data flow (T1/T2) + +**PSTN:** Twilio → HTTPS webhook → Caddy → FOB `/v1/trunk/webhook` (signature validated against +URL reconstructed from trusted `X-Forwarded-*`) → TwiML with derived wss URL → Twilio dials +media stream → Caddy (unbuffered WS tunnel) → FOB WS upgrade → `MediaCmd::RegisterTrunk` → +media-thread tick (G.711 ↔ PCM) ↔ brain tap (loopback). **WebRTC:** signaling via Caddy; SDP +answer advertises media IP; DTLS-SRTP UDP direct to FOB. **Events:** EventSink → Valkey stream +(loopback T1 / compose network T2). **Metrics:** Prometheus scrapes the internal bind. +**T3 flow** as §4. + +--- + +## 8. Failure modes (worst first) + +| Failure | Handling | +|---|---| +| Caddy `/data` loss | Persistent volume mandatory; LE duplicate-cert lockout documented as total-inbound-outage class | +| Caddy reload vs live calls | `stream_close_delay` > max call duration **+ CI reload-during-call e2e** (§9); documented reload procedure | +| Valkey down | EventSink counts-and-drops; calls unaffected. T3: degraded self-placement (§4) | +| Node/FOB death | Calls on it die (non-migratable — stated honestly); s6 restarts in T1; Valkey stream preserves evidence; T3 presence TTL stops new placements | +| Brain death | FOB reflexes continue (the FOB point); amnesiac reconnect documented; resume tokens are step 6 | +| Drain vs orchestrator | `.env.example` drain 600 s; compose `stop_grace_period` 660 s (grace > drain); systemd `TimeoutStopSec`/k8s equivalents in docs | +| Quiet WS killed by any middle timer | §5.2 pings + tuned edge timeouts | + +--- + +## 9. Testing + +- **Code:** NODELAY latency assertion in sim-bench; webhook URL derivation; signature validation + behind forwarded headers (trusted and untrusted source cases); WS ping emission; TLS hot-reload + (self-signed test certs); `ValkeyEventSink` XADD + drop-on-down (real valkey service container + in CI). +- **Artifacts (new CI muscle):** build all images; **all-in-one smoke** — boots, Caddy terminates + TLS via its internal CA (no ACME in CI), full sim call through the real edge→FOB WS path; + **compose smoke** — all services healthy, a lifecycle event lands in the Valkey stream; + **Caddy config reload during a live simulated call, asserting zero drops.** +- Existing seam-gates (loop_driver/rtc_session blob hashes), MSRV matrix, cargo-deny (now + covering redis-rs), sim-bench single-threaded — all unchanged and must stay green. + +## 10. Documentation (`docs/deploy/`) + +`topologies.md` (three shapes + decision guide), `quickstart-docker.md`, `homelab.md` (CGNAT +truth: PSTN-only behind tunnels; ngrok dev path; VPS+WireGuard graduation), `aws.md` (ALB +appendix, three mandatory overrides, never-NLB-TLS), `reverse-proxies.md` (tuned-timeout snippets +for BYO nginx/HAProxy/Traefik incl. the universal 60 s idle-timeout footgun), `certificates.md` +(ACME paths, wildcard traps, duplicate-cert lockout). + +**Acceptance bar: every topology doc contains a copy-paste path from zero to first call** — or +the honest statement of what cannot work there (e.g. CGNAT WebRTC). + +## 11. Sequencing — slices + +| Slice | Contents | Depends | +|---|---|---| +| **A** | Hygiene: §5.1 NODELAY + §5.2 pings + §5.3 trunk fixes/trusted-proxy | — | +| **B** | Images + compose + Caddyfile + smoke CI (§6, §9) | A | +| **C** | rustls Phase 1 (§5.4) | A | +| **D** | `/metrics` (§5.5) | — | +| **E** | `ValkeyEventSink` (§5.6) | — | +| **F** | Publish CI (§6.3) | B | +| **G** | `docs/deploy/` tree (§10) + ADR-0011 drafting/ratification | B for accuracy | + +C–F order-independent after their deps. Additive to the ADR-0010 sequence (benchmark → +escalation → trunk → spend) — not a reordering. The artifacts ship with WebRTC calling live now; +PSTN lights up end-to-end the moment the remaining trunk work lands (trunk registration is a +placeholder NACK today; the container wiring will be ready). + +## 12. Open questions deliberately left open (tracked, not blocking) + +dns-persist-01 GA timeline (gates rustls Phase 2 *and* dissolves the xcaddy plugin burden); +object-storage CDR shape; multi-tenancy ADR (before CDR/config schemas ossify); brain fleet + +tap auth/resume (step 6); media-thread sharding graduation; SBC layer-2 wire contract; +the real admission-cap number from the ADR-0010 benchmark. diff --git a/docs/superpowers/specs/2026-07-05-tls-edge-decision-brief.md b/docs/superpowers/specs/2026-07-05-tls-edge-decision-brief.md new file mode 100644 index 0000000..3f39841 --- /dev/null +++ b/docs/superpowers/specs/2026-07-05-tls-edge-decision-brief.md @@ -0,0 +1,115 @@ +# Rutster TLS/Edge Decision Brief + +Synthesized from six research reports (Caddy, classic proxies, in-process rustls, tunnels, cloud-native LB, CPaaS ground truth). 2026-07-05. + +--- + +## 1. Decision Matrix + +Scores 1–5. The ground-truth report is excluded from the matrix — it is the spec, not a candidate (its scores measure how *demanding* the CPaaS side is, per its own scoring note). + +| Criterion | Caddy | nginx | HAProxy | Traefik | In-process rustls | Tunnels (CF/Funnel/ngrok) | AWS ALB+ACM | +|---|---|---|---|---|---|---|---| +| WS media fitness | 4 | 5* | 5* | 5* | 5 | 2 | 3.5 | +| Cert automation | 5 | 3 | 4 | 5 | 3 | 5 | 4 | +| All-in-one fit | 5 | 4* | 4* | 4* | 5 | 3 | 1 | +| Compose fit | 5 | 5* | 5* | 5* | 4 | 4 | 2 | +| Fleet fit | 4 | 4* | 4* | 4* | 4 | 3 | 4 | +| No-public-IP story | 3 | 2 | 2 | 2 | 3 | 5 | 1 | +| Ops burden | 4 | 4* | 4* | 4* | 3 | 3 | 4 | +| **Third party in call path** | **No** | **No** | **No** | **No** | **No** | **Yes** (CF/ngrok see *plaintext* audio; Funnel ciphertext-only) | **Yes** (trust-neutral when engine is already on EC2) | + +\* The classic-proxy report scored the family jointly (5/5/4/5/4/2/4); per-member sub-rankings from its findings: cert automation Traefik 5, HAProxy 4, nginx 3; HAProxy best WS mental model; Traefik has a patch-release WS-regression record. + +--- + +## 2. Hard CPaaS Invariants (any answer must satisfy — ground-truth report) + +1. **Publicly-trusted CA cert, no self-signed path exists.** Twilio refuses self-signed for webhooks and Media Streams (error 31910); the console validation toggle is account-wide, webhook-only, dev-only. Auto-renewing Let's Encrypt-class certs are availability-critical, not a nicety. ([twilio.com/docs/usage/security](https://www.twilio.com/docs/usage/security), [errors/31910](https://www.twilio.com/docs/api/errors/31910)) +2. **wss:// only, on 443, open to the whole internet.** Twilio publishes no egress IPs ("any public IP address") and no CPaaS surveyed offers mTLS. Auth is application-layer only. IP allowlisting exists for Telnyx/Vonage/Plivo as an optional hardening layer — never for Twilio. +3. **One WS connection per call, up to 4h default / 24h configured, 50 msg/s/direction at 20ms cadence.** Zero tolerance for frame buffering, connection-lifetime caps, or write-side idle timers — the unidirectional case may legitimately send zero bytes from your side for hours, and neither Twilio nor Telnyx documents any WS ping behavior. Keepalive is entirely the edge's job. +4. **`` URLs carry no query strings** — wss routing must key on path/hostname. +5. **X-Twilio-Signature is HMAC over the URL as Twilio saw it.** Any TLS-terminating edge must forward X-Forwarded-Proto/Host untouched (including on the WS upgrade request) and the app must reconstruct the public URL, or genuine traffic fails validation. ([Twilio SSL-termination blog](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library)) +6. **Webhook budget: sub-second for UX, 15s hard platform cap.** No blocking operations (e.g. on-demand ACME issuance) may sit in the handshake/webhook path. +7. **TLS 1.2+1.3, mainstream ECDHE suites, never 1.3-only** (Twilio's 1.3 client support is undocumented); never pin Twilio certs. +8. **Fleet corollary:** node-addressed URLs force a publicly-valid cert+DNS name *per node* — wildcard DNS-01 or per-node ACME is imposed by the CPaaS, not the proxy choice. +9. **CGNAT corollary:** the CPaaS offers no relay/rendezvous; public reachability must be imported from the operator's side, and intermediaries are a documented frame-dropping hazard ([livekit/agents#3379](https://github.com/livekit/agents/issues/3379)). + +--- + +## 3. Recommendation (per topology) + +**Headline: Caddy is the shared edge for both shipped artifacts. In-process rustls is the ratified end-state, deferred — ship Phase 1 (BYO-cert) opportunistically, gate Phase 2 (in-binary ACME) on explicit triggers. Tunnels are dev-only.** + +### (a) All-in-one container: bundle Caddy under s6 + +A single 47MB static Apache-2.0 binary, ~6-line Caddyfile, dual-CA ACME with background issuance and **zero-downtime in-memory cert renewal that never touches live calls** — the routine periodic event drops nothing (Caddy report). Post-upgrade WS handling is verified protocol-unaware `io.CopyBuffer` tunneling — safe for 20ms PCM. Requirements baked into the image: + +- Custom xcaddy build with a curated DNS-provider plugin set (cloudflare Apache-2.0; route53/porkbun/hetzner/desec MIT; **skip duckdns** — no license file). The official image ships zero DNS plugins; without this, wildcard/behind-NAT issuance is impossible. +- `/data` on a persistent volume, non-negotiable — recreate loops otherwise hit LE's 5-certs/week duplicate limit and lock the domain out. +- `stream_close_delay` set above max call duration, **and** a CI e2e case "config reload during active simulated call" — the mitigation has an open bug trail (caddy [#6420](https://github.com/caddyserver/caddy/issues/6420), [#7222](https://github.com/caddyserver/caddy/issues/7222)) and must not be trusted untested. +- Tuned timeouts + honest X-Forwarded-* forwarding per invariants 3 and 5. + +### (b) Modular compose: same Caddy build, as a compose service + +Same image, same Caddyfile, same CI coverage — one edge choice across both artifacts, as preferred. Compose additionally documents "bring your own proxy": the engine keeps its plaintext `:8080` mode, and the docs ship the tuned-timeout snippets for nginx/HAProxy/Traefik (the 60s idle-timeout defaults in all three are the universal footgun — classic-proxy report). + +### (c) Fleet chapter of the ADR + +- **Per-node public TLS names are CPaaS-imposed** (invariant 8). Two blessed patterns: **(1) wildcard `*.pbx.domain` via DNS-01** + Caddy's documented shared-storage cluster coordination, or issue-once-and-distribute; **(2) per-node distinct certs** (`node-N.domain`) where you don't want the wildcard private key on every node — one compromised node otherwise burns the whole namespace (rustls report). Renewals are exempt from LE per-domain limits, so per-node scales. +- **Explicitly reject Caddy on-demand TLS for node names:** the first handshake to a fresh name blocks seconds on issuance, colliding with sub-second webhook budgets; the rate-limit knobs are deprecated. Node names known at provision time get pre-issued certs. +- **Trap to document:** N nodes independently requesting the *identical* wildcard hits the 5/week duplicate-cert limit — distinct names or central distribution only. +- **AWS appendix (docs + Terraform, never a shipped artifact):** ALB + wildcard ACM + host-header rules for operators already on EC2 — trust-neutral there, ACM renewal genuinely zero-touch, WebRTC UDP rides beside it via per-node EIPs. Mandatory attribute overrides: `idle_timeout=4000`, `client_keep_alive=604800` (the 3600s default can kill the >1h call), header forwarding. Hard 100-target-group cap per ALB; AWS fleet rotation can still cut a multi-hour call. **Never NLB-TLS** (fixed 350s silent drop, no HTTP context). + +### (d) Homelab / no-public-IP + +Tell the truth: **nothing makes CGNAT production-grade for free, and no tunnel carries inbound UDP — WebRTC callers are unreachable behind any tunnel, so homelab-behind-tunnel is PSTN-only, period** (tunnel report). The story, in three tiers: + +1. **Dev/demo (blessed):** `ngrok http 8080` — the only tunnel with a proven Twilio Media Streams record; cloudflared has an *open* Twilio 31920 handshake bug ([cloudflared#1465](https://github.com/cloudflare/cloudflared/issues/1465)) plus recurring 1006 closures (#1282), and Cloudflare documents killing WS on edge code releases. Free-tier arithmetic: ~7 call-hours/month. Tailscale Funnel is the privacy-clean single-user variant (TLS terminates on-node) with an undisclosed bandwidth cap. +2. **Production graduation path (recommended):** cheap VPS + WireGuard as a **dumb TCP forwarder to the home box, with TLS terminating at home** (Caddy via DNS-01, or later in-process rustls). The forwarder physically cannot read audio — the strongest privacy topology of any candidate (rustls report). Or simply: run the engine on the VPS. +3. **Explicitly unsupported for production:** Cloudflare Tunnel / ngrok in the live audio path — plaintext audio on the vendor's edge (unconsented subprocessor; DPA/BAA/PCI failure), documented mid-call WS terminations, zero SLA, and Cloudflare's discretionary "disproportionate audio" ToS clause aimed at exactly this traffic profile. + +### In-process rustls: reserved as the deliberate next step, not adopted now — and not rejected + +**Engage ADR-0008 honestly: the keep-the-FOB-minimal objection is factually dead in this codebase.** ADR-0008 explicitly lists the trust-domain edge as security-constitutive FOB territory, and str0m already terminates DTLS-SRTP in-process against hostile UDP on the same aws-lc-rs provider that rustls would use — rustls 0.23.41/tokio-rustls 0.26.4 are *already in Cargo.lock* (rustls report, citing `docs/adr/0008` and the lockfile). Doctrinally this is the intended end-state, and it is the only candidate with truly zero-downtime renewal (arc-swap) plus structural absence of the entire proxy-timeout/buffering failure class. + +The reason to defer is engineering economics, not doctrine: **no turnkey ACME crate fits this dependency tree** (rustls-acme hard-violates deny.toml on rcgen/x509-parser and lacks DNS-01; tokio-rustls-acme demands reqwest 0.13 + a second C crypto lib), so the path is a 1–2 week instant-acme integration after which **rutster owns a renewal state machine forever in a 45-day-cert-lifetime world** (~8 chances/year/cert for a DIY loop bug to cause total inbound outage — invariant 1 makes that the worst failure available). Caddy buys a decade-hardened ACME loop for one supervised process. + +- **Do now (cheap, 1–2 days):** Phase 1 — `axum-server` BYO-cert TLS with hot-reload, plus **TCP_NODELAY on accepted sockets** (axum [#2521](https://github.com/tokio-rs/axum/issues/2521)/#1961 — this latent frame-coalescing bug affects the plaintext `:8080` listener behind *every* proxy candidate today; fix it regardless of this decision, with a latency assertion in the e2e sim). +- **Triggers for Phase 2 (in-binary ACME):** (1) Let's Encrypt **dns-persist-01 confirmed GA in production** — it removes DNS-provider API plumbing and closes the wildcard/CGNAT issuance gap, and instant-acme 0.8.5 already parses it; (2) a field incident where Caddy reload/`stream_close_delay` drops live calls; (3) the fleet or VPS-forwarder topology making end-to-end-encrypted-to-the-binary a selling point; (4) fatigue with maintaining the curated xcaddy DNS-plugin build. Until then, DoS isolation (handshake floods landing on the media-tick runtime) remains a real cost the proxy absorbs for free. + +--- + +## 4. Runner-ups vs unsupported + +**Supported, documented alternatives** + +| Configuration | Where | Caveats to document | +|---|---|---| +| HAProxy 3.2 LTS/3.3 | Cloud-VM operators wanting zero-reload cert hot-load (acme.sh stats-socket deploy) and `timeout tunnel` semantics | Set `timeout tunnel` in `defaults` (issue #2280); native ACME still experimental; `hard-stop-after` kills drained calls | +| Traefik v3.x | Compose users already in the Traefik ecosystem; best built-in DNS-01/wildcard of the classic family | **Pin exact versions** — three WS-breaking patch releases (#10601, v2.11.2 timeout flip, #11405); never auto-update the edge | +| ALB + ACM (Terraform module) | EC2-resident fleets | The three attribute overrides; 100-TG cap; AWS rotation can cut long calls | +| axum-server BYO-cert (Phase 1) | Operators who already have cert distribution (e.g. corporate ACME, wildcard distribution) | Operator owns renewal delivery | +| ngrok / Tailscale Funnel | Dev/demo/quickstart only | Free-tier caps; PSTN-only (no UDP); never production | + +**Explicitly unsupported** + +- Cloudflare Tunnel or ngrok in the production audio path (plaintext audio at vendor edge, documented mid-call kills, ToS exposure). +- cloudflared for Media Streams even in dev (open Twilio 31920 bug). +- NLB TLS listener (fixed 350s silent drop; no headers for signature validation). +- Caddy on-demand TLS for fleet node names (blocking first handshake vs webhook SLA). +- nginx as a recommended edge (no native DNS-01/wildcard, reload-per-renewal vs hours-long calls, 60s footgun) — works if the operator insists, but rutster won't ship config for it beyond the timeout snippet. +- Self-signed certs anywhere, including via the Twilio console toggle (account-wide blast radius, doesn't help Media Streams). + +--- + +## 5. Open risks surviving the recommendation + +1. **Caddy config-reload kills active WS tunnels; the mitigation is unproven.** `stream_close_delay` has contested/open bugs (#6420, #7222). Backing failure mode: operator edits Caddyfile mid-call → every live Media Streams tunnel closes → all calls drop. Residual even after the mandated e2e test — the test can only tell us *whether* we're exposed per version. +2. **Curated DNS-plugin matrix is a permanent maintenance surface.** Any user on an unbundled provider needs a custom xcaddy build, contradicting "docker run and it takes calls" for the behind-NAT wildcard path. dns-persist-01 eventually dissolves this; until GA it stands. +3. **dns-persist-01 timeline risk cuts both ways.** Production target was Q2 2026 (i.e., now); if it slips, both the Caddy-plugin burden persists *and* the rustls Phase 2 trigger recedes, leaving classic DNS-01 credentials-in-container as the homelab wildcard story. +4. **Quiet signaling WS remains structurally fragile everywhere.** The 20ms media leg self-protects; the WebRTC signaling WS and any hold/parked state can idle past *some* timer in every topology (Caddy needs app pings; ALB hard-caps at 4000s). App-level WS ping/pong in the engine is required belt-and-braces, per invariant 3. +5. **Cert-storage loss → LE duplicate-cert lockout → total inbound outage** (31910 on streams, 11237 on webhooks) with no maintenance window in a 24/7 non-migratable-call product. The persistent-volume requirement mitigates but volume misconfiguration in the field will happen. +6. **Homelab WebRTC gap is unsolved by design.** UDP-direct advertised-IP media cannot cross CGNAT via any recommended path; only the VPS graduation (or engine-on-VPS) fixes it. Operators will read "homelab supported" as including browser calls unless the docs say PSTN-only loudly. +7. **Misdiagnosis tax.** Field reports of "WS drops through Caddy" are usually path/upstream issues (#6958 closed invalid); with Twilio's unpublished egress IPs, "is it Caddy, the path, or Twilio" debugging has no allowlist shortcut (invariant 2). +8. **If/when Phase 2 fires: rutster owns a renewal state machine in a 90→64→45-day world.** The risk merely moves from "Caddy reload behavior" to "our ARI scheduler has a bug", plus the rcgen 0.14.7 MSRV pin gains a second dependent (Cargo.toml pin rationale, rustls report). \ No newline at end of file