rutster/docs/superpowers/specs/2026-07-05-tls-edge-decision-brief.md

Rutster TLS/Edge Decision Brief

Synthesized from six research reports (Caddy, classic proxies, in-process rustls, tunnels, cloud-native LB, CPaaS ground truth). 2026-07-05.

---

1. Decision Matrix

Scores 1–5. The ground-truth report is excluded from the matrix — it is the spec, not a candidate (its scores measure how demanding the CPaaS side is, per its own scoring note).

Criterion	Caddy	nginx	HAProxy	Traefik	In-process rustls	Tunnels (CF/Funnel/ngrok)	AWS ALB+ACM
WS media fitness	4	5*	5*	5*	5	2	3.5
Cert automation	5	3	4	5	3	5	4
All-in-one fit	5	4*	4*	4*	5	3	1
Compose fit	5	5*	5*	5*	4	4	2
Fleet fit	4	4*	4*	4*	4	3	4
No-public-IP story	3	2	2	2	3	5	1
Ops burden	4	4*	4*	4*	3	3	4
Third party in call path	No	No	No	No	No	Yes (CF/ngrok see plaintext audio; Funnel ciphertext-only)	Yes (trust-neutral when engine is already on EC2)

* The classic-proxy report scored the family jointly (5/5/4/5/4/2/4); per-member sub-rankings from its findings: cert automation Traefik 5, HAProxy 4, nginx 3; HAProxy best WS mental model; Traefik has a patch-release WS-regression record.

---

2. Hard CPaaS Invariants (any answer must satisfy — ground-truth report)

1. Publicly-trusted CA cert, no self-signed path exists. Twilio refuses self-signed for webhooks and Media Streams (error 31910); the console validation toggle is account-wide, webhook-only, dev-only. Auto-renewing Let's Encrypt-class certs are availability-critical, not a nicety. (twilio.com/docs/usage/security, errors/31910)
2. wss:// only, on 443, open to the whole internet. Twilio publishes no egress IPs ("any public IP address") and no CPaaS surveyed offers mTLS. Auth is application-layer only. IP allowlisting exists for Telnyx/Vonage/Plivo as an optional hardening layer — never for Twilio.
3. One WS connection per call, up to 4h default / 24h configured, 50 msg/s/direction at 20ms cadence. Zero tolerance for frame buffering, connection-lifetime caps, or write-side idle timers — the unidirectional case may legitimately send zero bytes from your side for hours, and neither Twilio nor Telnyx documents any WS ping behavior. Keepalive is entirely the edge's job.
4. <Stream> URLs carry no query strings — wss routing must key on path/hostname.
5. X-Twilio-Signature is HMAC over the URL as Twilio saw it. Any TLS-terminating edge must forward X-Forwarded-Proto/Host untouched (including on the WS upgrade request) and the app must reconstruct the public URL, or genuine traffic fails validation. (Twilio SSL-termination blog)
6. Webhook budget: sub-second for UX, 15s hard platform cap. No blocking operations (e.g. on-demand ACME issuance) may sit in the handshake/webhook path.
7. TLS 1.2+1.3, mainstream ECDHE suites, never 1.3-only (Twilio's 1.3 client support is undocumented); never pin Twilio certs.
8. Fleet corollary: node-addressed URLs force a publicly-valid cert+DNS name per node — wildcard DNS-01 or per-node ACME is imposed by the CPaaS, not the proxy choice.
9. CGNAT corollary: the CPaaS offers no relay/rendezvous; public reachability must be imported from the operator's side, and intermediaries are a documented frame-dropping hazard (livekit/agents#3379).

---

3. Recommendation (per topology)

Headline: Caddy is the shared edge for both shipped artifacts. In-process rustls is the ratified end-state, deferred — ship Phase 1 (BYO-cert) opportunistically, gate Phase 2 (in-binary ACME) on explicit triggers. Tunnels are dev-only.

(a) All-in-one container: bundle Caddy under s6

A single 47MB static Apache-2.0 binary, ~6-line Caddyfile, dual-CA ACME with background issuance and zero-downtime in-memory cert renewal that never touches live calls — the routine periodic event drops nothing (Caddy report). Post-upgrade WS handling is verified protocol-unaware io.CopyBuffer tunneling — safe for 20ms PCM. Requirements baked into the image:

• Custom xcaddy build with a curated DNS-provider plugin set (cloudflare Apache-2.0; route53/porkbun/hetzner/desec MIT; skip duckdns — no license file). The official image ships zero DNS plugins; without this, wildcard/behind-NAT issuance is impossible.
• /data on a persistent volume, non-negotiable — recreate loops otherwise hit LE's 5-certs/week duplicate limit and lock the domain out.
• stream_close_delay set above max call duration, and a CI e2e case "config reload during active simulated call" — the mitigation has an open bug trail (caddy #6420, #7222) and must not be trusted untested.
• Tuned timeouts + honest X-Forwarded-* forwarding per invariants 3 and 5.

(b) Modular compose: same Caddy build, as a compose service

Same image, same Caddyfile, same CI coverage — one edge choice across both artifacts, as preferred. Compose additionally documents "bring your own proxy": the engine keeps its plaintext :8080 mode, and the docs ship the tuned-timeout snippets for nginx/HAProxy/Traefik (the 60s idle-timeout defaults in all three are the universal footgun — classic-proxy report).

(c) Fleet chapter of the ADR

• Per-node public TLS names are CPaaS-imposed (invariant 8). Two blessed patterns: (1) wildcard *.pbx.domain via DNS-01 + Caddy's documented shared-storage cluster coordination, or issue-once-and-distribute; (2) per-node distinct certs (node-N.domain) where you don't want the wildcard private key on every node — one compromised node otherwise burns the whole namespace (rustls report). Renewals are exempt from LE per-domain limits, so per-node scales.
• Explicitly reject Caddy on-demand TLS for node names: the first handshake to a fresh name blocks seconds on issuance, colliding with sub-second webhook budgets; the rate-limit knobs are deprecated. Node names known at provision time get pre-issued certs.
• Trap to document: N nodes independently requesting the identical wildcard hits the 5/week duplicate-cert limit — distinct names or central distribution only.
• AWS appendix (docs + Terraform, never a shipped artifact): ALB + wildcard ACM + host-header rules for operators already on EC2 — trust-neutral there, ACM renewal genuinely zero-touch, WebRTC UDP rides beside it via per-node EIPs. Mandatory attribute overrides: idle_timeout=4000, client_keep_alive=604800 (the 3600s default can kill the >1h call), header forwarding. Hard 100-target-group cap per ALB; AWS fleet rotation can still cut a multi-hour call. Never NLB-TLS (fixed 350s silent drop, no HTTP context).

(d) Homelab / no-public-IP

Tell the truth: nothing makes CGNAT production-grade for free, and no tunnel carries inbound UDP — WebRTC callers are unreachable behind any tunnel, so homelab-behind-tunnel is PSTN-only, period (tunnel report). The story, in three tiers:

1. Dev/demo (blessed): ngrok http 8080 — the only tunnel with a proven Twilio Media Streams record; cloudflared has an open Twilio 31920 handshake bug (cloudflared#1465) plus recurring 1006 closures (#1282), and Cloudflare documents killing WS on edge code releases. Free-tier arithmetic: ~7 call-hours/month. Tailscale Funnel is the privacy-clean single-user variant (TLS terminates on-node) with an undisclosed bandwidth cap.
2. Production graduation path (recommended): cheap VPS + WireGuard as a dumb TCP forwarder to the home box, with TLS terminating at home (Caddy via DNS-01, or later in-process rustls). The forwarder physically cannot read audio — the strongest privacy topology of any candidate (rustls report). Or simply: run the engine on the VPS.
3. Explicitly unsupported for production: Cloudflare Tunnel / ngrok in the live audio path — plaintext audio on the vendor's edge (unconsented subprocessor; DPA/BAA/PCI failure), documented mid-call WS terminations, zero SLA, and Cloudflare's discretionary "disproportionate audio" ToS clause aimed at exactly this traffic profile.

In-process rustls: reserved as the deliberate next step, not adopted now — and not rejected

Engage ADR-0008 honestly: the keep-the-FOB-minimal objection is factually dead in this codebase. ADR-0008 explicitly lists the trust-domain edge as security-constitutive FOB territory, and str0m already terminates DTLS-SRTP in-process against hostile UDP on the same aws-lc-rs provider that rustls would use — rustls 0.23.41/tokio-rustls 0.26.4 are already in Cargo.lock (rustls report, citing docs/adr/0008 and the lockfile). Doctrinally this is the intended end-state, and it is the only candidate with truly zero-downtime renewal (arc-swap) plus structural absence of the entire proxy-timeout/buffering failure class.

The reason to defer is engineering economics, not doctrine: no turnkey ACME crate fits this dependency tree (rustls-acme hard-violates deny.toml on rcgen/x509-parser and lacks DNS-01; tokio-rustls-acme demands reqwest 0.13 + a second C crypto lib), so the path is a 1–2 week instant-acme integration after which rutster owns a renewal state machine forever in a 45-day-cert-lifetime world (~8 chances/year/cert for a DIY loop bug to cause total inbound outage — invariant 1 makes that the worst failure available). Caddy buys a decade-hardened ACME loop for one supervised process.

• Do now (cheap, 1–2 days): Phase 1 — axum-server BYO-cert TLS with hot-reload, plus TCP_NODELAY on accepted sockets (axum #2521/#1961 — this latent frame-coalescing bug affects the plaintext :8080 listener behind every proxy candidate today; fix it regardless of this decision, with a latency assertion in the e2e sim).
• Triggers for Phase 2 (in-binary ACME): (1) Let's Encrypt dns-persist-01 confirmed GA in production — it removes DNS-provider API plumbing and closes the wildcard/CGNAT issuance gap, and instant-acme 0.8.5 already parses it; (2) a field incident where Caddy reload/stream_close_delay drops live calls; (3) the fleet or VPS-forwarder topology making end-to-end-encrypted-to-the-binary a selling point; (4) fatigue with maintaining the curated xcaddy DNS-plugin build. Until then, DoS isolation (handshake floods landing on the media-tick runtime) remains a real cost the proxy absorbs for free.

---

4. Runner-ups vs unsupported

Supported, documented alternatives

Configuration	Where	Caveats to document
HAProxy 3.2 LTS/3.3	Cloud-VM operators wanting zero-reload cert hot-load (acme.sh stats-socket deploy) and timeout tunnel semantics	Set timeout tunnel in defaults (issue #2280); native ACME still experimental; hard-stop-after kills drained calls
Traefik v3.x	Compose users already in the Traefik ecosystem; best built-in DNS-01/wildcard of the classic family	Pin exact versions — three WS-breaking patch releases (#10601, v2.11.2 timeout flip, #11405); never auto-update the edge
ALB + ACM (Terraform module)	EC2-resident fleets	The three attribute overrides; 100-TG cap; AWS rotation can cut long calls
axum-server BYO-cert (Phase 1)	Operators who already have cert distribution (e.g. corporate ACME, wildcard distribution)	Operator owns renewal delivery
ngrok / Tailscale Funnel	Dev/demo/quickstart only	Free-tier caps; PSTN-only (no UDP); never production

Explicitly unsupported

• Cloudflare Tunnel or ngrok in the production audio path (plaintext audio at vendor edge, documented mid-call kills, ToS exposure).
• cloudflared for Media Streams even in dev (open Twilio 31920 bug).
• NLB TLS listener (fixed 350s silent drop; no headers for signature validation).
• Caddy on-demand TLS for fleet node names (blocking first handshake vs webhook SLA).
• nginx as a recommended edge (no native DNS-01/wildcard, reload-per-renewal vs hours-long calls, 60s footgun) — works if the operator insists, but rutster won't ship config for it beyond the timeout snippet.
• Self-signed certs anywhere, including via the Twilio console toggle (account-wide blast radius, doesn't help Media Streams).

---

5. Open risks surviving the recommendation

1. Caddy config-reload kills active WS tunnels; the mitigation is unproven. stream_close_delay has contested/open bugs (#6420, #7222). Backing failure mode: operator edits Caddyfile mid-call → every live Media Streams tunnel closes → all calls drop. Residual even after the mandated e2e test — the test can only tell us whether we're exposed per version.
2. Curated DNS-plugin matrix is a permanent maintenance surface. Any user on an unbundled provider needs a custom xcaddy build, contradicting "docker run and it takes calls" for the behind-NAT wildcard path. dns-persist-01 eventually dissolves this; until GA it stands.
3. dns-persist-01 timeline risk cuts both ways. Production target was Q2 2026 (i.e., now); if it slips, both the Caddy-plugin burden persists and the rustls Phase 2 trigger recedes, leaving classic DNS-01 credentials-in-container as the homelab wildcard story.
4. Quiet signaling WS remains structurally fragile everywhere. The 20ms media leg self-protects; the WebRTC signaling WS and any hold/parked state can idle past some timer in every topology (Caddy needs app pings; ALB hard-caps at 4000s). App-level WS ping/pong in the engine is required belt-and-braces, per invariant 3.
5. Cert-storage loss → LE duplicate-cert lockout → total inbound outage (31910 on streams, 11237 on webhooks) with no maintenance window in a 24/7 non-migratable-call product. The persistent-volume requirement mitigates but volume misconfiguration in the field will happen.
6. Homelab WebRTC gap is unsolved by design. UDP-direct advertised-IP media cannot cross CGNAT via any recommended path; only the VPS graduation (or engine-on-VPS) fixes it. Operators will read "homelab supported" as including browser calls unless the docs say PSTN-only loudly.
7. Misdiagnosis tax. Field reports of "WS drops through Caddy" are usually path/upstream issues (#6958 closed invalid); with Twilio's unpublished egress IPs, "is it Caddy, the path, or Twilio" debugging has no allowlist shortcut (invariant 2).
8. If/when Phase 2 fires: rutster owns a renewal state machine in a 90→64→45-day world. The risk merely moves from "Caddy reload behavior" to "our ARI scheduler has a bug", plus the rcgen 0.14.7 MSRV pin gains a second dependent (Cargo.toml pin rationale, rustls report).