rutster/docs/superpowers/specs/2026-07-05-tls-edge-decision-brief.md

# Rutster TLS/Edge Decision Brief

Synthesized from six research reports (Caddy, classic proxies, in-process rustls, tunnels, cloud-native LB, CPaaS ground truth). 2026-07-05.

---

## 1. Decision Matrix

Scores 1–5. The ground-truth report is excluded from the matrix — it is the spec, not a candidate (its scores measure how *demanding* the CPaaS side is, per its own scoring note).

| Criterion | Caddy | nginx | HAProxy | Traefik | In-process rustls | Tunnels (CF/Funnel/ngrok) | AWS ALB+ACM |
|---|---|---|---|---|---|---|---|
| WS media fitness | 4 | 5* | 5* | 5* | 5 | 2 | 3.5 |
| Cert automation | 5 | 3 | 4 | 5 | 3 | 5 | 4 |
| All-in-one fit | 5 | 4* | 4* | 4* | 5 | 3 | 1 |
| Compose fit | 5 | 5* | 5* | 5* | 4 | 4 | 2 |
| Fleet fit | 4 | 4* | 4* | 4* | 4 | 3 | 4 |
| No-public-IP story | 3 | 2 | 2 | 2 | 3 | 5 | 1 |
| Ops burden | 4 | 4* | 4* | 4* | 3 | 3 | 4 |
| **Third party in call path** | **No** | **No** | **No** | **No** | **No** | **Yes** (CF/ngrok see *plaintext* audio; Funnel ciphertext-only) | **Yes** (trust-neutral when engine is already on EC2) |

\* The classic-proxy report scored the family jointly (5/5/4/5/4/2/4); per-member sub-rankings from its findings: cert automation Traefik 5, HAProxy 4, nginx 3; HAProxy best WS mental model; Traefik has a patch-release WS-regression record.

---

## 2. Hard CPaaS Invariants (any answer must satisfy — ground-truth report)

1. **Publicly-trusted CA cert, no self-signed path exists.** Twilio refuses self-signed for webhooks and Media Streams (error 31910); the console validation toggle is account-wide, webhook-only, dev-only. Auto-renewing Let's Encrypt-class certs are availability-critical, not a nicety. ([twilio.com/docs/usage/security](https://www.twilio.com/docs/usage/security), [errors/31910](https://www.twilio.com/docs/api/errors/31910))
2. **wss:// only, on 443, open to the whole internet.** Twilio publishes no egress IPs ("any public IP address") and no CPaaS surveyed offers mTLS. Auth is application-layer only. IP allowlisting exists for Telnyx/Vonage/Plivo as an optional hardening layer — never for Twilio.
3. **One WS connection per call, up to 4h default / 24h configured, 50 msg/s/direction at 20ms cadence.** Zero tolerance for frame buffering, connection-lifetime caps, or write-side idle timers — the unidirectional case may legitimately send zero bytes from your side for hours, and neither Twilio nor Telnyx documents any WS ping behavior. Keepalive is entirely the edge's job.
4. **`<Stream>` URLs carry no query strings** — wss routing must key on path/hostname.
5. **X-Twilio-Signature is HMAC over the URL as Twilio saw it.** Any TLS-terminating edge must forward X-Forwarded-Proto/Host untouched (including on the WS upgrade request) and the app must reconstruct the public URL, or genuine traffic fails validation. ([Twilio SSL-termination blog](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library))
6. **Webhook budget: sub-second for UX, 15s hard platform cap.** No blocking operations (e.g. on-demand ACME issuance) may sit in the handshake/webhook path.
7. **TLS 1.2+1.3, mainstream ECDHE suites, never 1.3-only** (Twilio's 1.3 client support is undocumented); never pin Twilio certs.
8. **Fleet corollary:** node-addressed URLs force a publicly-valid cert+DNS name *per node* — wildcard DNS-01 or per-node ACME is imposed by the CPaaS, not the proxy choice.
9. **CGNAT corollary:** the CPaaS offers no relay/rendezvous; public reachability must be imported from the operator's side, and intermediaries are a documented frame-dropping hazard ([livekit/agents#3379](https://github.com/livekit/agents/issues/3379)).

---

## 3. Recommendation (per topology)

**Headline: Caddy is the shared edge for both shipped artifacts. In-process rustls is the ratified end-state, deferred — ship Phase 1 (BYO-cert) opportunistically, gate Phase 2 (in-binary ACME) on explicit triggers. Tunnels are dev-only.**

### (a) All-in-one container: bundle Caddy under s6

A single 47MB static Apache-2.0 binary, ~6-line Caddyfile, dual-CA ACME with background issuance and **zero-downtime in-memory cert renewal that never touches live calls** — the routine periodic event drops nothing (Caddy report). Post-upgrade WS handling is verified protocol-unaware `io.CopyBuffer` tunneling — safe for 20ms PCM. Requirements baked into the image:

- Custom xcaddy build with a curated DNS-provider plugin set (cloudflare Apache-2.0; route53/porkbun/hetzner/desec MIT; **skip duckdns** — no license file). The official image ships zero DNS plugins; without this, wildcard/behind-NAT issuance is impossible.
- `/data` on a persistent volume, non-negotiable — recreate loops otherwise hit LE's 5-certs/week duplicate limit and lock the domain out.
- `stream_close_delay` set above max call duration, **and** a CI e2e case "config reload during active simulated call" — the mitigation has an open bug trail (caddy [#6420](https://github.com/caddyserver/caddy/issues/6420), [#7222](https://github.com/caddyserver/caddy/issues/7222)) and must not be trusted untested.
- Tuned timeouts + honest X-Forwarded-* forwarding per invariants 3 and 5.

### (b) Modular compose: same Caddy build, as a compose service

Same image, same Caddyfile, same CI coverage — one edge choice across both artifacts, as preferred. Compose additionally documents "bring your own proxy": the engine keeps its plaintext `:8080` mode, and the docs ship the tuned-timeout snippets for nginx/HAProxy/Traefik (the 60s idle-timeout defaults in all three are the universal footgun — classic-proxy report).

### (c) Fleet chapter of the ADR

- **Per-node public TLS names are CPaaS-imposed** (invariant 8). Two blessed patterns: **(1) wildcard `*.pbx.domain` via DNS-01** + Caddy's documented shared-storage cluster coordination, or issue-once-and-distribute; **(2) per-node distinct certs** (`node-N.domain`) where you don't want the wildcard private key on every node — one compromised node otherwise burns the whole namespace (rustls report). Renewals are exempt from LE per-domain limits, so per-node scales.
- **Explicitly reject Caddy on-demand TLS for node names:** the first handshake to a fresh name blocks seconds on issuance, colliding with sub-second webhook budgets; the rate-limit knobs are deprecated. Node names known at provision time get pre-issued certs.
- **Trap to document:** N nodes independently requesting the *identical* wildcard hits the 5/week duplicate-cert limit — distinct names or central distribution only.
- **AWS appendix (docs + Terraform, never a shipped artifact):** ALB + wildcard ACM + host-header rules for operators already on EC2 — trust-neutral there, ACM renewal genuinely zero-touch, WebRTC UDP rides beside it via per-node EIPs. Mandatory attribute overrides: `idle_timeout=4000`, `client_keep_alive=604800` (the 3600s default can kill the >1h call), header forwarding. Hard 100-target-group cap per ALB; AWS fleet rotation can still cut a multi-hour call. **Never NLB-TLS** (fixed 350s silent drop, no HTTP context).

### (d) Homelab / no-public-IP

Tell the truth: **nothing makes CGNAT production-grade for free, and no tunnel carries inbound UDP — WebRTC callers are unreachable behind any tunnel, so homelab-behind-tunnel is PSTN-only, period** (tunnel report). The story, in three tiers:

1. **Dev/demo (blessed):** `ngrok http 8080` — the only tunnel with a proven Twilio Media Streams record; cloudflared has an *open* Twilio 31920 handshake bug ([cloudflared#1465](https://github.com/cloudflare/cloudflared/issues/1465)) plus recurring 1006 closures (#1282), and Cloudflare documents killing WS on edge code releases. Free-tier arithmetic: ~7 call-hours/month. Tailscale Funnel is the privacy-clean single-user variant (TLS terminates on-node) with an undisclosed bandwidth cap.
2. **Production graduation path (recommended):** cheap VPS + WireGuard as a **dumb TCP forwarder to the home box, with TLS terminating at home** (Caddy via DNS-01, or later in-process rustls). The forwarder physically cannot read audio — the strongest privacy topology of any candidate (rustls report). Or simply: run the engine on the VPS.
3. **Explicitly unsupported for production:** Cloudflare Tunnel / ngrok in the live audio path — plaintext audio on the vendor's edge (unconsented subprocessor; DPA/BAA/PCI failure), documented mid-call WS terminations, zero SLA, and Cloudflare's discretionary "disproportionate audio" ToS clause aimed at exactly this traffic profile.

### In-process rustls: reserved as the deliberate next step, not adopted now — and not rejected

**Engage ADR-0008 honestly: the keep-the-FOB-minimal objection is factually dead in this codebase.** ADR-0008 explicitly lists the trust-domain edge as security-constitutive FOB territory, and str0m already terminates DTLS-SRTP in-process against hostile UDP on the same aws-lc-rs provider that rustls would use — rustls 0.23.41/tokio-rustls 0.26.4 are *already in Cargo.lock* (rustls report, citing `docs/adr/0008` and the lockfile). Doctrinally this is the intended end-state, and it is the only candidate with truly zero-downtime renewal (arc-swap) plus structural absence of the entire proxy-timeout/buffering failure class.

The reason to defer is engineering economics, not doctrine: **no turnkey ACME crate fits this dependency tree** (rustls-acme hard-violates deny.toml on rcgen/x509-parser and lacks DNS-01; tokio-rustls-acme demands reqwest 0.13 + a second C crypto lib), so the path is a 1–2 week instant-acme integration after which **rutster owns a renewal state machine forever in a 45-day-cert-lifetime world** (~8 chances/year/cert for a DIY loop bug to cause total inbound outage — invariant 1 makes that the worst failure available). Caddy buys a decade-hardened ACME loop for one supervised process.

- **Do now (cheap, 1–2 days):** Phase 1 — `axum-server` BYO-cert TLS with hot-reload, plus **TCP_NODELAY on accepted sockets** (axum [#2521](https://github.com/tokio-rs/axum/issues/2521)/#1961 — this latent frame-coalescing bug affects the plaintext `:8080` listener behind *every* proxy candidate today; fix it regardless of this decision, with a latency assertion in the e2e sim).
- **Triggers for Phase 2 (in-binary ACME):** (1) Let's Encrypt **dns-persist-01 confirmed GA in production** — it removes DNS-provider API plumbing and closes the wildcard/CGNAT issuance gap, and instant-acme 0.8.5 already parses it; (2) a field incident where Caddy reload/`stream_close_delay` drops live calls; (3) the fleet or VPS-forwarder topology making end-to-end-encrypted-to-the-binary a selling point; (4) fatigue with maintaining the curated xcaddy DNS-plugin build. Until then, DoS isolation (handshake floods landing on the media-tick runtime) remains a real cost the proxy absorbs for free.

---

## 4. Runner-ups vs unsupported

**Supported, documented alternatives**

| Configuration | Where | Caveats to document |
|---|---|---|
| HAProxy 3.2 LTS/3.3 | Cloud-VM operators wanting zero-reload cert hot-load (acme.sh stats-socket deploy) and `timeout tunnel` semantics | Set `timeout tunnel` in `defaults` (issue #2280); native ACME still experimental; `hard-stop-after` kills drained calls |
| Traefik v3.x | Compose users already in the Traefik ecosystem; best built-in DNS-01/wildcard of the classic family | **Pin exact versions** — three WS-breaking patch releases (#10601, v2.11.2 timeout flip, #11405); never auto-update the edge |
| ALB + ACM (Terraform module) | EC2-resident fleets | The three attribute overrides; 100-TG cap; AWS rotation can cut long calls |
| axum-server BYO-cert (Phase 1) | Operators who already have cert distribution (e.g. corporate ACME, wildcard distribution) | Operator owns renewal delivery |
| ngrok / Tailscale Funnel | Dev/demo/quickstart only | Free-tier caps; PSTN-only (no UDP); never production |

**Explicitly unsupported**

- Cloudflare Tunnel or ngrok in the production audio path (plaintext audio at vendor edge, documented mid-call kills, ToS exposure).
- cloudflared for Media Streams even in dev (open Twilio 31920 bug).
- NLB TLS listener (fixed 350s silent drop; no headers for signature validation).
- Caddy on-demand TLS for fleet node names (blocking first handshake vs webhook SLA).
- nginx as a recommended edge (no native DNS-01/wildcard, reload-per-renewal vs hours-long calls, 60s footgun) — works if the operator insists, but rutster won't ship config for it beyond the timeout snippet.
- Self-signed certs anywhere, including via the Twilio console toggle (account-wide blast radius, doesn't help Media Streams).

---

## 5. Open risks surviving the recommendation

1. **Caddy config-reload kills active WS tunnels; the mitigation is unproven.** `stream_close_delay` has contested/open bugs (#6420, #7222). Backing failure mode: operator edits Caddyfile mid-call → every live Media Streams tunnel closes → all calls drop. Residual even after the mandated e2e test — the test can only tell us *whether* we're exposed per version.
2. **Curated DNS-plugin matrix is a permanent maintenance surface.** Any user on an unbundled provider needs a custom xcaddy build, contradicting "docker run and it takes calls" for the behind-NAT wildcard path. dns-persist-01 eventually dissolves this; until GA it stands.
3. **dns-persist-01 timeline risk cuts both ways.** Production target was Q2 2026 (i.e., now); if it slips, both the Caddy-plugin burden persists *and* the rustls Phase 2 trigger recedes, leaving classic DNS-01 credentials-in-container as the homelab wildcard story.
4. **Quiet signaling WS remains structurally fragile everywhere.** The 20ms media leg self-protects; the WebRTC signaling WS and any hold/parked state can idle past *some* timer in every topology (Caddy needs app pings; ALB hard-caps at 4000s). App-level WS ping/pong in the engine is required belt-and-braces, per invariant 3.
5. **Cert-storage loss → LE duplicate-cert lockout → total inbound outage** (31910 on streams, 11237 on webhooks) with no maintenance window in a 24/7 non-migratable-call product. The persistent-volume requirement mitigates but volume misconfiguration in the field will happen.
6. **Homelab WebRTC gap is unsolved by design.** UDP-direct advertised-IP media cannot cross CGNAT via any recommended path; only the VPS graduation (or engine-on-VPS) fixes it. Operators will read "homelab supported" as including browser calls unless the docs say PSTN-only loudly.
7. **Misdiagnosis tax.** Field reports of "WS drops through Caddy" are usually path/upstream issues (#6958 closed invalid); with Twilio's unpublished egress IPs, "is it Caddy, the path, or Twilio" debugging has no allowlist shortcut (invariant 2).
8. **If/when Phase 2 fires: rutster owns a renewal state machine in a 90→64→45-day world.** The risk merely moves from "Caddy reload behavior" to "our ARI scheduler has a bug", plus the rcgen 0.14.7 MSRV pin gains a second dependent (Cargo.toml pin rationale, rustls report).