docs(spec): deployment topology — one binary, three blessed shapes
Closes the deployment-topology design ADR-0008 reserved: T1 all-in-one / T2 modular compose / T3 fleet-on-paper, same FOB binary, config-only differences. Companion TLS/edge decision brief (six-family research survey) included; spec produces ADR-0011 plus slices A–G. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8 Signed-off-by: Aaron D. Lee <himself@adlee.work>
This commit is contained in:
391
docs/superpowers/specs/2026-07-05-deployment-topology-design.md
Normal file
391
docs/superpowers/specs/2026-07-05-deployment-topology-design.md
Normal file
@@ -0,0 +1,391 @@
|
||||
# Rutster deployment topology — one binary, three blessed shapes
|
||||
|
||||
- **Status:** Draft (pending review)
|
||||
- **Date:** 2026-07-05
|
||||
- **Origin:** [ADR-0008](../../adr/0008-fob-and-green-zone.md) explicitly reserved a deployment
|
||||
topology ADR "once that design closes." This spec closes it. Brainstormed 2026-07-05; the
|
||||
TLS/edge fork was researched via a six-family survey — see the companion
|
||||
[TLS/edge decision brief](2026-07-05-tls-edge-decision-brief.md).
|
||||
- **Produces:** **ADR-0011 (deployment topology)** — drafted from §2–§4 of this spec, Proposed,
|
||||
ratified per repo process — plus the artifacts and code slices in §5–§6.
|
||||
- **Depends on (already merged):** slice-5/seams (drain lifecycle, `/readyz`, admission cap,
|
||||
`EventSink`, `MediaAddressConfig`, `config.rs` env-parser pattern); slice-4½ benchmark/sim
|
||||
(S1–S8, CI-regressed — the future source of the real per-node admission cap).
|
||||
- **Related:** [ADR-0002](../../adr/0002-north-star-and-fused-core.md) (fused per-call vertical),
|
||||
[ADR-0005](../../adr/0005-event-bus.md) (Valkey = bus + KV + presence; media never rides the
|
||||
bus; bus is never billing source-of-truth), [ADR-0006](../../adr/0006-ingress-posture.md)
|
||||
(two inbound protocols only; tap is egress-only), [ADR-0007](../../adr/0007-trunk-rented-transport.md)
|
||||
(rented transport ⇒ public HTTPS/WSS reachability), [ADR-0009](../../adr/0009-spend-gate-honest-rescope.md)
|
||||
(ledger trait: in-memory single-node, Valkey-backed fleets), [ADR-0010](../../adr/0010-spearhead-benchmark-sim-harness.md)
|
||||
(sequencing this spec must not reorder), README pillar 6 ("one binary, one bus, one deploy").
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
**The FOB never decomposes; topology is a configuration property, not a code property.** The same
|
||||
binary runs in every blessed shape; only `RUTSTER_*` env and what is deployed alongside it differ.
|
||||
"Microservices" in rutster means *green-zone services get their own processes/containers* — never
|
||||
that the per-call vertical splits (ADR-0002).
|
||||
|
||||
Three blessed topologies:
|
||||
|
||||
| | Shape | Status after this work |
|
||||
|---|---|---|
|
||||
| **T1 — Solo** | One all-in-one container: s6 supervises Caddy + FOB + brain + valkey-server (EventSink consumer day one; ledger/directory later). `docker run` with a domain + Twilio creds → taking calls. | Shipped artifact |
|
||||
| **T2 — Modular** | Compose stack: `caddy`, `engine`, `brain` (shared netns with engine), `valkey` as separate services. `compose up` is the pillar. | Shipped artifact (reference deployment) |
|
||||
| **T3 — Fleet** | N symmetric nodes + shared green zone (one Valkey, object storage). Node-addressed URLs, answer-time placement via Valkey presence. | ADR chapter only (paper) |
|
||||
|
||||
Riders shipping with the artifacts: TCP_NODELAY + app-level WS-ping hygiene, trunk config fixes,
|
||||
in-process TLS **Phase 1** (BYO-cert, hot-reload), a hand-rolled Prometheus `/metrics` endpoint,
|
||||
`ValkeyEventSink` (first Valkey consumer), image-publish CI, and a `docs/deploy/` documentation
|
||||
tree with a copy-paste-to-first-call acceptance bar.
|
||||
|
||||
**Six things this design deliberately is NOT:**
|
||||
|
||||
- **Not a decomposition of the FOB into services.** Media/reflex/tap/spend stay one process,
|
||||
one trust domain, per ADR-0002. Rejected permanently, not deferred.
|
||||
- **Not serverless.** Lambdas are categorically unfit for the engine: non-migratable multi-hour
|
||||
calls, per-session UDP sockets, in-process media state. Webhook-on-Lambda buys nothing because
|
||||
placement needs fleet state anyway. Stated as a rejected shape in ADR-0011.
|
||||
- **Not k8s manifests/Helm.** Compose-first (pillar 6). k8s notes live in docs only (drain vs
|
||||
`terminationGracePeriodSeconds`); manifests are a later rung if demand appears.
|
||||
- **Not the durable CDR pipeline.** `ValkeyEventSink` is evidence preservation on a capped
|
||||
stream — explicitly NOT the billing ledger. CDR → object storage stays deferred
|
||||
(ADR-0005 source-of-truth rule).
|
||||
- **Not the zero-egress / air-gap profile.** T1 is about one-command simplicity, not isolation.
|
||||
Zero-egress (SBC layer-2 ingress + self-hosted brain) is named as a defined-but-deferred
|
||||
profile in ADR-0011, one footnote section, no code.
|
||||
- **Not in-binary ACME (Phase 2).** Deferred behind named triggers (§3.4). No turnkey crate fits
|
||||
the tree today (rustls-acme conflicts with the deny.toml rcgen/x509-parser posture); owning a
|
||||
renewal state machine in a 45-day-cert world is a deliberate, triggered decision.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope
|
||||
|
||||
### 1.1 In scope
|
||||
|
||||
- **ADR-0011** (deployment topology): the three shapes, the edge doctrine, the fleet placement
|
||||
paradigm, the rejected shapes.
|
||||
- **Artifacts** under `deploy/`: multi-stage `Dockerfile` (four first-party images), `compose.yaml`,
|
||||
`Caddyfile`, `.env.example`.
|
||||
- **Code:** TCP_NODELAY, app-level WS pings, trunk config fixes (webhook base URL, credential
|
||||
wiring, trusted-proxy header posture), rustls Phase 1 (BYO-cert TLS), `/metrics`,
|
||||
`ValkeyEventSink`.
|
||||
- **CI:** containerized smoke suite (incl. Caddy-reload-during-live-call), image publish on tag.
|
||||
- **Docs:** `docs/deploy/` tree (§10).
|
||||
|
||||
### 1.2 Out of scope (named in ADR-0011 as deferred, so the boundary is deliberate)
|
||||
|
||||
- T3 fleet **implementation** (presence heartbeats, directory redirects, placement code).
|
||||
- Object-storage CDR durability; multi-tenancy (needs its own ADR before schemas ossify);
|
||||
wss:// tap + brain fleet + resume tokens (step 6); media-thread sharding graduation;
|
||||
in-binary ACME (Phase 2); SBC layer-2 wire contract; arm64 images.
|
||||
|
||||
---
|
||||
|
||||
## 2. Topologies
|
||||
|
||||
### 2.1 T1 — Solo (all-in-one container)
|
||||
|
||||
One image (`rutster-allinone`), tini/s6-overlay as PID 1 supervising four processes:
|
||||
|
||||
| Process | Role | Binding |
|
||||
|---|---|---|
|
||||
| `caddy` | Edge: ACME, TLS termination, WS proxy to FOB | :443/:80 public |
|
||||
| `rutster` (FOB) | Engine | 127.0.0.1:8080 (behind Caddy); media UDP direct |
|
||||
| `rutster-brain-realtime` | Brain | 127.0.0.1:8082 loopback tap (as today) |
|
||||
| `valkey-server` | **Dark on day one** except EventSink (§5.6) | 127.0.0.1:6379 |
|
||||
|
||||
Volumes (both non-negotiable, documented loudly):
|
||||
- `/data` — Caddy cert/ACME state. Loss risks Let's Encrypt duplicate-cert lockout (5/week)
|
||||
→ total inbound outage class.
|
||||
- `/var/lib/valkey` — stream/state persistence.
|
||||
|
||||
WebRTC media UDP goes **direct to the FOB**, never through Caddy: host networking recommended;
|
||||
otherwise published `RUTSTER_MEDIA_PORT_RANGE` + `RUTSTER_MEDIA_ADVERTISED_IP` (NAT 1:1 model,
|
||||
no STUN — unchanged from slice-5/seams).
|
||||
|
||||
Valkey ships bundled from v1 **before most consumers exist** (decision 2026-07-05): the operator
|
||||
contract (volumes, ports, upgrade shape) is stable from day one, so the spend ledger and fleet
|
||||
directory land later without changing the deployment shape. Cost accepted: ~15 MB and one
|
||||
mostly-idle process.
|
||||
|
||||
### 2.2 T2 — Modular (compose stack) — the reference deployment
|
||||
|
||||
Services: `caddy` (image `rutster-edge`), `engine` (`rutster-engine`), `brain` (`rutster-brain`),
|
||||
`valkey` (upstream `valkey/valkey`).
|
||||
|
||||
- **Brain shares the engine's network namespace** (`network_mode: "service:engine"`) so the
|
||||
loopback-only tap posture (`resolve_tap_url` rejects non-loopback until step 6) survives
|
||||
unchanged. ADR-0011 notes the brain graduates to its own netns when the wss:// tap lands.
|
||||
- Valkey is its own service on the compose network — network-near, satisfies ADR-0005's sub-ms
|
||||
rule.
|
||||
- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` (§8).
|
||||
- BYO-proxy supported: disable the `caddy` service; the engine keeps its plaintext `:8080` mode;
|
||||
tuned-timeout snippets for nginx/HAProxy/Traefik ship in docs (§10).
|
||||
|
||||
### 2.3 T3 — Fleet (ADR chapter only; no code in this epoch)
|
||||
|
||||
N **symmetric** nodes, each running the T1 process set *minus* valkey, plus a shared green zone:
|
||||
one Valkey (presence + directory + ADR-0009 spend ledger), object storage (named as required,
|
||||
still deferred). Wildcard DNS `*.pbx.domain`; per-node certs (§3.3). Per-node admission cap =
|
||||
the ADR-0010 benchmark number (replacing the `RUTSTER_MAX_SESSIONS=64` placeholder). Scale-in is
|
||||
drain-then-terminate only; spot instances rejected for the engine tier (2026-07-04 review).
|
||||
Placement paradigm: §4.
|
||||
|
||||
---
|
||||
|
||||
## 3. Edge doctrine
|
||||
|
||||
### 3.1 CPaaS invariants (the spec any edge must satisfy — research §2)
|
||||
|
||||
1. Publicly-CA-trusted cert; **no self-signed path exists** (Twilio error 31910). Auto-renewal is
|
||||
availability-critical.
|
||||
2. wss:// on 443, open to the internet: Twilio publishes no egress IPs, offers no mTLS. Auth is
|
||||
application-layer only (HMAC signatures).
|
||||
3. One WS per call, up to 24 h, 50 msg/s/direction at 20 ms cadence; zero tolerance for frame
|
||||
buffering, lifetime caps, or write-side idle timers. Neither Twilio nor Telnyx documents WS
|
||||
keepalive — **keepalive is entirely our job** (§5.2).
|
||||
4. `<Stream>` URLs carry **no query strings** — wss routing keys on hostname/path only
|
||||
(constrains §4).
|
||||
5. `X-Twilio-Signature` is HMAC over the URL *as Twilio saw it*: the edge must forward
|
||||
`X-Forwarded-Proto/Host` honestly and the FOB must reconstruct the public URL (§5.3).
|
||||
6. Webhook budget: sub-second for UX, 15 s hard cap — no blocking issuance in the handshake path.
|
||||
7. TLS 1.2+1.3, mainstream ECDHE, never 1.3-only; never pin Twilio certs.
|
||||
|
||||
Design for Twilio; Telnyx/Vonage come for free (they are strictly less demanding, plus optional
|
||||
source-IP allowlisting as extra hardening Twilio can't use).
|
||||
|
||||
### 3.2 Bundled edge: Caddy (both shipped artifacts, same build, same Caddyfile)
|
||||
|
||||
Custom xcaddy build (`rutster-edge` image): Caddy core (Apache-2.0) + curated DNS-01 plugin set —
|
||||
cloudflare, route53, porkbun, hetzner, desec (duckdns excluded: no license file). Zero-downtime
|
||||
in-memory cert renewal; verified protocol-unaware WS tunneling (no frame buffering). Config:
|
||||
~6-line Caddyfile with tuned timeouts, honest `X-Forwarded-*`, `stream_close_delay` above max
|
||||
call duration — **and a CI e2e case for config-reload-during-live-call, because the upstream
|
||||
mitigation has an open bug trail (caddy #6420/#7222) and is not trusted untested.**
|
||||
|
||||
### 3.3 Fleet cert story (ADR chapter)
|
||||
|
||||
Per-node public TLS names are CPaaS-imposed (invariant 4 + node-addressed placement). Two blessed
|
||||
patterns: wildcard `*.pbx.domain` via DNS-01 with central issuance/distribution, or per-node
|
||||
distinct certs (`node-N.domain`) where the wildcard private key must not live on every node.
|
||||
Traps stated in the ADR: N nodes independently requesting the identical wildcard hit the LE
|
||||
duplicate-cert limit; **Caddy on-demand TLS is rejected for node names** (blocking first
|
||||
handshake vs the webhook budget).
|
||||
|
||||
### 3.4 In-process TLS: staged (the ratified end-state, not adopted wholesale now)
|
||||
|
||||
ADR-0008 claims the trust-domain edge as security-constitutive FOB territory, and str0m already
|
||||
terminates DTLS-SRTP in-process on the same aws-lc-rs provider — "keep TLS out of the FOB" is not
|
||||
the objection. The objection is economics: owning an ACME renewal state machine forever.
|
||||
|
||||
- **Phase 1 (this epoch, §5.4):** `axum-server` + rustls with operator-supplied certs, hot-reload
|
||||
without dropping live WS. Plaintext-behind-Caddy stays the default.
|
||||
- **Phase 2 (in-binary ACME) triggers, named in ADR-0011:** (a) Let's Encrypt dns-persist-01
|
||||
confirmed GA (closes the wildcard/CGNAT issuance gap; instant-acme already parses it);
|
||||
(b) a field incident where Caddy reload/`stream_close_delay` drops live calls; (c) fleet or
|
||||
VPS-forwarder topology making encrypted-to-the-binary a selling point; (d) maintenance fatigue
|
||||
with the curated xcaddy plugin build.
|
||||
|
||||
### 3.5 Tunnels and cloud LBs
|
||||
|
||||
- **Tunnels are dev/demo-only, stated bluntly.** Cloudflare Tunnel and ngrok see plaintext audio
|
||||
at the vendor edge (unconsented subprocessor — DPA/BAA/PCI failure), document mid-call WS kills,
|
||||
and cloudflared has an open Twilio handshake bug (#1465). ngrok is the blessed 5-minute demo
|
||||
path (only tunnel with a proven Media Streams record). Tailscale Funnel: privacy-clean
|
||||
single-user demo (TLS terminates on-node), unsizable beyond that.
|
||||
- **No tunnel carries inbound UDP → homelab behind CGNAT is PSTN-only, period.** Production
|
||||
graduation path: cheap VPS + WireGuard as a dumb TCP forwarder with TLS terminating at home
|
||||
(forwarder physically cannot read audio), or run the engine on the VPS.
|
||||
- **AWS appendix (docs only, never a shipped artifact):** ALB + wildcard ACM + host-header rules;
|
||||
mandatory attribute overrides `idle_timeout=4000`, `client_keep_alive=604800`, header
|
||||
forwarding. **Never NLB-TLS** (fixed 350 s silent drop, no HTTP context for signatures).
|
||||
|
||||
---
|
||||
|
||||
## 4. Fleet placement paradigm (ADR chapter; the trunk webhook handler and session API grow
|
||||
*toward* this, no fleet code in this epoch)
|
||||
|
||||
**Node-addressed URLs + Valkey presence; no router tier.** (Decision 2026-07-05 over
|
||||
routing-tier-proxies-everything and distinguished-control-node.)
|
||||
|
||||
- Nodes heartbeat capacity into Valkey (TTL'd presence keys — the ADR-0005 presence layer).
|
||||
- Placement happens at answer time by whichever node the request lands on (`api.domain`,
|
||||
DNS round-robin): read fleet capacity → pick owner → return TwiML whose Stream URL is
|
||||
`wss://node-K.pbx.domain/...`. The media WS dials the owning node directly — zero extra hops,
|
||||
consistent with connection-follows-ownership.
|
||||
- WebRTC: create-session response carries the owner's node-addressed base URL + advertised media
|
||||
IP (UDP is already direct).
|
||||
- Misdirected control ops (e.g. DELETE hitting a non-owner) consult the Valkey directory and
|
||||
**307-redirect** to the owner.
|
||||
- Degraded mode (Valkey dark): the answering node self-places if it has capacity; presence heals
|
||||
via TTL on recovery. Fail-safe degradation, consistent with ADR-0009's posture.
|
||||
- Single-node is the degenerate case of the same code: a one-entry directory.
|
||||
|
||||
Rationale: Twilio's answer-time Stream URL is purpose-shaped for this (invariant 4 permits
|
||||
hostname routing); a proxying router tier would put every 20 ms frame through an extra userspace
|
||||
hop and re-create the SPOF the symmetric design avoids.
|
||||
|
||||
---
|
||||
|
||||
## 5. Code changes (workspace)
|
||||
|
||||
### 5.1 TCP_NODELAY (hygiene, unconditional)
|
||||
|
||||
Set on every accepted socket (axum #2521 — Nagle can coalesce 20 ms WS frames on the plaintext
|
||||
listener behind *any* proxy). Latency assertion joins the CI-regressed sim-bench.
|
||||
|
||||
### 5.2 App-level WS pings (hygiene, unconditional)
|
||||
|
||||
Engine-originated ping/pong on the trunk media-stream WS (and any future long-lived WS surface;
|
||||
WebRTC signaling is plain HTTP today, so the media stream is the only such socket), interval
|
||||
configurable (`RUTSTER_WS_PING_SECS`, default 20). Belt-and-braces against every idle timer
|
||||
between the FOB and the CPaaS (invariant 3).
|
||||
|
||||
### 5.3 Trunk config fixes + trusted-proxy posture
|
||||
|
||||
- TwiML Stream URL derived from `RUTSTER_TWILIO_WEBHOOK_BASE` (kills the hardcoded loopback at
|
||||
`routes.rs:258`).
|
||||
- `config::twilio_credentials` wired into startup validation (parsed-but-never-called today).
|
||||
- Trusted-proxy: `X-Forwarded-Proto/Host` honored **only** from the configured edge
|
||||
(`RUTSTER_TRUSTED_PROXIES`, CIDR list; empty = headers ignored); signature validation
|
||||
reconstructs the public URL from them (invariant 5).
|
||||
|
||||
### 5.4 rustls Phase 1 — BYO-cert in-process TLS
|
||||
|
||||
`axum-server` + rustls (already in-tree transitively): `RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY`
|
||||
enable a TLS listener; hot-reload on file change without dropping live WS (arc-swap resolver).
|
||||
Always compiled, runtime-gated; plaintext `:8080` behind Caddy remains the artifact default.
|
||||
|
||||
### 5.5 `/metrics` — hand-rolled Prometheus text endpoint
|
||||
|
||||
Reads the *existing* atomics: active sessions, `tick_overruns`, `last_tick_micros`, admission
|
||||
rejects, EventSink drops (§5.6). Zero new deps; node-level cardinality only. Own internal bind
|
||||
`RUTSTER_METRICS_BIND` (default `127.0.0.1:9090`) — never routed through Caddy.
|
||||
|
||||
### 5.6 `ValkeyEventSink` — first Valkey consumer
|
||||
|
||||
Implements the existing `EventSink` trait: `XADD` of current lifecycle events as JSON to a capped
|
||||
stream (`XADD MAXLEN ~`), strictly fire-and-forget — Valkey down ⇒ count-and-drop (counter
|
||||
surfaces in `/metrics`), never blocks the call path. `RUTSTER_VALKEY_URL` unset ⇒ today's
|
||||
`TracingEventSink`. Client crate: redis-rs (BSD-3; verify against deny.toml in the slice).
|
||||
Closes the "OOM-killed node erases all evidence of its calls" gap. **Evidence preservation, not
|
||||
the billing ledger** (ADR-0005 rule restated in code docs).
|
||||
|
||||
### 5.7 Env additions (all follow the `config.rs` fail-fast parser pattern)
|
||||
|
||||
| Var | Default | Used by |
|
||||
|---|---|---|
|
||||
| `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY` | unset (plaintext) | §5.4 |
|
||||
| `RUTSTER_TRUSTED_PROXIES` | empty (ignore forwarded headers) | §5.3 |
|
||||
| `RUTSTER_WS_PING_SECS` | 20 | §5.2 |
|
||||
| `RUTSTER_METRICS_BIND` | 127.0.0.1:9090 | §5.5 |
|
||||
| `RUTSTER_VALKEY_URL` | unset (tracing sink) | §5.6 |
|
||||
| `RUTSTER_DRAIN_DEADLINE_SECS` | unchanged code default 0; **600 in `.env.example`** | §8 |
|
||||
|
||||
---
|
||||
|
||||
## 6. Artifacts
|
||||
|
||||
### 6.1 Images (four first-party, published to the git.adlee.work registry)
|
||||
|
||||
| Image | Contents | Base |
|
||||
|---|---|---|
|
||||
| `rutster-allinone` | s6-overlay: Caddy + FOB + brain + valkey-server | debian-slim + `libopus0` |
|
||||
| `rutster-engine` | FOB only | debian-slim + `libopus0` |
|
||||
| `rutster-brain` | brain only | debian-slim |
|
||||
| `rutster-edge` | custom xcaddy build (§3.2) | scratch/alpine |
|
||||
|
||||
Multi-stage `deploy/Dockerfile`, pinned 1.85 toolchain per `rust-toolchain.toml`. Distro
|
||||
`libopus0` over static-bundled (matches dev; static noted as future size work). linux/amd64
|
||||
first; arm64 deferred. Bundled-binary licenses are aggregation-clean vs GPL-3 (Caddy Apache-2.0,
|
||||
valkey BSD-3, s6 ISC) — noted for the cargo-deny-adjacent image license scan in CI.
|
||||
|
||||
### 6.2 `deploy/` tree
|
||||
|
||||
`Dockerfile`, `compose.yaml`, `Caddyfile`, `.env.example` (documented defaults incl. drain 600 s),
|
||||
plus the smoke-test harness consumed by CI (§9).
|
||||
|
||||
### 6.3 Publish CI
|
||||
|
||||
Gitea Actions job on tag push: build all four images, push to the registry. Runs after (and
|
||||
gated on) the existing fmt/clippy/test/deny/sim-bench pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 7. Data flow (T1/T2)
|
||||
|
||||
**PSTN:** Twilio → HTTPS webhook → Caddy → FOB `/v1/trunk/webhook` (signature validated against
|
||||
URL reconstructed from trusted `X-Forwarded-*`) → TwiML with derived wss URL → Twilio dials
|
||||
media stream → Caddy (unbuffered WS tunnel) → FOB WS upgrade → `MediaCmd::RegisterTrunk` →
|
||||
media-thread tick (G.711 ↔ PCM) ↔ brain tap (loopback). **WebRTC:** signaling via Caddy; SDP
|
||||
answer advertises media IP; DTLS-SRTP UDP direct to FOB. **Events:** EventSink → Valkey stream
|
||||
(loopback T1 / compose network T2). **Metrics:** Prometheus scrapes the internal bind.
|
||||
**T3 flow** as §4.
|
||||
|
||||
---
|
||||
|
||||
## 8. Failure modes (worst first)
|
||||
|
||||
| Failure | Handling |
|
||||
|---|---|
|
||||
| Caddy `/data` loss | Persistent volume mandatory; LE duplicate-cert lockout documented as total-inbound-outage class |
|
||||
| Caddy reload vs live calls | `stream_close_delay` > max call duration **+ CI reload-during-call e2e** (§9); documented reload procedure |
|
||||
| Valkey down | EventSink counts-and-drops; calls unaffected. T3: degraded self-placement (§4) |
|
||||
| Node/FOB death | Calls on it die (non-migratable — stated honestly); s6 restarts in T1; Valkey stream preserves evidence; T3 presence TTL stops new placements |
|
||||
| Brain death | FOB reflexes continue (the FOB point); amnesiac reconnect documented; resume tokens are step 6 |
|
||||
| Drain vs orchestrator | `.env.example` drain 600 s; compose `stop_grace_period` 660 s (grace > drain); systemd `TimeoutStopSec`/k8s equivalents in docs |
|
||||
| Quiet WS killed by any middle timer | §5.2 pings + tuned edge timeouts |
|
||||
|
||||
---
|
||||
|
||||
## 9. Testing
|
||||
|
||||
- **Code:** NODELAY latency assertion in sim-bench; webhook URL derivation; signature validation
|
||||
behind forwarded headers (trusted and untrusted source cases); WS ping emission; TLS hot-reload
|
||||
(self-signed test certs); `ValkeyEventSink` XADD + drop-on-down (real valkey service container
|
||||
in CI).
|
||||
- **Artifacts (new CI muscle):** build all images; **all-in-one smoke** — boots, Caddy terminates
|
||||
TLS via its internal CA (no ACME in CI), full sim call through the real edge→FOB WS path;
|
||||
**compose smoke** — all services healthy, a lifecycle event lands in the Valkey stream;
|
||||
**Caddy config reload during a live simulated call, asserting zero drops.**
|
||||
- Existing seam-gates (loop_driver/rtc_session blob hashes), MSRV matrix, cargo-deny (now
|
||||
covering redis-rs), sim-bench single-threaded — all unchanged and must stay green.
|
||||
|
||||
## 10. Documentation (`docs/deploy/`)
|
||||
|
||||
`topologies.md` (three shapes + decision guide), `quickstart-docker.md`, `homelab.md` (CGNAT
|
||||
truth: PSTN-only behind tunnels; ngrok dev path; VPS+WireGuard graduation), `aws.md` (ALB
|
||||
appendix, three mandatory overrides, never-NLB-TLS), `reverse-proxies.md` (tuned-timeout snippets
|
||||
for BYO nginx/HAProxy/Traefik incl. the universal 60 s idle-timeout footgun), `certificates.md`
|
||||
(ACME paths, wildcard traps, duplicate-cert lockout).
|
||||
|
||||
**Acceptance bar: every topology doc contains a copy-paste path from zero to first call** — or
|
||||
the honest statement of what cannot work there (e.g. CGNAT WebRTC).
|
||||
|
||||
## 11. Sequencing — slices
|
||||
|
||||
| Slice | Contents | Depends |
|
||||
|---|---|---|
|
||||
| **A** | Hygiene: §5.1 NODELAY + §5.2 pings + §5.3 trunk fixes/trusted-proxy | — |
|
||||
| **B** | Images + compose + Caddyfile + smoke CI (§6, §9) | A |
|
||||
| **C** | rustls Phase 1 (§5.4) | A |
|
||||
| **D** | `/metrics` (§5.5) | — |
|
||||
| **E** | `ValkeyEventSink` (§5.6) | — |
|
||||
| **F** | Publish CI (§6.3) | B |
|
||||
| **G** | `docs/deploy/` tree (§10) + ADR-0011 drafting/ratification | B for accuracy |
|
||||
|
||||
C–F order-independent after their deps. Additive to the ADR-0010 sequence (benchmark →
|
||||
escalation → trunk → spend) — not a reordering. The artifacts ship with WebRTC calling live now;
|
||||
PSTN lights up end-to-end the moment the remaining trunk work lands (trunk registration is a
|
||||
placeholder NACK today; the container wiring will be ready).
|
||||
|
||||
## 12. Open questions deliberately left open (tracked, not blocking)
|
||||
|
||||
dns-persist-01 GA timeline (gates rustls Phase 2 *and* dissolves the xcaddy plugin burden);
|
||||
object-storage CDR shape; multi-tenancy ADR (before CDR/config schemas ossify); brain fleet +
|
||||
tap auth/resume (step 6); media-thread sharding graduation; SBC layer-2 wire contract;
|
||||
the real admission-cap number from the ADR-0010 benchmark.
|
||||
115
docs/superpowers/specs/2026-07-05-tls-edge-decision-brief.md
Normal file
115
docs/superpowers/specs/2026-07-05-tls-edge-decision-brief.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# Rutster TLS/Edge Decision Brief
|
||||
|
||||
Synthesized from six research reports (Caddy, classic proxies, in-process rustls, tunnels, cloud-native LB, CPaaS ground truth). 2026-07-05.
|
||||
|
||||
---
|
||||
|
||||
## 1. Decision Matrix
|
||||
|
||||
Scores 1–5. The ground-truth report is excluded from the matrix — it is the spec, not a candidate (its scores measure how *demanding* the CPaaS side is, per its own scoring note).
|
||||
|
||||
| Criterion | Caddy | nginx | HAProxy | Traefik | In-process rustls | Tunnels (CF/Funnel/ngrok) | AWS ALB+ACM |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| WS media fitness | 4 | 5* | 5* | 5* | 5 | 2 | 3.5 |
|
||||
| Cert automation | 5 | 3 | 4 | 5 | 3 | 5 | 4 |
|
||||
| All-in-one fit | 5 | 4* | 4* | 4* | 5 | 3 | 1 |
|
||||
| Compose fit | 5 | 5* | 5* | 5* | 4 | 4 | 2 |
|
||||
| Fleet fit | 4 | 4* | 4* | 4* | 4 | 3 | 4 |
|
||||
| No-public-IP story | 3 | 2 | 2 | 2 | 3 | 5 | 1 |
|
||||
| Ops burden | 4 | 4* | 4* | 4* | 3 | 3 | 4 |
|
||||
| **Third party in call path** | **No** | **No** | **No** | **No** | **No** | **Yes** (CF/ngrok see *plaintext* audio; Funnel ciphertext-only) | **Yes** (trust-neutral when engine is already on EC2) |
|
||||
|
||||
\* The classic-proxy report scored the family jointly (5/5/4/5/4/2/4); per-member sub-rankings from its findings: cert automation Traefik 5, HAProxy 4, nginx 3; HAProxy best WS mental model; Traefik has a patch-release WS-regression record.
|
||||
|
||||
---
|
||||
|
||||
## 2. Hard CPaaS Invariants (any answer must satisfy — ground-truth report)
|
||||
|
||||
1. **Publicly-trusted CA cert, no self-signed path exists.** Twilio refuses self-signed for webhooks and Media Streams (error 31910); the console validation toggle is account-wide, webhook-only, dev-only. Auto-renewing Let's Encrypt-class certs are availability-critical, not a nicety. ([twilio.com/docs/usage/security](https://www.twilio.com/docs/usage/security), [errors/31910](https://www.twilio.com/docs/api/errors/31910))
|
||||
2. **wss:// only, on 443, open to the whole internet.** Twilio publishes no egress IPs ("any public IP address") and no CPaaS surveyed offers mTLS. Auth is application-layer only. IP allowlisting exists for Telnyx/Vonage/Plivo as an optional hardening layer — never for Twilio.
|
||||
3. **One WS connection per call, up to 4h default / 24h configured, 50 msg/s/direction at 20ms cadence.** Zero tolerance for frame buffering, connection-lifetime caps, or write-side idle timers — the unidirectional case may legitimately send zero bytes from your side for hours, and neither Twilio nor Telnyx documents any WS ping behavior. Keepalive is entirely the edge's job.
|
||||
4. **`<Stream>` URLs carry no query strings** — wss routing must key on path/hostname.
|
||||
5. **X-Twilio-Signature is HMAC over the URL as Twilio saw it.** Any TLS-terminating edge must forward X-Forwarded-Proto/Host untouched (including on the WS upgrade request) and the app must reconstruct the public URL, or genuine traffic fails validation. ([Twilio SSL-termination blog](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library))
|
||||
6. **Webhook budget: sub-second for UX, 15s hard platform cap.** No blocking operations (e.g. on-demand ACME issuance) may sit in the handshake/webhook path.
|
||||
7. **TLS 1.2+1.3, mainstream ECDHE suites, never 1.3-only** (Twilio's 1.3 client support is undocumented); never pin Twilio certs.
|
||||
8. **Fleet corollary:** node-addressed URLs force a publicly-valid cert+DNS name *per node* — wildcard DNS-01 or per-node ACME is imposed by the CPaaS, not the proxy choice.
|
||||
9. **CGNAT corollary:** the CPaaS offers no relay/rendezvous; public reachability must be imported from the operator's side, and intermediaries are a documented frame-dropping hazard ([livekit/agents#3379](https://github.com/livekit/agents/issues/3379)).
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommendation (per topology)
|
||||
|
||||
**Headline: Caddy is the shared edge for both shipped artifacts. In-process rustls is the ratified end-state, deferred — ship Phase 1 (BYO-cert) opportunistically, gate Phase 2 (in-binary ACME) on explicit triggers. Tunnels are dev-only.**
|
||||
|
||||
### (a) All-in-one container: bundle Caddy under s6
|
||||
|
||||
A single 47MB static Apache-2.0 binary, ~6-line Caddyfile, dual-CA ACME with background issuance and **zero-downtime in-memory cert renewal that never touches live calls** — the routine periodic event drops nothing (Caddy report). Post-upgrade WS handling is verified protocol-unaware `io.CopyBuffer` tunneling — safe for 20ms PCM. Requirements baked into the image:
|
||||
|
||||
- Custom xcaddy build with a curated DNS-provider plugin set (cloudflare Apache-2.0; route53/porkbun/hetzner/desec MIT; **skip duckdns** — no license file). The official image ships zero DNS plugins; without this, wildcard/behind-NAT issuance is impossible.
|
||||
- `/data` on a persistent volume, non-negotiable — recreate loops otherwise hit LE's 5-certs/week duplicate limit and lock the domain out.
|
||||
- `stream_close_delay` set above max call duration, **and** a CI e2e case "config reload during active simulated call" — the mitigation has an open bug trail (caddy [#6420](https://github.com/caddyserver/caddy/issues/6420), [#7222](https://github.com/caddyserver/caddy/issues/7222)) and must not be trusted untested.
|
||||
- Tuned timeouts + honest X-Forwarded-* forwarding per invariants 3 and 5.
|
||||
|
||||
### (b) Modular compose: same Caddy build, as a compose service
|
||||
|
||||
Same image, same Caddyfile, same CI coverage — one edge choice across both artifacts, as preferred. Compose additionally documents "bring your own proxy": the engine keeps its plaintext `:8080` mode, and the docs ship the tuned-timeout snippets for nginx/HAProxy/Traefik (the 60s idle-timeout defaults in all three are the universal footgun — classic-proxy report).
|
||||
|
||||
### (c) Fleet chapter of the ADR
|
||||
|
||||
- **Per-node public TLS names are CPaaS-imposed** (invariant 8). Two blessed patterns: **(1) wildcard `*.pbx.domain` via DNS-01** + Caddy's documented shared-storage cluster coordination, or issue-once-and-distribute; **(2) per-node distinct certs** (`node-N.domain`) where you don't want the wildcard private key on every node — one compromised node otherwise burns the whole namespace (rustls report). Renewals are exempt from LE per-domain limits, so per-node scales.
|
||||
- **Explicitly reject Caddy on-demand TLS for node names:** the first handshake to a fresh name blocks seconds on issuance, colliding with sub-second webhook budgets; the rate-limit knobs are deprecated. Node names known at provision time get pre-issued certs.
|
||||
- **Trap to document:** N nodes independently requesting the *identical* wildcard hits the 5/week duplicate-cert limit — distinct names or central distribution only.
|
||||
- **AWS appendix (docs + Terraform, never a shipped artifact):** ALB + wildcard ACM + host-header rules for operators already on EC2 — trust-neutral there, ACM renewal genuinely zero-touch, WebRTC UDP rides beside it via per-node EIPs. Mandatory attribute overrides: `idle_timeout=4000`, `client_keep_alive=604800` (the 3600s default can kill the >1h call), header forwarding. Hard 100-target-group cap per ALB; AWS fleet rotation can still cut a multi-hour call. **Never NLB-TLS** (fixed 350s silent drop, no HTTP context).
|
||||
|
||||
### (d) Homelab / no-public-IP
|
||||
|
||||
Tell the truth: **nothing makes CGNAT production-grade for free, and no tunnel carries inbound UDP — WebRTC callers are unreachable behind any tunnel, so homelab-behind-tunnel is PSTN-only, period** (tunnel report). The story, in three tiers:
|
||||
|
||||
1. **Dev/demo (blessed):** `ngrok http 8080` — the only tunnel with a proven Twilio Media Streams record; cloudflared has an *open* Twilio 31920 handshake bug ([cloudflared#1465](https://github.com/cloudflare/cloudflared/issues/1465)) plus recurring 1006 closures (#1282), and Cloudflare documents killing WS on edge code releases. Free-tier arithmetic: ~7 call-hours/month. Tailscale Funnel is the privacy-clean single-user variant (TLS terminates on-node) with an undisclosed bandwidth cap.
|
||||
2. **Production graduation path (recommended):** cheap VPS + WireGuard as a **dumb TCP forwarder to the home box, with TLS terminating at home** (Caddy via DNS-01, or later in-process rustls). The forwarder physically cannot read audio — the strongest privacy topology of any candidate (rustls report). Or simply: run the engine on the VPS.
|
||||
3. **Explicitly unsupported for production:** Cloudflare Tunnel / ngrok in the live audio path — plaintext audio on the vendor's edge (unconsented subprocessor; DPA/BAA/PCI failure), documented mid-call WS terminations, zero SLA, and Cloudflare's discretionary "disproportionate audio" ToS clause aimed at exactly this traffic profile.
|
||||
|
||||
### In-process rustls: reserved as the deliberate next step, not adopted now — and not rejected
|
||||
|
||||
**Engage ADR-0008 honestly: the keep-the-FOB-minimal objection is factually dead in this codebase.** ADR-0008 explicitly lists the trust-domain edge as security-constitutive FOB territory, and str0m already terminates DTLS-SRTP in-process against hostile UDP on the same aws-lc-rs provider that rustls would use — rustls 0.23.41/tokio-rustls 0.26.4 are *already in Cargo.lock* (rustls report, citing `docs/adr/0008` and the lockfile). Doctrinally this is the intended end-state, and it is the only candidate with truly zero-downtime renewal (arc-swap) plus structural absence of the entire proxy-timeout/buffering failure class.
|
||||
|
||||
The reason to defer is engineering economics, not doctrine: **no turnkey ACME crate fits this dependency tree** (rustls-acme hard-violates deny.toml on rcgen/x509-parser and lacks DNS-01; tokio-rustls-acme demands reqwest 0.13 + a second C crypto lib), so the path is a 1–2 week instant-acme integration after which **rutster owns a renewal state machine forever in a 45-day-cert-lifetime world** (~8 chances/year/cert for a DIY loop bug to cause total inbound outage — invariant 1 makes that the worst failure available). Caddy buys a decade-hardened ACME loop for one supervised process.
|
||||
|
||||
- **Do now (cheap, 1–2 days):** Phase 1 — `axum-server` BYO-cert TLS with hot-reload, plus **TCP_NODELAY on accepted sockets** (axum [#2521](https://github.com/tokio-rs/axum/issues/2521)/#1961 — this latent frame-coalescing bug affects the plaintext `:8080` listener behind *every* proxy candidate today; fix it regardless of this decision, with a latency assertion in the e2e sim).
|
||||
- **Triggers for Phase 2 (in-binary ACME):** (1) Let's Encrypt **dns-persist-01 confirmed GA in production** — it removes DNS-provider API plumbing and closes the wildcard/CGNAT issuance gap, and instant-acme 0.8.5 already parses it; (2) a field incident where Caddy reload/`stream_close_delay` drops live calls; (3) the fleet or VPS-forwarder topology making end-to-end-encrypted-to-the-binary a selling point; (4) fatigue with maintaining the curated xcaddy DNS-plugin build. Until then, DoS isolation (handshake floods landing on the media-tick runtime) remains a real cost the proxy absorbs for free.
|
||||
|
||||
---
|
||||
|
||||
## 4. Runner-ups vs unsupported
|
||||
|
||||
**Supported, documented alternatives**
|
||||
|
||||
| Configuration | Where | Caveats to document |
|
||||
|---|---|---|
|
||||
| HAProxy 3.2 LTS/3.3 | Cloud-VM operators wanting zero-reload cert hot-load (acme.sh stats-socket deploy) and `timeout tunnel` semantics | Set `timeout tunnel` in `defaults` (issue #2280); native ACME still experimental; `hard-stop-after` kills drained calls |
|
||||
| Traefik v3.x | Compose users already in the Traefik ecosystem; best built-in DNS-01/wildcard of the classic family | **Pin exact versions** — three WS-breaking patch releases (#10601, v2.11.2 timeout flip, #11405); never auto-update the edge |
|
||||
| ALB + ACM (Terraform module) | EC2-resident fleets | The three attribute overrides; 100-TG cap; AWS rotation can cut long calls |
|
||||
| axum-server BYO-cert (Phase 1) | Operators who already have cert distribution (e.g. corporate ACME, wildcard distribution) | Operator owns renewal delivery |
|
||||
| ngrok / Tailscale Funnel | Dev/demo/quickstart only | Free-tier caps; PSTN-only (no UDP); never production |
|
||||
|
||||
**Explicitly unsupported**
|
||||
|
||||
- Cloudflare Tunnel or ngrok in the production audio path (plaintext audio at vendor edge, documented mid-call kills, ToS exposure).
|
||||
- cloudflared for Media Streams even in dev (open Twilio 31920 bug).
|
||||
- NLB TLS listener (fixed 350s silent drop; no headers for signature validation).
|
||||
- Caddy on-demand TLS for fleet node names (blocking first handshake vs webhook SLA).
|
||||
- nginx as a recommended edge (no native DNS-01/wildcard, reload-per-renewal vs hours-long calls, 60s footgun) — works if the operator insists, but rutster won't ship config for it beyond the timeout snippet.
|
||||
- Self-signed certs anywhere, including via the Twilio console toggle (account-wide blast radius, doesn't help Media Streams).
|
||||
|
||||
---
|
||||
|
||||
## 5. Open risks surviving the recommendation
|
||||
|
||||
1. **Caddy config-reload kills active WS tunnels; the mitigation is unproven.** `stream_close_delay` has contested/open bugs (#6420, #7222). Backing failure mode: operator edits Caddyfile mid-call → every live Media Streams tunnel closes → all calls drop. Residual even after the mandated e2e test — the test can only tell us *whether* we're exposed per version.
|
||||
2. **Curated DNS-plugin matrix is a permanent maintenance surface.** Any user on an unbundled provider needs a custom xcaddy build, contradicting "docker run and it takes calls" for the behind-NAT wildcard path. dns-persist-01 eventually dissolves this; until GA it stands.
|
||||
3. **dns-persist-01 timeline risk cuts both ways.** Production target was Q2 2026 (i.e., now); if it slips, both the Caddy-plugin burden persists *and* the rustls Phase 2 trigger recedes, leaving classic DNS-01 credentials-in-container as the homelab wildcard story.
|
||||
4. **Quiet signaling WS remains structurally fragile everywhere.** The 20ms media leg self-protects; the WebRTC signaling WS and any hold/parked state can idle past *some* timer in every topology (Caddy needs app pings; ALB hard-caps at 4000s). App-level WS ping/pong in the engine is required belt-and-braces, per invariant 3.
|
||||
5. **Cert-storage loss → LE duplicate-cert lockout → total inbound outage** (31910 on streams, 11237 on webhooks) with no maintenance window in a 24/7 non-migratable-call product. The persistent-volume requirement mitigates but volume misconfiguration in the field will happen.
|
||||
6. **Homelab WebRTC gap is unsolved by design.** UDP-direct advertised-IP media cannot cross CGNAT via any recommended path; only the VPS graduation (or engine-on-VPS) fixes it. Operators will read "homelab supported" as including browser calls unless the docs say PSTN-only loudly.
|
||||
7. **Misdiagnosis tax.** Field reports of "WS drops through Caddy" are usually path/upstream issues (#6958 closed invalid); with Twilio's unpublished egress IPs, "is it Caddy, the path, or Twilio" debugging has no allowlist shortcut (invariant 2).
|
||||
8. **If/when Phase 2 fires: rutster owns a renewal state machine in a 90→64→45-day world.** The risk merely moves from "Caddy reload behavior" to "our ARI scheduler has a bug", plus the rcgen 0.14.7 MSRV pin gains a second dependent (Cargo.toml pin rationale, rustls report).
|
||||
Reference in New Issue
Block a user