deploy slice G (docs + ADR-0011): docs/deploy/ tree + ADR-0011 Proposed (#23)

Co-authored-by: Aaron D. Lee <himself@adlee.work> Co-committed-by: Aaron D. Lee <himself@adlee.work>
2026-07-06 04:07:15 +00:00
parent 7e2ae6d2d2
commit 0c307acdb5
8 changed files with 912 additions and 0 deletions
--- a/docs/deploy/aws.md
+++ b/docs/deploy/aws.md
@@ -0,0 +1,99 @@
+# AWS appendix — ALB + ACM (docs only, never a shipped artifact)
+
+Scope: operators **already on EC2**. There, an ALB is trust-neutral (the engine is on AWS
+infrastructure regardless) and ACM renewal is genuinely zero-touch. This page is an appendix,
+not a recommendation to move to AWS — the shipped edges are Caddy
+([topologies.md](topologies.md)) and BYO proxies ([reverse-proxies.md](reverse-proxies.md)).
+All ALB/NLB facts below are grounded in the [TLS/edge decision brief §3(c)](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md).
+
+## Never NLB-TLS
+
+Do not put the engine behind an NLB TLS listener. Two disqualifiers, ratified in
+[ADR-0011](../adr/0011-deployment-topology.md):
+
+1. **Fixed 350-second idle timeout** that silently drops flows — any quiet WebSocket (hold,
+   parked call, unidirectional stream) dies mid-call with no close frame.
+2. **No HTTP context**: an NLB adds no `X-Forwarded-Proto/Host`, so the engine cannot
+   reconstruct the public URL that `X-Twilio-Signature` is HMAC'd over
+   ([Twilio's SSL-termination guidance](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library))
+   — genuine traffic fails validation.
+
+## The three mandatory ALB attribute overrides
+
+The defaults kill telephony. All three overrides are **mandatory**, not tuning:
+
+| Attribute | Value | Why |
+|---|---|---|
+| `idle_timeout.timeout_seconds` | `4000` | The default 60 s kills any quiet WS. 4000 is the ALB maximum — still finite, so the engine's app-level WS pings (`RUTSTER_WS_PING_SECS`, default 20) remain load-bearing here. |
+| `client_keep_alive.seconds` | `604800` | The 3600 s default can terminate the client connection under a **>1-hour call**. Set to the 7-day maximum. |
+| `routing.http.preserve_host_header.enabled` | `true` | Signature validation reconstructs the URL *as Twilio saw it* — the engine must receive the honest public `Host`, and its `RUTSTER_TRUSTED_PROXIES` must include the ALB's subnet CIDRs. |
+
+## Copy-paste path (CLI)
+
+Assumes: engine node(s) running per [quickstart-docker.md](quickstart-docker.md) with the
+`caddy` service disabled (plaintext `:8080`), a VPC, two subnets, and a security group
+allowing `:443` from anywhere and `:8080` from the ALB.
+
+```bash
+VPC=vpc-0123456789abcdef0
+SUBNETS="subnet-0aaa subnet-0bbb"
+SG=sg-0ccc
+
+ALB_ARN=$(aws elbv2 create-load-balancer --name rutster-edge --type application \
+  --subnets $SUBNETS --security-groups $SG \
+  --query 'LoadBalancers[0].LoadBalancerArn' --output text)
+
+# THE THREE MANDATORY OVERRIDES — do not skip:
+aws elbv2 modify-load-balancer-attributes --load-balancer-arn "$ALB_ARN" --attributes \
+  Key=idle_timeout.timeout_seconds,Value=4000 \
+  Key=client_keep_alive.seconds,Value=604800 \
+  Key=routing.http.preserve_host_header.enabled,Value=true
+
+TG_ARN=$(aws elbv2 create-target-group --name rutster-node-1 \
+  --protocol HTTP --port 8080 --vpc-id "$VPC" \
+  --health-check-path /readyz \
+  --query 'TargetGroups[0].TargetGroupArn' --output text)
+aws elbv2 register-targets --target-group-arn "$TG_ARN" --targets Id=i-0node1
+
+CERT_ARN=$(aws acm request-certificate --domain-name '*.pbx.example.com' \
+  --validation-method DNS --query CertificateArn --output text)
+# Create the DNS validation CNAME ACM reports, wait for ISSUED, then:
+
+LISTENER_ARN=$(aws elbv2 create-listener --load-balancer-arn "$ALB_ARN" \
+  --protocol HTTPS --port 443 \
+  --ssl-policy ELBSecurityPolicy-TLS13-1-2-2021-06 \
+  --certificates CertificateArn="$CERT_ARN" \
+  --default-actions Type=forward,TargetGroupArn="$TG_ARN" \
+  --query 'Listeners[0].ListenerArn' --output text)
+
+# Fleet growth: one target group + one host-header rule per node
+# (node-addressed placement — ADR-0011):
+aws elbv2 create-rule --listener-arn "$LISTENER_ARN" --priority 10 \
+  --conditions Field=host-header,Values=node-1.pbx.example.com \
+  --actions Type=forward,TargetGroupArn="$TG_ARN"
+```
+
+Point `*.pbx.example.com` (alias record) at the ALB. First call: identical to
+[quickstart-docker.md](quickstart-docker.md) — browser at your domain, Twilio webhook at
+`https://pbx.example.com/v1/trunk/webhook`.
+
+The `--ssl-policy` above serves TLS 1.2+1.3 — never pick a 1.3-only policy (Twilio's TLS 1.3
+client support is undocumented; see the invariants in
+[certificates.md](certificates.md)).
+
+## WebRTC media does not ride the ALB
+
+The ALB carries HTTPS/WSS only. WebRTC media UDP goes **direct to each node**: give every
+engine node an Elastic IP, open the media UDP range in the node's security group, and set
+`RUTSTER_MEDIA_ADVERTISED_IP` to that EIP.
+
+## Caps and traps
+
+- **Hard cap: 100 target groups per ALB.** With one TG per node (host-header routing), that
+  is your fleet ceiling per ALB.
+- **AWS fleet rotation can still cut a multi-hour call** — ALB infrastructure is replaced
+  under maintenance without regard for your WS lifetimes. Accepted residual risk; there is no
+  attribute for it.
+- Health checks target `/readyz` (503 while draining or at the admission cap) — scale-in must
+  be drain-then-terminate: deregister the target, wait out
+  `RUTSTER_DRAIN_DEADLINE_SECS`, then terminate the instance.
--- a/docs/deploy/certificates.md
+++ b/docs/deploy/certificates.md
@@ -0,0 +1,79 @@
+# Certificates & ACME — the availability-critical subsystem
+
+**A publicly-CA-trusted certificate is a hard CPaaS requirement; no self-signed path exists.**
+Twilio refuses self-signed certs for webhooks and Media Streams (error
+[31910](https://www.twilio.com/docs/api/errors/31910); see
+[twilio.com/docs/usage/security](https://www.twilio.com/docs/usage/security)) — and the
+console's cert-validation toggle is account-wide, webhook-only, and dev-only, so it is not an
+escape hatch. In a 24/7 product whose calls are non-migratable there is no maintenance window:
+**auto-renewal is availability-critical, not a nicety.**
+
+Protocol posture everywhere: TLS 1.2+1.3, mainstream ECDHE suites, **never 1.3-only**
+(Twilio's TLS 1.3 client support is undocumented), and never pin Twilio's certificates.
+
+## The `/data` volume — read this before anything else
+
+Caddy keeps its certificates and ACME account state in `/data`. If you recreate the container
+without that volume, Caddy re-requests the same certificate — and Let's Encrypt's
+**duplicate-certificate limit (5 per week for an identical name set,
+[letsencrypt.org/docs/rate-limits](https://letsencrypt.org/docs/rate-limits/))** will lock
+your domain out for up to a week. The failure class is **total inbound outage**: 31910 on
+Media Streams, [11237](https://www.twilio.com/docs/api/errors/11237) on webhooks, no calls in
+or out of the trunk. A container recreate loop (crash-looping deploy, CI that recreates on
+every push) burns all five in minutes.
+
+- Always mount `/data` on a persistent volume (`-v rutster-caddy-data:/data`).
+- Back it up like state, because it is:
+  `docker run --rm -v rutster-caddy-data:/data -v "$PWD":/backup debian:stable-slim tar czf /backup/caddy-data.tgz /data`
+
+## ACME challenge paths
+
+| Path | When | Notes |
+|---|---|---|
+| **HTTP-01** (default) | Port 80 publicly reachable, single hostname | Zero config beyond the domain + email. |
+| **DNS-01** | Wildcards (`*.pbx.domain`), CGNAT/behind-NAT hosts, no port 80 | The bundled `rutster-edge` Caddy build ships a curated plugin set: **cloudflare, route53, porkbun, hetzner, desec** (duckdns excluded — no license file). Any other DNS provider requires your own xcaddy build. DNS API credentials live in the container env — scope them to the zone. |
+| **BYO-cert** (in-process rustls, Phase 1) | You already have cert distribution (corporate ACME, central wildcard issuance) | Set `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY`; the engine serves TLS itself and **hot-reloads on file change without dropping live WS**. You own renewal delivery. |
+
+In-binary ACME (Phase 2) is deliberately deferred behind named triggers — among them Let's
+Encrypt's dns-persist-01 reaching GA, which would dissolve the DNS-plugin matrix entirely. See
+[ADR-0011](../adr/0011-deployment-topology.md).
+
+## Caddy reload vs live calls
+
+Routine renewals are safe: Caddy swaps certs **in memory, zero-downtime — the routine periodic
+event drops nothing.** Config *reloads* (editing the Caddyfile) are the risk: the mitigation
+(`stream_close_delay` above max call duration) has an open upstream bug trail
+([caddy#6420](https://github.com/caddyserver/caddy/issues/6420),
+[caddy#7222](https://github.com/caddyserver/caddy/issues/7222)), which is why the shipped
+artifacts carry a CI e2e test for config-reload-during-live-call. Operator rule: **do not edit
+the Caddyfile while calls are live** unless your image version passed that test; drain first
+(`/readyz` returns 503 while draining).
+
+## Fleet certificate patterns (T3)
+
+Node-addressed placement means every node needs its own publicly-valid TLS name — this is
+CPaaS-imposed (`<Stream>` URLs route on hostname only), not a proxy choice. Two blessed
+patterns:
+
+1. **Wildcard `*.pbx.domain`** via DNS-01, issued **centrally** and distributed to nodes.
+   Understand the trade: the wildcard private key on every node means one compromised node
+   burns the whole namespace.
+2. **Per-node distinct certs** (`node-1.pbx.domain`, `node-2.…`) — no shared key material.
+   Renewals are exempt from Let's Encrypt's per-domain limits, so this scales with fleet size.
+
+Traps:
+
+- **N nodes independently requesting the identical wildcard** are N duplicate requests — the
+  5/week duplicate-cert limit locks the fleet out. Central issuance or distinct names only.
+- **Caddy on-demand TLS is rejected for node names**: the first TLS handshake to a fresh name
+  blocks for seconds on issuance, colliding with the sub-second webhook budget (Twilio's hard
+  cap is 15 s, UX budget sub-second), and its rate-limit knobs are deprecated. Node names are
+  known at provision time — pre-issue their certs.
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| Twilio 31910 on Media Streams | Cert not publicly trusted / expired / SNI mismatch. Check `openssl s_client -connect pbx.example.com:443 -servername pbx.example.com`. |
+| `rateLimited` / `too many certificates` in Caddy logs | You hit the duplicate-cert limit — almost always a lost `/data` volume. Restore the volume or wait out the window; then fix the volume mount so it never recurs. |
+| DNS-01 fails for your provider | Provider not in the bundled plugin set — use a delegated zone on a supported provider, or build your own xcaddy image. |
--- a/docs/deploy/homelab.md
+++ b/docs/deploy/homelab.md
@@ -0,0 +1,132 @@
+# Homelab & CGNAT — the honest story
+
+**The truth first: no tunnel carries inbound UDP. WebRTC callers are unreachable behind any
+ tunnel, so a homelab behind CGNAT is PSTN-only, period.** The engine's WebRTC media is
+UDP-direct to an advertised IP (no STUN, no TURN); the CPaaS side offers no relay or
+rendezvous either, and intermediaries in the media path are a documented frame-dropping hazard
+([livekit/agents#3379](https://github.com/livekit/agents/issues/3379)). Nothing on this page
+makes CGNAT production-grade for free. Three tiers, worst to best:
+
+## Tier 1 — dev/demo: ngrok (the blessed 5-minute path)
+
+ngrok is the **only** tunnel with a proven Twilio Media Streams record. Do **not** use
+Cloudflare Tunnel even for dev: cloudflared has an open Twilio 31920 handshake bug
+([cloudflared#1465](https://github.com/cloudflare/cloudflared/issues/1465)), recurring 1006
+closures ([cloudflared#1282](https://github.com/cloudflare/cloudflared/issues/1282)), and
+Cloudflare documents killing WebSockets on edge code releases.
+
+```bash
+# 1. Run the engine, plaintext :8080 (no Caddy needed — ngrok terminates TLS):
+docker run -d --name rutster-engine --network host \
+  -e RUTSTER_TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
+  -e RUTSTER_TWILIO_AUTH_TOKEN=your_auth_token \
+  -e RUTSTER_TWILIO_MEDIA_BIND=0.0.0.0:8081 \
+  -e RUTSTER_TWILIO_WEBHOOK_BASE=https://REPLACE-ME.ngrok-free.app \
+  -e RUTSTER_TRUSTED_PROXIES=127.0.0.1/32 \
+  git.adlee.work/alee/rutster-engine:latest
+# (or from source: cargo run — see docs/QUICKSTART.md)
+
+# 2. Tunnel it:
+ngrok http 8080
+# 3. Put the printed https://xxxx.ngrok-free.app URL into RUTSTER_TWILIO_WEBHOOK_BASE
+#    (restart the container), and into the Twilio number's webhook:
+#    POST https://xxxx.ngrok-free.app/v1/trunk/webhook
+# 4. Dial your Twilio number.
+```
+
+**Free-tier arithmetic** (why this is a demo, not a deployment — research: [TLS brief §3(d)](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)): a Media Streams call is
+8 kHz µ-law — 8 kB/s of audio per direction, base64-encoded inside a JSON envelope at 50
+messages/s, both directions ≈ **~140 MB per call-hour** on the wire. ngrok's free 1 GB/month
+data cap ([ngrok pricing](https://ngrok.com/pricing)) therefore buys roughly **seven call-hours a month**. And the audio transits ngrok's
+edge **in plaintext** (they terminate TLS) — an unconsented subprocessor, which is a
+DPA/BAA/PCI failure. Dev only. Never production.
+
+## Tier 2 — single-user demo: Tailscale Funnel
+
+Privacy-clean variant: TLS terminates **on your node**, so Funnel relays ciphertext it cannot
+read.
+
+```bash
+tailscale funnel 8080
+# webhook base = https://<your-node>.<tailnet>.ts.net
+```
+
+Still PSTN-only (no inbound UDP), bandwidth cap undisclosed, one user. Unsizable beyond a
+personal demo.
+
+## Tier 3 — production graduation: cheap VPS + WireGuard, TLS at home
+
+The recommended path. The VPS is a **dumb layer-4 forwarder**: TLS terminates at home (your
+Caddy, DNS-01 certs), so the forwarder **physically cannot read the audio** — the strongest
+privacy topology available. Bonus: forwarding the media UDP range over the same tunnel
+restores WebRTC, which no tunnel product can do. Cost: one small VPS (~$5/mo) and one extra
+network hop of media latency — pick a VPS region near home.
+
+Point DNS at the VPS: `pbx.example.com A <VPS-public-IP>`.
+
+**VPS `/etc/wireguard/wg0.conf`:**
+
+```ini
+[Interface]
+Address = 10.88.0.1/24
+ListenPort = 51820
+PrivateKey = <vps-private-key>
+
+[Peer]
+PublicKey = <home-public-key>
+AllowedIPs = 10.88.0.2/32
+```
+
+**Home box `/etc/wireguard/wg0.conf`** (home initiates — CGNAT-friendly; the keepalive holds
+the NAT mapping):
+
+```ini
+[Interface]
+Address = 10.88.0.2/24
+PrivateKey = <home-private-key>
+
+[Peer]
+PublicKey = <vps-public-key>
+Endpoint = <VPS-public-IP>:51820
+AllowedIPs = 10.88.0.0/24
+PersistentKeepalive = 25
+```
+
+**VPS forwarding** (`sysctl -w net.ipv4.ip_forward=1`, persist it, then nftables):
+
+```nft
+table ip rutster-fwd {
+  chain prerouting {
+    type nat hook prerouting priority dstnat;
+    iifname "eth0" tcp dport { 80, 443 } dnat to 10.88.0.2
+    iifname "eth0" udp dport 49152-49407 dnat to 10.88.0.2
+  }
+  chain postrouting {
+    type nat hook postrouting priority srcnat;
+    oifname "wg0" masquerade
+  }
+}
+```
+
+**Home side:** run T1 or T2 exactly per [quickstart-docker.md](quickstart-docker.md), with:
+
+- `RUTSTER_MEDIA_ADVERTISED_IP=<VPS-public-IP>` — callers send media UDP to the VPS; the DNAT
+  delivers it home through the tunnel.
+- `RUTSTER_MEDIA_PORT_RANGE=49152-49407` (must match the nftables rule).
+- Certificates: DNS-01 is the robust choice here ([certificates.md](certificates.md));
+  HTTP-01 also works since `:80` is forwarded.
+
+First call: same two paths as [quickstart-docker.md](quickstart-docker.md) — browser at
+`https://pbx.example.com/` (WebRTC now works — the UDP range rides the tunnel) and the Twilio
+webhook at `https://pbx.example.com/v1/trunk/webhook`.
+
+**Or skip the tunnel entirely:** run the engine *on* the VPS (that is just
+[T1](quickstart-docker.md) on rented hardware). You trade at-home media for zero forwarding
+complexity.
+
+## Explicitly unsupported for production
+
+Cloudflare Tunnel or ngrok in the live audio path: plaintext audio at the vendor edge
+(unconsented subprocessor — DPA/BAA/PCI failure), documented mid-call WS terminations, zero
+SLA, and Cloudflare's discretionary "disproportionate audio" ToS clause aimed at exactly this
+traffic profile (research basis: [TLS brief §4](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)). Ratified in [ADR-0011](../adr/0011-deployment-topology.md).
--- a/docs/deploy/quickstart-docker.md
+++ b/docs/deploy/quickstart-docker.md
@@ -0,0 +1,104 @@
+# Docker quickstart — zero to first call
+
+Two shapes, one binary ([topologies.md](topologies.md)). T1 is one `docker run`; T2 is the
+reference `compose up`. Both end the same way: a browser call at your domain, and (with Twilio
+credentials) a real phone call.
+
+> **Artifact names note:** images are published to the `git.adlee.work/alee/` registry
+> namespace by the release CI (plan B / slice F settled this). If your Gitea registry uses a
+> different owner segment, substitute accordingly — check `deploy/compose.yaml` in your
+> checkout, which is authoritative.
+
+## Prerequisites (both shapes)
+
+1. A Linux host with Docker, a **public IPv4**, and inbound `80/tcp`, `443/tcp`, and your
+   chosen media UDP range open. (No public IP? → [homelab.md](homelab.md).)
+2. A domain with an A record pointing at that host, e.g. `pbx.example.com`. Certificates are
+   issued automatically via ACME — no self-signed path exists, the CPaaS refuses it
+   ([certificates.md](certificates.md)).
+3. For the phone call: a [Twilio](https://www.twilio.com/) account + a Voice-capable number
+   (trial works).
+
+## T1 — Solo (all-in-one)
+
+```bash
+docker run -d --name rutster --restart unless-stopped \
+  --network host \
+  -v rutster-caddy-data:/data \
+  -v rutster-valkey:/var/lib/valkey \
+  -e RUTSTER_DOMAIN=pbx.example.com \
+  -e RUTSTER_ACME_EMAIL=you@example.com \
+  -e RUTSTER_MEDIA_ADVERTISED_IP=203.0.113.7 \
+  -e RUTSTER_MEDIA_PORT_RANGE=49152-49407 \
+  -e RUTSTER_TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
+  -e RUTSTER_TWILIO_AUTH_TOKEN=your_auth_token \
+  -e RUTSTER_TWILIO_MEDIA_BIND=0.0.0.0:8081 \
+  -e RUTSTER_TWILIO_WEBHOOK_BASE=https://pbx.example.com \
+  git.adlee.work/alee/rutster-allinone:latest
+```
+
+Notes:
+
+- `--network host` is the recommended mode: WebRTC media UDP goes **direct to the engine**,
+  never through Caddy. If you can't use host networking, publish `-p 80:80 -p 443:443
+  -p 49152-49407:49152-49407/udp` instead and keep `RUTSTER_MEDIA_ADVERTISED_IP` = the host's
+  public IP (NAT 1:1, no STUN).
+- The two volumes are **non-negotiable**. `/data` in particular: recreating the container
+  without it re-requests certificates and can hit Let's Encrypt's duplicate-certificate
+  limit — a week-long total inbound outage ([certificates.md](certificates.md)).
+- The four `RUTSTER_TWILIO_*` vars are all-or-none — the engine refuses partial Twilio config
+  at startup. Omit all four to run WebRTC-only.
+- `RUTSTER_DOMAIN` / `RUTSTER_ACME_EMAIL` feed the bundled Caddyfile. Domains that can't do
+  HTTP-01 (no port 80, wildcard needs) use DNS-01 — [certificates.md](certificates.md).
+
+Verify it's up:
+
+```bash
+curl -fsS https://pbx.example.com/healthz && echo OK      # liveness
+curl -fsS https://pbx.example.com/readyz  && echo READY   # can accept a new call
+```
+
+### First call (browser, WebRTC)
+
+Open `https://pbx.example.com/`, click **Start call**, grant the microphone. You're on a call
+with the engine.
+
+### First call (phone, PSTN)
+
+1. Twilio Console → Phone Numbers → your number → **A call comes in** → Webhook,
+   `POST https://pbx.example.com/v1/trunk/webhook`.
+2. Dial your Twilio number from any phone. Twilio hits the webhook, the engine answers with
+   TwiML pointing Twilio's Media Streams at `wss://pbx.example.com/twilio/media-stream`, and
+   the call's audio enters the same reflex loop a WebRTC call uses.
+
+## T2 — Modular (compose stack)
+
+```bash
+git clone https://git.adlee.work/alee/rutster.git
+cd rutster/deploy
+cp .env.example .env
+$EDITOR .env    # set DOMAIN, ACME email, RUTSTER_MEDIA_ADVERTISED_IP, RUTSTER_TWILIO_*
+docker compose up -d
+```
+
+What comes up: `caddy` (edge, `rutster-edge`), `engine` (`rutster-engine`), `brain`
+(`rutster-brain`, sharing the engine's network namespace), `valkey` (upstream image). The
+same two first-call paths as T1 apply verbatim.
+
+Operational notes:
+
+- `.env.example` ships `RUTSTER_DRAIN_DEADLINE_SECS=600` and the stack sets
+  `stop_grace_period: 660s` — grace exceeds drain, so `docker compose down` lets in-flight
+  calls finish (up to 10 minutes) instead of killing them.
+- Upgrade: `docker compose pull && docker compose up -d`.
+- Bring your own proxy instead of Caddy: [reverse-proxies.md](reverse-proxies.md).
+
+## Troubleshooting
+
+| Symptom | Cause / fix |
+|---|---|
+| Twilio error **31910** on calls | Your cert isn't publicly trusted or expired — check `/data` volume survived, see [certificates.md](certificates.md) ([twilio.com/docs/api/errors/31910](https://www.twilio.com/docs/api/errors/31910)) |
+| Webhook signature validation fails | The edge isn't forwarding `X-Forwarded-Proto/Host` honestly, or `RUTSTER_TRUSTED_PROXIES` doesn't include your proxy — the engine ignores forwarded headers from unlisted sources |
+| Browser call connects, no audio | Media UDP blocked or `RUTSTER_MEDIA_ADVERTISED_IP` wrong — the SDP advertises that IP; callers send UDP straight to it |
+| `curl /readyz` returns 503 | Node is draining or at the admission cap (`RUTSTER_MAX_SESSIONS`) — liveness (`/healthz`) stays 200 |
+| Let's Encrypt rate-limit errors in Caddy logs | You recreated the container without the `/data` volume — [certificates.md](certificates.md) |
--- a/docs/deploy/reverse-proxies.md
+++ b/docs/deploy/reverse-proxies.md
@@ -0,0 +1,139 @@
+# Bring your own reverse proxy — tuned-timeout configs
+
+Supported: disable the `caddy` service in the compose stack
+([quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack)); the engine keeps its
+plaintext `:8080` listener. Everything below terminates TLS in *your* proxy and forwards to
+the engine.
+
+## The universal 60-second footgun
+
+**nginx, HAProxy, and Traefik all default their idle/read timeouts to ~60–180 seconds — every
+one of them will kill a quiet, perfectly healthy call WebSocket.** A Media Streams WS can
+legitimately be near-silent in one direction for hours, and neither Twilio nor Telnyx sends
+protocol-level keepalives. The engine ships app-level WS pings (`RUTSTER_WS_PING_SECS`,
+default 20) as belt-and-braces, but your proxy timeouts must still exceed the maximum call
+duration. Every snippet below does that.
+
+## The contract any proxy must honor
+
+1. **Publicly-trusted cert**, auto-renewed — no self-signed path exists
+   ([certificates.md](certificates.md)).
+2. **WS upgrade with no frame buffering and no connection-lifetime cap** — 20 ms audio frames,
+   calls up to 24 h.
+3. **Honest `X-Forwarded-Proto` and `X-Forwarded-Host`** (including on the WS upgrade
+   request): `X-Twilio-Signature` is HMAC over the URL as Twilio saw it
+   ([Twilio's SSL-termination guidance](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library)).
+4. **Engine-side trust**: set `RUTSTER_TRUSTED_PROXIES` to your proxy's source IP/CIDR —
+   forwarded headers from unlisted sources are ignored, and signature validation will fail.
+5. **TLS 1.2+1.3, mainstream ECDHE — never 1.3-only** (Twilio's 1.3 client support is
+   undocumented).
+
+## nginx (works if you insist — not a recommended edge)
+
+Why not recommended: no native DNS-01/wildcard support, and cert renewal requires a reload
+whose old workers linger against hours-long WS connections (research basis: [TLS brief §4](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)). rutster ships this snippet and
+nothing more for nginx.
+
+```nginx
+map $http_upgrade $connection_upgrade { default upgrade; '' close; }
+
+server {
+    listen 443 ssl;
+    server_name pbx.example.com;
+    ssl_certificate     /etc/letsencrypt/live/pbx.example.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/pbx.example.com/privkey.pem;
+    ssl_protocols TLSv1.2 TLSv1.3;   # never 1.3-only
+
+    location / {
+        proxy_pass http://127.0.0.1:8080;
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection $connection_upgrade;
+        proxy_set_header Host $host;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $host;
+        proxy_buffering off;             # 20 ms frames must not be coalesced
+        proxy_read_timeout 86400s;       # the 60s default is the footgun
+        proxy_send_timeout 86400s;
+    }
+}
+```
+
+## HAProxy (3.2 LTS or newer)
+
+The best WS mental model of the classic family: after the upgrade, only `timeout tunnel`
+governs the connection — **but only if you set it**; put it in `defaults` or it silently
+doesn't apply to the paths you think it does (see
+[haproxy#2280](https://github.com/haproxy/haproxy/issues/2280)). Two extra notes: `acme.sh`
+can hot-load renewed certs through the stats socket with zero reload; and `hard-stop-after`
+kills calls that are still draining — leave it unset or above your drain deadline.
+
+```haproxy
+defaults
+    mode http
+    timeout connect 5s
+    timeout client  75s
+    timeout server  75s
+    timeout tunnel  24h        # governs upgraded WS; unset = the 60s-class footgun
+
+frontend fe_rutster
+    bind :443 ssl crt /etc/haproxy/certs/ ssl-min-ver TLSv1.2
+    http-request set-header X-Forwarded-Proto https
+    http-request set-header X-Forwarded-Host %[req.hdr(Host)]
+    default_backend be_rutster
+
+backend be_rutster
+    server fob 127.0.0.1:8080
+```
+
+## Traefik v3 — PIN THE EXACT VERSION
+
+Traefik has the best built-in DNS-01/wildcard of the classic family **and a track record of
+breaking WebSockets in patch releases**:
+[#10601](https://github.com/traefik/traefik/issues/10601), the v2.11.2 timeout-behavior flip,
+and [#11405](https://github.com/traefik/traefik/issues/11405). **Pin the exact image tag and
+never auto-update the edge.**
+
+```yaml
+# traefik.yml (static config)
+entryPoints:
+  websecure:
+    address: ":443"
+    transport:
+      respondingTimeouts:
+        readTimeout: 0    # 0 disables; the 60s-class default kills quiet WS
+        idleTimeout: 0
+        writeTimeout: 0
+certificatesResolvers:
+  le:
+    acme:
+      email: you@example.com
+      storage: /data/acme.json     # persist this volume — certificates.md
+      dnsChallenge:
+        provider: cloudflare
+```
+
+```yaml
+# dynamic config
+http:
+  routers:
+    rutster:
+      rule: "Host(`pbx.example.com`)"
+      entryPoints: [websecure]
+      service: rutster
+      tls: { certResolver: le }
+  services:
+    rutster:
+      loadBalancer:
+        servers:
+          - url: "http://127.0.0.1:8080"
+```
+
+Traefik forwards `X-Forwarded-Proto/Host` by default — you still must list its address in
+`RUTSTER_TRUSTED_PROXIES`.
+
+## First call
+
+Any of the above in front of the engine, then the same two first-call paths as
+[quickstart-docker.md](quickstart-docker.md): browser at `https://pbx.example.com/`, Twilio
+webhook at `https://pbx.example.com/v1/trunk/webhook`.
--- a/docs/deploy/topologies.md
+++ b/docs/deploy/topologies.md
@@ -0,0 +1,105 @@
+# Deployment topologies — one binary, three blessed shapes
+
+The FOB never decomposes; topology is a configuration property, not a code property. The same
+binary runs in every shape below; only `RUTSTER_*` env and what is deployed alongside it
+differ. Ratified (Proposed) in [ADR-0011](../adr/0011-deployment-topology.md).
+
+## Decision guide
+
+| You are… | Use | Doc |
+|---|---|---|
+| Trying rutster for the first time; one box, one domain, a public IP | **T1 — Solo** | [quickstart-docker.md](quickstart-docker.md) |
+| Running production and want independently upgradable parts | **T2 — Modular** (the reference deployment) | [quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack) |
+| Behind CGNAT / no public IP (homelab) | T1/T2 + tunnel (dev, **PSTN-only**) or VPS forwarder (production) | [homelab.md](homelab.md) |
+| Already running nginx / HAProxy / Traefik | T2 minus the `caddy` service | [reverse-proxies.md](reverse-proxies.md) |
+| On EC2 and want ALB + ACM to terminate TLS | T2/T3 behind an ALB | [aws.md](aws.md) |
+| Outgrowing one node | **T3 — Fleet** — **paper only today** | [ADR-0011](../adr/0011-deployment-topology.md) |
+
+## T1 — Solo (all-in-one container)
+
+One image, `rutster-allinone`, with s6-overlay supervising four processes:
+
+| Process | Role | Binding |
+|---|---|---|
+| `caddy` | Edge: ACME issuance/renewal, TLS termination, WS proxy | `:443` / `:80` public |
+| `rutster` (the FOB) | Engine — signaling, trunk webhook, media, reflexes | `127.0.0.1:8080` behind Caddy; **media UDP direct** |
+| `rutster-brain-realtime` | Brain | `127.0.0.1:8082` loopback tap |
+| `valkey-server` | Event stream (`EventSink`); ledger/directory later | `127.0.0.1:6379` |
+
+Two volumes, **both non-negotiable**:
+
+- `/data` — Caddy's cert/ACME state. Losing it can lock your domain out of Let's Encrypt for
+  up to a week (duplicate-certificate limit) — a **total inbound outage**. See
+  [certificates.md](certificates.md).
+- `/var/lib/valkey` — stream/state persistence.
+
+WebRTC media (UDP) goes **direct to the FOB, never through Caddy**: run with host networking,
+or publish `RUTSTER_MEDIA_PORT_RANGE` and set `RUTSTER_MEDIA_ADVERTISED_IP` to your public IP
+(NAT 1:1 model — no STUN).
+
+Copy-paste path from zero to first call: [quickstart-docker.md](quickstart-docker.md).
+
+## T2 — Modular (compose stack) — the reference deployment
+
+Four services in `deploy/compose.yaml`:
+
+| Service | Image | Notes |
+|---|---|---|
+| `caddy` | `rutster-edge` | Same custom Caddy build + same Caddyfile as T1 |
+| `engine` | `rutster-engine` | The FOB; plaintext `:8080` on the compose network; media UDP published |
+| `brain` | `rutster-brain` | `network_mode: "service:engine"` — shares the engine's netns so the loopback-only tap posture survives unchanged |
+| `valkey` | `valkey/valkey` (upstream) | Own service; network-near |
+
+Operational contract:
+
+- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` — grace **must**
+  exceed drain, or the orchestrator kills calls the engine was gracefully draining.
+- Bring-your-own-proxy: disable the `caddy` service; the engine keeps its plaintext `:8080`
+  mode. Tuned-timeout configs: [reverse-proxies.md](reverse-proxies.md).
+- Upgrades: `docker compose pull && docker compose up -d` — the drain window lets in-flight
+  calls finish (calls are non-migratable; a killed engine kills its calls, honestly stated).
+
+Copy-paste path from zero to first call:
+[quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack).
+
+## T3 — Fleet (paper only — no shipped artifacts in this epoch)
+
+**Honest statement: you cannot deploy T3 today by copy-paste; there are no fleet artifacts
+yet.** The design is ratified on paper in
+[ADR-0011](../adr/0011-deployment-topology.md) so single-node deployments grow toward it, not
+away from it:
+
+- N symmetric nodes (the T1 process set minus valkey) + a shared green zone: one Valkey
+  (presence + directory + spend ledger), object storage.
+- Wildcard DNS `*.pbx.domain`; per-node certs — the traps live in
+  [certificates.md](certificates.md).
+- Answer-time placement via Valkey presence; node-addressed `wss://node-K.pbx.domain/...`
+  Stream URLs; no router tier.
+- Per-node admission cap = the benchmark number (`RUTSTER_MAX_SESSIONS`, default 64, is a
+  placeholder until then).
+- Scale-in is drain-then-terminate only. No spot instances for the engine tier.
+
+## What is deliberately not supported
+
+- **Splitting the FOB into services** — permanent rejection, not a roadmap item.
+- **Serverless** — non-migratable multi-hour calls, per-session UDP sockets, in-process media
+  state. Categorically unfit.
+- **Tunnels in production** — plaintext audio at the tunnel vendor's edge + documented
+  mid-call WS kills. Dev/demo only; the whole story is in [homelab.md](homelab.md).
+- **NLB TLS listeners** — see [aws.md](aws.md) for the 350-second reason.
+- **k8s manifests/Helm** — compose-first. If you run k8s anyway: set
+  `terminationGracePeriodSeconds` above `RUTSTER_DRAIN_DEADLINE_SECS`, same grace-exceeds-drain
+  rule as compose.
+
+## Ports & volumes reference (all shapes)
+
+| Surface | Where | Notes |
+|---|---|---|
+| `:80` / `:443` TCP | Caddy, public | ACME HTTP-01 + TLS + all HTTPS/WSS traffic |
+| `:8080` TCP | FOB, behind the edge | Signaling + `/v1/trunk/webhook` + `/twilio/media-stream` WS — one listener (`RUTSTER_HTTP_BIND`) |
+| `RUTSTER_MEDIA_PORT_RANGE` UDP | FOB, **public, direct** | WebRTC media; never proxied |
+| `:8082` TCP | Brain, loopback only | The tap; non-loopback rejected until step 6 |
+| `:6379` TCP | Valkey, private | Loopback (T1) / compose network (T2) / green zone (T3) |
+| `:9090` TCP | FOB `/metrics`, internal | `RUTSTER_METRICS_BIND`; never routed through Caddy |
+| `/data` volume | Caddy | Cert/ACME state — loss class: total inbound outage |
+| `/var/lib/valkey` volume | Valkey | Event stream persistence |