Files
rutster/docs/superpowers/plans/2026-07-05-deploy-g-docs-adr-0011.md
Aaron D. Lee fae9fcc82a docs(plans): four deployment-epoch implementation plans (slices A–G)
Plan A: engine hygiene (TCP_NODELAY via axum 0.7.9 Serve::tcp_nodelay,
trunk WS pings, webhook-base TwiML derivation, trusted-proxy posture).
Plan B: packaging + CI (four images, s6 all-in-one, compose, Caddyfile,
smoke suite incl. reload-during-call, slice-F publish workflow).
Plan C: in-binary features (rustls Phase 1 BYO-cert, /metrics,
ValkeyEventSink + smoke hook fill).
Plan G: docs/deploy tree + ADR-0011 drafting.

Authored by parallel plan-writers (this session + omo rescue after a
session-limit interruption); verified: placeholder scan, cited-path
existence, cross-plan image/env-var consistency, seam-gate compliance.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012QndwfhjyTiZcUYp87dwW8
Signed-off-by: Aaron D. Lee <himself@adlee.work>
2026-07-05 21:37:17 -04:00

1611 lines
76 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Slice G — `docs/deploy/` tree + ADR-0011 (deployment topology) — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use `superpowers:subagent-driven-development` (recommended) or `superpowers:executing-plans` to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Draft ADR-0011 (deployment topology, status Proposed) from spec §2§4 and ship the
six-file `docs/deploy/` documentation tree, where **every topology doc contains a copy-paste
path from zero to first call — or the honest statement of what cannot work there** (spec §10
acceptance bar), then point `docs/QUICKSTART.md` at the new Docker fast path.
**Architecture:** Docs-only slice — no code, no artifacts. ADR-0011 is drafted verbatim-faithful
to the [deployment-topology spec](../specs/2026-07-05-deployment-topology-design.md) §2 (three
shapes + rejected shapes), §3 (edge doctrine incl. the rustls Phase-2 triggers verbatim), §4
(node-addressed placement), in the house ADR format established by ADR-0007/ADR-0008
(bullet-list header → Context → Decision → Consequences → References, heavy relative
cross-linking). The `docs/deploy/` tree sources every factual claim from the spec + the
[TLS/edge decision brief](../specs/2026-07-05-tls-edge-decision-brief.md) and cites the brief's
external URLs wherever a doc asserts a third-party fact.
**Tech Stack:** Markdown only. No new dependencies, no build steps. Verification is
`bash` content-greps + a relative-link resolver check per task.
## Global Constraints
House rules (every task's requirements implicitly include this section):
- **License:** GPL-3.0-or-later on every crate manifest (ADR-0004). No crates are touched in
this slice; the rule stands for any task that would.
- **DCO:** every commit signed off — `git commit -s` (AGENTS.md Git workflow).
- **SEAM GATE:** `crates/rutster-media/src/loop_driver.rs` and
`crates/rutster-media/src/rtc_session.rs` stay byte-identical (CI pinned-blob). **NO task may
touch them.** This slice touches only `docs/` — trivially satisfied, still checked in the
final sweep.
- **Hot-path policy:** never `?`-propagate on the 20 ms tick; no `unwrap()`/`expect()` outside
tests. (No code in this slice; stated because every plan carries it.)
- **Code style:** `cargo fmt --all --check` + `cargo clippy --all --all-targets -- -D warnings`
must stay green (docs changes cannot break them; run them in the final sweep anyway).
- **MSRV:** CI matrix runs stable + 1.85; docs must not instruct anything that contradicts the
pinned `rust-toolchain.toml`.
- **Workspace dep pinning:** new deps go in the ROOT `Cargo.toml` `[workspace.dependencies]`;
members reference with `dep.workspace = true`. (None added here.)
- **cargo-deny license gate** stays green (unaffected; final sweep runs it).
- **sim-bench CI job** runs `--test-threads=1` (unaffected).
Slice-G-specific constraints:
- **Acceptance bar (spec §10):** every topology doc contains a copy-paste path from zero to
first call, **or** the honest statement of what cannot work there (e.g. CGNAT WebRTC). Each
doc task's verification step greps for the load-bearing statements.
- **Artifact names:** plan B (`docs/superpowers/plans/2026-07-05-deploy-b-packaging-ci.md`)
has landed and settled the registry namespace to **`git.adlee.work/alee/rutster-*`**
(matches the `alee` tea login per AGENTS.md + the root `Cargo.toml`'s
`repository = "https://git.adlee.work/alee/rutster"`). Images: `rutster-allinone`,
`rutster-engine`, `rutster-brain`, `rutster-edge`. `deploy/` tree = `Dockerfile`,
`compose.yaml`, `Caddyfile`, `.env.example`. Every doc task still carries a **cross-check
step**: if `deploy/` exists in the worktree by execution time, reconcile env-var names and
compose service names against the real files before committing.
- **Env vars:** grounded in `crates/rutster/src/config.rs` for what exists today
(`RUTSTER_HTTP_BIND`, `RUTSTER_MEDIA_BIND_IP`, `RUTSTER_MEDIA_ADVERTISED_IP`,
`RUTSTER_MEDIA_PORT_RANGE`, `RUTSTER_MAX_SESSIONS`, `RUTSTER_DRAIN_DEADLINE_SECS`, the four
`RUTSTER_TWILIO_*` vars) and in spec §5.7 for what sibling slices A/C/D/E add
(`RUTSTER_TRUSTED_PROXIES`, `RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY`, `RUTSTER_WS_PING_SECS`,
`RUTSTER_METRICS_BIND`, `RUTSTER_VALKEY_URL`). The docs document the **post-epoch** artifact
set — slice G is sequenced last (spec §11) — so referencing §5.7 vars is correct, not
speculative.
- **Citations:** wherever a `docs/deploy/` file makes a factual claim about third-party
behavior, cite the TLS brief's external source URL inline (Twilio docs/error pages, the
Caddy/cloudflared/axum/livekit issue URLs, etc.).
- **Ports (grounded):** one FOB listener `:8080` (`RUTSTER_HTTP_BIND`;
`crates/rutster/src/main.rs` merges the trunk WS router into the main app — the
`/twilio/media-stream` WS rides the same listener); brain tap `127.0.0.1:8082`; valkey
`:6379`; metrics `127.0.0.1:9090` (spec §5.5); Caddy `:80`/`:443` public; WebRTC media =
UDP `RUTSTER_MEDIA_PORT_RANGE` direct to the FOB, never through Caddy.
- **Do NOT restyle existing docs.** Task 8 (QUICKSTART) is a minimal-diff pointer insertion,
nothing else.
## File Structure
| Path | Action | Responsibility |
|---|---|---|
| `docs/adr/0011-deployment-topology.md` | Create | The topology ADR: three blessed shapes, edge doctrine, fleet placement paradigm, rejected shapes. Status: Proposed. |
| `docs/deploy/topologies.md` | Create | The three shapes + decision-guide table; entry point of the tree. |
| `docs/deploy/quickstart-docker.md` | Create | T1 `docker run` + T2 `compose up`, zero to first call (browser + PSTN). |
| `docs/deploy/homelab.md` | Create | The CGNAT truth: PSTN-only behind tunnels; ngrok blessed dev path; Tailscale Funnel; VPS+WireGuard graduation with TLS at home. |
| `docs/deploy/aws.md` | Create | ALB appendix: three mandatory attribute overrides, 100-TG cap, never-NLB-TLS. |
| `docs/deploy/reverse-proxies.md` | Create | BYO nginx/HAProxy/Traefik tuned-timeout snippets; the universal 60 s footgun; Traefik version pinning. |
| `docs/deploy/certificates.md` | Create | ACME paths, wildcard traps, LE duplicate-cert lockout, `/data` persistence, per-node vs wildcard fleet patterns. |
| `docs/QUICKSTART.md` | Modify (minimal) | One blockquote pointing at `deploy/quickstart-docker.md` as the fast path. |
## Task ordering
- **Task 1 (ADR-0011) has NO dependency — execute immediately.** It is drafted from the spec,
not from artifacts.
- **Tasks 27** are order-independent among themselves and parallelizable; each carries the
slice-B cross-check step (spec §11: G depends on B "for accuracy").
- **Task 8** depends on Task 3 (the file it links to must exist).
Each task = one commit. Verification per task is a bash content-check that fails before the
file is written and passes after.
---
### Task 1: ADR-0011 — `docs/adr/0011-deployment-topology.md`
**Files:**
- Create: `docs/adr/0011-deployment-topology.md`
- Test: bash content-check (Step 1/Step 3 below)
**Interfaces:**
- Consumes: spec §2/§3/§4 (`docs/superpowers/specs/2026-07-05-deployment-topology-design.md`),
the TLS brief (`docs/superpowers/specs/2026-07-05-tls-edge-decision-brief.md`), the house ADR
format (read `docs/adr/0007-trunk-rented-transport.md` and `docs/adr/0008-fob-and-green-zone.md`
before writing — the header bullet-list, Context/Decision/Consequences/References structure,
and relative cross-link style below match them exactly).
- Produces: the ADR every `docs/deploy/` file cross-links as
`../adr/0011-deployment-topology.md` (Tasks 27 rely on this exact path).
- [ ] **Step 1: Run the failing content-check**
```bash
cd /home/alee/Sources/rutster
test -f docs/adr/0011-deployment-topology.md \
&& grep -q 'Status:\*\* Proposed' docs/adr/0011-deployment-topology.md \
&& grep -q 'dns-persist-01' docs/adr/0011-deployment-topology.md \
&& grep -q '307' docs/adr/0011-deployment-topology.md \
&& grep -q 'on-demand TLS' docs/adr/0011-deployment-topology.md \
&& echo CHECK-PASS || echo CHECK-FAIL
```
Expected: `CHECK-FAIL` (file does not exist).
- [ ] **Step 2: Write the complete ADR**
Write `docs/adr/0011-deployment-topology.md` with exactly this content:
````markdown
# ADR-0011 — Deployment topology: one binary, three blessed shapes
- **Status:** Proposed
- **Date:** 2026-07-05
- **Closes:** the deployment-topology ADR reserved by [ADR-0008](0008-fob-and-green-zone.md)'s
worked example ("the deployment topology itself may get its own ADR once that design closes").
- **Origin:** [2026-07-05 deployment-topology spec](../superpowers/specs/2026-07-05-deployment-topology-design.md)
§2§4. The TLS/edge fork was researched via a six-family survey — see the companion
[TLS/edge decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md).
- **Related:** [ADR-0002](0002-north-star-and-fused-core.md) (the fused per-call vertical no
topology may split), [ADR-0005](0005-event-bus.md) (Valkey = bus + KV + presence; media never
rides the bus; the bus is never billing source-of-truth), [ADR-0006](0006-ingress-posture.md)
(two inbound protocols only; the tap is egress-only), [ADR-0007](0007-trunk-rented-transport.md)
(rented transport ⇒ public HTTPS/WSS reachability is non-negotiable),
[ADR-0008](0008-fob-and-green-zone.md) (the FOB/green-zone rule this ADR applies to
deployment), [ADR-0009](0009-spend-gate-honest-rescope.md) (ledger trait: in-memory
single-node, Valkey-backed fleets), [ADR-0010](0010-spearhead-benchmark-sim-harness.md)
(sequencing this ADR must not reorder), README pillar 6 ("one binary, one bus, one deploy").
## Context
[ADR-0008](0008-fob-and-green-zone.md) made the build-vs-reuse doctrine physical in a worked
example — the all-in-one container — and reserved the topology ADR for when the design closed.
Two things have since closed it: the rented transport
([ADR-0007](0007-trunk-rented-transport.md)) is merged, so every deployment must now present a
public HTTPS/WSS edge to a CPaaS; and slice-5/seams landed the operational primitives a
deployment story needs (drain lifecycle, `/readyz`, admission cap, `MediaAddressConfig`,
`EventSink`).
The CPaaS side imposes a hard external spec on any edge rutster ships (researched in the
[decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md); restated here
because these invariants are load-bearing for every decision below):
1. Publicly-CA-trusted cert; **no self-signed path exists** (Twilio error 31910). Auto-renewal
is availability-critical.
2. wss:// on 443, open to the whole internet: Twilio publishes no egress IPs and offers no
mTLS. Auth is application-layer only (HMAC signatures).
3. One WS per call, up to 24 h, 50 msg/s/direction at 20 ms cadence; zero tolerance for frame
buffering, connection-lifetime caps, or write-side idle timers. Neither Twilio nor Telnyx
documents WS keepalive — keepalive is entirely our job.
4. `<Stream>` URLs carry **no query strings** — wss routing keys on hostname/path only.
5. `X-Twilio-Signature` is HMAC over the URL *as Twilio saw it*: the edge must forward
`X-Forwarded-Proto/Host` honestly and the FOB must reconstruct the public URL.
6. Webhook budget: sub-second for UX, 15 s hard cap — no blocking issuance in the handshake
path.
7. TLS 1.2+1.3, mainstream ECDHE, never 1.3-only; never pin Twilio certs.
Design for Twilio; Telnyx/Vonage come for free (strictly less demanding, plus optional
source-IP allowlisting as extra hardening Twilio can't use).
On the other side, the FOB's own physics constrain every shape: per-call state is in-process
and non-migratable (the fused vertical, [ADR-0002](0002-north-star-and-fused-core.md)), WebRTC
media is per-session UDP direct to the process, and calls run for hours. Any topology that
assumes migratable, short-lived, connectionless workloads is disqualified before it starts.
## Decision
**The FOB never decomposes; topology is a configuration property, not a code property.** The
same binary runs in every blessed shape; only `RUTSTER_*` env and what is deployed alongside it
differ. "Microservices" in rutster means *green-zone services get their own
processes/containers* — never that the per-call vertical splits
([ADR-0002](0002-north-star-and-fused-core.md)).
### The three blessed shapes
#### T1 — Solo (all-in-one container) — shipped artifact
One image (`rutster-allinone`), s6-overlay as PID 1 supervising four processes:
| Process | Role | Binding |
|---|---|---|
| `caddy` | Edge: ACME, TLS termination, WS proxy to FOB | :443/:80 public |
| `rutster` (FOB) | Engine | 127.0.0.1:8080 (behind Caddy); media UDP direct |
| `rutster-brain-realtime` | Brain | 127.0.0.1:8082 loopback tap (as today) |
| `valkey-server` | Dark on day one except `EventSink` | 127.0.0.1:6379 |
Two volumes, both non-negotiable: `/data` (Caddy cert/ACME state — loss risks the Let's
Encrypt duplicate-cert lockout, a total-inbound-outage class of failure) and `/var/lib/valkey`
(stream/state persistence). WebRTC media UDP goes **direct to the FOB**, never through Caddy:
host networking recommended; otherwise published `RUTSTER_MEDIA_PORT_RANGE` +
`RUTSTER_MEDIA_ADVERTISED_IP` (NAT 1:1 model, no STUN — unchanged from slice-5/seams).
Valkey ships bundled from v1 **before most consumers exist**: the operator contract (volumes,
ports, upgrade shape) is stable from day one, so the spend ledger
([ADR-0009](0009-spend-gate-honest-rescope.md)) and the fleet directory land later without
changing the deployment shape. Cost accepted: ~15 MB and one mostly-idle process.
#### T2 — Modular (compose stack) — shipped artifact, the reference deployment
Services: `caddy` (image `rutster-edge`), `engine` (`rutster-engine`), `brain`
(`rutster-brain`), `valkey` (upstream `valkey/valkey`).
- **The brain shares the engine's network namespace** (`network_mode: "service:engine"`) so
the loopback-only tap posture (`resolve_tap_url` rejects non-loopback until step 6) survives
unchanged. The brain graduates to its own netns when the wss:// tap lands (step 6).
- Valkey is its own service on the compose network — network-near, satisfying
[ADR-0005](0005-event-bus.md)'s sub-ms rule.
- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` (grace > drain).
- BYO-proxy supported: disable the `caddy` service; the engine keeps its plaintext `:8080`
mode; tuned-timeout snippets for nginx/HAProxy/Traefik ship in
[`docs/deploy/reverse-proxies.md`](../deploy/reverse-proxies.md).
#### T3 — Fleet — ADR chapter only (paper; no code in this epoch)
N **symmetric** nodes, each running the T1 process set *minus* valkey, plus a shared green
zone: one Valkey (presence + directory + the [ADR-0009](0009-spend-gate-honest-rescope.md)
spend ledger) and object storage (named as required, still deferred). Wildcard DNS
`*.pbx.domain`; per-node certs (see the fleet cert story below). The per-node admission cap is
the [ADR-0010](0010-spearhead-benchmark-sim-harness.md) benchmark number (replacing the
`RUTSTER_MAX_SESSIONS=64` placeholder). Scale-in is drain-then-terminate only; spot instances
are rejected for the engine tier (2026-07-04 review — a reclaimed node kills every
non-migratable call on it).
### Edge doctrine
#### Bundled edge: Caddy (both shipped artifacts, same build, same Caddyfile)
Custom xcaddy build (the `rutster-edge` image): Caddy core (Apache-2.0) + a curated DNS-01
plugin set — cloudflare, route53, porkbun, hetzner, desec (duckdns excluded: no license file).
Zero-downtime in-memory cert renewal; verified protocol-unaware WS tunneling (no frame
buffering). Config is a ~6-line Caddyfile with tuned timeouts, honest `X-Forwarded-*`, and
`stream_close_delay` above max call duration — **plus a CI e2e case for
config-reload-during-live-call, because the upstream mitigation has an open bug trail
(caddy #6420/#7222) and is not trusted untested.**
#### In-process TLS: staged — the ratified end-state, not adopted wholesale now
[ADR-0008](0008-fob-and-green-zone.md) claims the trust-domain edge as security-constitutive
FOB territory, and str0m already terminates DTLS-SRTP in-process on the same aws-lc-rs
provider — "keep TLS out of the FOB" is not the objection. The objection is economics: owning
an ACME renewal state machine forever, in a 45-day-cert world.
- **Phase 1 (this epoch):** `axum-server` + rustls with operator-supplied certs
(`RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY`), hot-reload without dropping live WS.
Plaintext-behind-Caddy stays the artifact default.
- **Phase 2 (in-binary ACME) triggers, named:** (a) Let's Encrypt dns-persist-01 confirmed GA
(closes the wildcard/CGNAT issuance gap; instant-acme already parses it); (b) a field
incident where Caddy reload/`stream_close_delay` drops live calls; (c) fleet or
VPS-forwarder topology making encrypted-to-the-binary a selling point; (d) maintenance
fatigue with the curated xcaddy plugin build. Until a trigger fires, Phase 2 is deliberately
not built — no turnkey ACME crate fits the tree today (rustls-acme conflicts with the
deny.toml rcgen/x509-parser posture).
#### Fleet cert story
Per-node public TLS names are CPaaS-imposed (invariant 4 + node-addressed placement). Two
blessed patterns:
1. **Wildcard `*.pbx.domain` via DNS-01** with central issuance/distribution.
2. **Per-node distinct certs** (`node-N.domain`) where the wildcard private key must not live
on every node — one compromised node otherwise burns the whole namespace.
Traps, stated so nobody re-discovers them in production: N nodes independently requesting the
*identical* wildcard hit the Let's Encrypt duplicate-cert limit (5/week) — distinct names or
central distribution only. **Caddy on-demand TLS is rejected for node names**: the first
handshake to a fresh name blocks seconds on issuance, colliding with the sub-second webhook
budget (invariant 6), and its rate-limit knobs are deprecated. Node names known at provision
time get pre-issued certs.
### Fleet placement paradigm (the trunk webhook handler and session API grow *toward* this; no fleet code in this epoch)
**Node-addressed URLs + Valkey presence; no router tier.** (Decision 2026-07-05 over the two
rejected alternatives below.)
- Nodes heartbeat capacity into Valkey (TTL'd presence keys — the
[ADR-0005](0005-event-bus.md) presence layer).
- Placement happens **at answer time** by whichever node the request lands on (`api.domain`,
DNS round-robin): read fleet capacity → pick owner → return TwiML whose Stream URL is
`wss://node-K.pbx.domain/...`. The media WS dials the owning node directly — zero extra
hops, consistent with connection-follows-ownership. Twilio's answer-time Stream URL is
purpose-shaped for this: invariant 4 permits hostname routing.
- WebRTC: the create-session response carries the owner's node-addressed base URL + advertised
media IP (UDP is already direct).
- Misdirected control ops (e.g. a DELETE hitting a non-owner) consult the Valkey directory and
**307-redirect** to the owner.
- **Degraded mode (Valkey dark): the answering node self-places if it has capacity**; presence
heals via TTL on recovery. Fail-safe degradation, consistent with
[ADR-0009](0009-spend-gate-honest-rescope.md)'s posture.
- Single-node is the degenerate case of the same code: a one-entry directory.
**Rejected: a routing tier that proxies everything.** It would put every 20 ms media frame
through an extra userspace hop and re-create exactly the SPOF the symmetric design avoids —
and it buys nothing, because the CPaaS already lets the answer-time URL do the routing.
**Rejected: a distinguished control node.** It breaks node symmetry (two artifacts, two
failure stories, a promotion protocol) to solve a problem the presence directory already
solves; placement needs fleet state either way, and Valkey is already the fleet-state home
([ADR-0005](0005-event-bus.md)).
### Rejected shapes
- **Decomposing the FOB into services** — rejected **permanently**, not deferred.
Media/reflex/tap/spend stay one process, one trust domain
([ADR-0002](0002-north-star-and-fused-core.md)).
- **Serverless.** Lambdas are categorically unfit for the engine: non-migratable multi-hour
calls, per-session UDP sockets, in-process media state. Webhook-on-Lambda buys nothing
because placement needs fleet state anyway.
- **Tunnels in production.** Cloudflare Tunnel and ngrok see plaintext audio at the vendor
edge (an unconsented subprocessor — DPA/BAA/PCI failure), document mid-call WS kills, and
cloudflared has an open Twilio handshake bug (cloudflared #1465). ngrok is the blessed
5-minute demo path (the only tunnel with a proven Media Streams record); Tailscale Funnel is
the privacy-clean single-user demo (TLS terminates on-node), unsizable beyond that. **No
tunnel carries inbound UDP → homelab behind CGNAT is PSTN-only, period.** The production
graduation path is a cheap VPS + WireGuard as a dumb TCP forwarder with TLS terminating at
home, or running the engine on the VPS — see
[`docs/deploy/homelab.md`](../deploy/homelab.md).
- **NLB TLS listeners — never.** Fixed 350 s idle timeout that silently drops flows, and no
HTTP context, so `X-Forwarded-*` headers for signature validation (invariant 5) don't exist.
The ALB path is documented as a docs-only appendix
([`docs/deploy/aws.md`](../deploy/aws.md)), never a shipped artifact.
- **k8s manifests/Helm.** Compose-first (README pillar 6). k8s notes live in docs only (drain
vs `terminationGracePeriodSeconds`); manifests are a later rung if demand appears.
### Deferred, named (so the boundary is deliberate)
- **T3 fleet implementation** (presence heartbeats, directory redirects, placement code).
- **The zero-egress / air-gap profile** (SBC layer-2 ingress per
[ADR-0007](0007-trunk-rented-transport.md) layer 2 + a self-hosted brain): a defined
profile, no code this epoch. T1 is about one-command simplicity, not isolation.
- **In-binary ACME (rustls Phase 2)** — behind the four named triggers above.
- **Durable CDR pipeline**: `ValkeyEventSink` is evidence preservation on a capped stream —
explicitly NOT the billing ledger ([ADR-0005](0005-event-bus.md) source-of-truth rule).
CDR → object storage stays deferred.
- **Multi-tenancy** (needs its own ADR before schemas ossify); wss:// tap + brain fleet +
resume tokens (step 6); media-thread sharding graduation; arm64 images.
## Consequences
- **Positive:** one binary and one mental model across all three shapes — an operator's
knowledge transfers from `docker run` to a fleet; the CPaaS invariants are satisfied by a
decade-hardened ACME loop (Caddy) rather than a DIY renewal state machine; the fleet
paradigm needs zero new infrastructure beyond what [ADR-0005](0005-event-bus.md) already
bundles; the rejected shapes are written down, so nobody burns a week discovering that
Lambda or NLB-TLS can't work.
- **Negative:** the curated xcaddy DNS-plugin set is a permanent maintenance surface until
dns-persist-01 GA; Caddy config-reload behavior vs live calls is mitigated
(`stream_close_delay` + CI e2e) but has an open upstream bug trail; homelab WebRTC behind
CGNAT is unsolved **by design** — only the VPS graduation fixes it, and the docs must say so
loudly or operators will read "homelab supported" as including browser calls.
- **Mitigation:** the Phase-2 triggers convert both negatives into a planned exit; the
`docs/deploy/` tree ships with a copy-paste-to-first-call acceptance bar so the honest
limits are in the operator's face, not in a footnote.
## References
- [2026-07-05 deployment-topology spec](../superpowers/specs/2026-07-05-deployment-topology-design.md) — §2§4 are this ADR's source
- [TLS/edge decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md) — six-family survey + the CPaaS ground truth (external citations live there)
- [ADR-0002](0002-north-star-and-fused-core.md) · [ADR-0005](0005-event-bus.md) · [ADR-0006](0006-ingress-posture.md) · [ADR-0007](0007-trunk-rented-transport.md) · [ADR-0008](0008-fob-and-green-zone.md) · [ADR-0009](0009-spend-gate-honest-rescope.md) · [ADR-0010](0010-spearhead-benchmark-sim-harness.md)
- The `docs/deploy/` tree: [topologies](../deploy/topologies.md) · [quickstart-docker](../deploy/quickstart-docker.md) · [homelab](../deploy/homelab.md) · [aws](../deploy/aws.md) · [reverse-proxies](../deploy/reverse-proxies.md) · [certificates](../deploy/certificates.md)
````
- [ ] **Step 3: Re-run the content-check + link check**
```bash
cd /home/alee/Sources/rutster
test -f docs/adr/0011-deployment-topology.md \
&& grep -q 'Status:\*\* Proposed' docs/adr/0011-deployment-topology.md \
&& grep -q 'dns-persist-01' docs/adr/0011-deployment-topology.md \
&& grep -q '307' docs/adr/0011-deployment-topology.md \
&& grep -q 'on-demand TLS' docs/adr/0011-deployment-topology.md \
&& echo CHECK-PASS || echo CHECK-FAIL
# Relative links resolve (the ../deploy/ links will report BROKEN until Tasks 27 land —
# acceptable inside this slice; re-run in the final sweep where all must resolve):
grep -oE '\]\([^)#]+\)' docs/adr/0011-deployment-topology.md | sed 's/](//;s/)//' \
| grep -v '^http' | while read -r p; do [ -e "docs/adr/$p" ] || echo "BROKEN: $p"; done
```
Expected: `CHECK-PASS`; link check reports only `../deploy/*.md` as BROKEN if Tasks 27
haven't run yet (nothing else).
- [ ] **Step 4: Commit**
```bash
cd /home/alee/Sources/rutster
git add docs/adr/0011-deployment-topology.md
git commit -s -m "docs(adr): ADR-0011 deployment topology — one binary, three blessed shapes (Proposed) (slice-G)
Drafted from the 2026-07-05 deployment-topology spec §2§4: T1 solo /
T2 modular / T3 fleet (paper), the edge doctrine (bundled Caddy; rustls
Phase 1 now, Phase 2 behind four named triggers), the fleet cert story
(wildcard vs per-node; on-demand TLS rejected), node-addressed placement
via Valkey presence (routing tier + distinguished control node rejected),
and the rejected shapes (FOB decomposition, serverless, tunnels-in-prod,
NLB-TLS, k8s manifests). Status Proposed — ratification per repo process.
Closes the topology ADR reserved by ADR-0008's worked example."
```
---
### Task 2: `docs/deploy/topologies.md` — the three shapes + decision guide
**Files:**
- Create: `docs/deploy/topologies.md`
- Test: bash content-check (Steps 1/3)
**Interfaces:**
- Consumes: ADR-0011 (Task 1) as `../adr/0011-deployment-topology.md`; spec §2, §7, §8.
- Produces: the tree's entry point; Tasks 37 link to it as `topologies.md` (same dir).
- [ ] **Step 1: Run the failing content-check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/topologies.md \
&& grep -q 'decision guide' docs/deploy/topologies.md \
&& grep -qi 'paper' docs/deploy/topologies.md \
&& echo CHECK-PASS || echo CHECK-FAIL
```
Expected: `CHECK-FAIL`.
- [ ] **Step 2: Write the complete file**
Write `docs/deploy/topologies.md` with exactly this content:
````markdown
# Deployment topologies — one binary, three blessed shapes
The FOB never decomposes; topology is a configuration property, not a code property. The same
binary runs in every shape below; only `RUTSTER_*` env and what is deployed alongside it
differ. Ratified (Proposed) in [ADR-0011](../adr/0011-deployment-topology.md).
## Decision guide
| You are… | Use | Doc |
|---|---|---|
| Trying rutster for the first time; one box, one domain, a public IP | **T1 — Solo** | [quickstart-docker.md](quickstart-docker.md) |
| Running production and want independently upgradable parts | **T2 — Modular** (the reference deployment) | [quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack) |
| Behind CGNAT / no public IP (homelab) | T1/T2 + tunnel (dev, **PSTN-only**) or VPS forwarder (production) | [homelab.md](homelab.md) |
| Already running nginx / HAProxy / Traefik | T2 minus the `caddy` service | [reverse-proxies.md](reverse-proxies.md) |
| On EC2 and want ALB + ACM to terminate TLS | T2/T3 behind an ALB | [aws.md](aws.md) |
| Outgrowing one node | **T3 — Fleet** — **paper only today** | [ADR-0011](../adr/0011-deployment-topology.md) |
## T1 — Solo (all-in-one container)
One image, `rutster-allinone`, with s6-overlay supervising four processes:
| Process | Role | Binding |
|---|---|---|
| `caddy` | Edge: ACME issuance/renewal, TLS termination, WS proxy | `:443` / `:80` public |
| `rutster` (the FOB) | Engine — signaling, trunk webhook, media, reflexes | `127.0.0.1:8080` behind Caddy; **media UDP direct** |
| `rutster-brain-realtime` | Brain | `127.0.0.1:8082` loopback tap |
| `valkey-server` | Event stream (`EventSink`); ledger/directory later | `127.0.0.1:6379` |
Two volumes, **both non-negotiable**:
- `/data` — Caddy's cert/ACME state. Losing it can lock your domain out of Let's Encrypt for
up to a week (duplicate-certificate limit) — a **total inbound outage**. See
[certificates.md](certificates.md).
- `/var/lib/valkey` — stream/state persistence.
WebRTC media (UDP) goes **direct to the FOB, never through Caddy**: run with host networking,
or publish `RUTSTER_MEDIA_PORT_RANGE` and set `RUTSTER_MEDIA_ADVERTISED_IP` to your public IP
(NAT 1:1 model — no STUN).
Copy-paste path from zero to first call: [quickstart-docker.md](quickstart-docker.md).
## T2 — Modular (compose stack) — the reference deployment
Four services in `deploy/compose.yaml`:
| Service | Image | Notes |
|---|---|---|
| `caddy` | `rutster-edge` | Same custom Caddy build + same Caddyfile as T1 |
| `engine` | `rutster-engine` | The FOB; plaintext `:8080` on the compose network; media UDP published |
| `brain` | `rutster-brain` | `network_mode: "service:engine"` — shares the engine's netns so the loopback-only tap posture survives unchanged |
| `valkey` | `valkey/valkey` (upstream) | Own service; network-near |
Operational contract:
- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` — grace **must**
exceed drain, or the orchestrator kills calls the engine was gracefully draining.
- Bring-your-own-proxy: disable the `caddy` service; the engine keeps its plaintext `:8080`
mode. Tuned-timeout configs: [reverse-proxies.md](reverse-proxies.md).
- Upgrades: `docker compose pull && docker compose up -d` — the drain window lets in-flight
calls finish (calls are non-migratable; a killed engine kills its calls, honestly stated).
Copy-paste path from zero to first call:
[quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack).
## T3 — Fleet (paper only — no shipped artifacts in this epoch)
**Honest statement: you cannot deploy T3 today by copy-paste; there are no fleet artifacts
yet.** The design is ratified on paper in
[ADR-0011](../adr/0011-deployment-topology.md) so single-node deployments grow toward it, not
away from it:
- N symmetric nodes (the T1 process set minus valkey) + a shared green zone: one Valkey
(presence + directory + spend ledger), object storage.
- Wildcard DNS `*.pbx.domain`; per-node certs — the traps live in
[certificates.md](certificates.md).
- Answer-time placement via Valkey presence; node-addressed `wss://node-K.pbx.domain/...`
Stream URLs; no router tier.
- Per-node admission cap = the benchmark number (`RUTSTER_MAX_SESSIONS`, default 64, is a
placeholder until then).
- Scale-in is drain-then-terminate only. No spot instances for the engine tier.
## What is deliberately not supported
- **Splitting the FOB into services** — permanent rejection, not a roadmap item.
- **Serverless** — non-migratable multi-hour calls, per-session UDP sockets, in-process media
state. Categorically unfit.
- **Tunnels in production** — plaintext audio at the tunnel vendor's edge + documented
mid-call WS kills. Dev/demo only; the whole story is in [homelab.md](homelab.md).
- **NLB TLS listeners** — see [aws.md](aws.md) for the 350-second reason.
- **k8s manifests/Helm** — compose-first. If you run k8s anyway: set
`terminationGracePeriodSeconds` above `RUTSTER_DRAIN_DEADLINE_SECS`, same grace-exceeds-drain
rule as compose.
## Ports & volumes reference (all shapes)
| Surface | Where | Notes |
|---|---|---|
| `:80` / `:443` TCP | Caddy, public | ACME HTTP-01 + TLS + all HTTPS/WSS traffic |
| `:8080` TCP | FOB, behind the edge | Signaling + `/v1/trunk/webhook` + `/twilio/media-stream` WS — one listener (`RUTSTER_HTTP_BIND`) |
| `RUTSTER_MEDIA_PORT_RANGE` UDP | FOB, **public, direct** | WebRTC media; never proxied |
| `:8082` TCP | Brain, loopback only | The tap; non-loopback rejected until step 6 |
| `:6379` TCP | Valkey, private | Loopback (T1) / compose network (T2) / green zone (T3) |
| `:9090` TCP | FOB `/metrics`, internal | `RUTSTER_METRICS_BIND`; never routed through Caddy |
| `/data` volume | Caddy | Cert/ACME state — loss class: total inbound outage |
| `/var/lib/valkey` volume | Valkey | Event stream persistence |
````
- [ ] **Step 3: Re-run the content-check + link check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/topologies.md \
&& grep -qi 'decision guide' docs/deploy/topologies.md \
&& grep -q 'paper only' docs/deploy/topologies.md \
&& grep -q 'cannot deploy T3 today' docs/deploy/topologies.md \
&& echo CHECK-PASS || echo CHECK-FAIL
grep -oE '\]\([^)#]+\)' docs/deploy/topologies.md | sed 's/](//;s/)//' \
| grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done
```
Expected: `CHECK-PASS`; BROKEN lines only for sibling docs whose tasks haven't run yet.
- [ ] **Step 4: Cross-check against slice B (if landed)**
```bash
cd /home/alee/Sources/rutster
ls deploy/ 2>/dev/null && grep -n 'image:' deploy/compose.yaml
```
If `deploy/` exists and any image/service name differs from this doc's tables, fix the doc to
match reality before committing. If `deploy/` does not exist, the doc stands on spec §6.1
names — no change.
- [ ] **Step 5: Commit**
```bash
cd /home/alee/Sources/rutster
git add docs/deploy/topologies.md
git commit -s -m "docs(deploy): topologies.md — three blessed shapes + decision guide (slice-G)
The docs/deploy/ tree entry point. T1/T2 process+service tables, the
grace-exceeds-drain rule, the honest T3 statement (paper only, no
copy-paste deploy exists), the not-supported list, and the shared
ports/volumes reference. Acceptance bar: copy-paste paths delegate to
quickstart-docker.md; T3 states plainly what cannot be done today."
```
---
### Task 3: `docs/deploy/quickstart-docker.md` — zero to first call
**Files:**
- Create: `docs/deploy/quickstart-docker.md`
- Test: bash content-check (Steps 1/3)
**Interfaces:**
- Consumes: spec §2.1/§2.2/§5.7/§6, image names from spec §6.1, env vars grounded in
`crates/rutster/src/config.rs` (the four `RUTSTER_TWILIO_*` vars are all-or-none — the
parser rejects partial config), routes grounded in `crates/rutster/src/routes.rs`
(`/healthz`, `/readyz`, `/v1/trunk/webhook`).
- Produces: the file `docs/QUICKSTART.md` links to in Task 8 (exact path
`docs/deploy/quickstart-docker.md`).
- [ ] **Step 1: Run the failing content-check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/quickstart-docker.md \
&& grep -q 'rutster-allinone' docs/deploy/quickstart-docker.md \
&& grep -q 'docker compose up' docs/deploy/quickstart-docker.md \
&& echo CHECK-PASS || echo CHECK-FAIL
```
Expected: `CHECK-FAIL`.
- [ ] **Step 2: Write the complete file**
Write `docs/deploy/quickstart-docker.md` with exactly this content:
````markdown
# Docker quickstart — zero to first call
Two shapes, one binary ([topologies.md](topologies.md)). T1 is one `docker run`; T2 is the
reference `compose up`. Both end the same way: a browser call at your domain, and (with Twilio
credentials) a real phone call.
> **Artifact names note:** images are published to the `git.adlee.work/alee/` registry
> namespace by the release CI (plan B / slice F settled this). If your Gitea registry uses a
> different owner segment, substitute accordingly — check `deploy/compose.yaml` in your
> checkout, which is authoritative.
## Prerequisites (both shapes)
1. A Linux host with Docker, a **public IPv4**, and inbound `80/tcp`, `443/tcp`, and your
chosen media UDP range open. (No public IP? → [homelab.md](homelab.md).)
2. A domain with an A record pointing at that host, e.g. `pbx.example.com`. Certificates are
issued automatically via ACME — no self-signed path exists, the CPaaS refuses it
([certificates.md](certificates.md)).
3. For the phone call: a [Twilio](https://www.twilio.com/) account + a Voice-capable number
(trial works).
## T1 — Solo (all-in-one)
```bash
docker run -d --name rutster --restart unless-stopped \
--network host \
-v rutster-caddy-data:/data \
-v rutster-valkey:/var/lib/valkey \
-e RUTSTER_DOMAIN=pbx.example.com \
-e RUTSTER_ACME_EMAIL=you@example.com \
-e RUTSTER_MEDIA_ADVERTISED_IP=203.0.113.7 \
-e RUTSTER_MEDIA_PORT_RANGE=49152-49407 \
-e RUTSTER_TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
-e RUTSTER_TWILIO_AUTH_TOKEN=your_auth_token \
-e RUTSTER_TWILIO_MEDIA_BIND=0.0.0.0:8081 \
-e RUTSTER_TWILIO_WEBHOOK_BASE=https://pbx.example.com \
git.adlee.work/alee/rutster-allinone:latest
```
Notes:
- `--network host` is the recommended mode: WebRTC media UDP goes **direct to the engine**,
never through Caddy. If you can't use host networking, publish `-p 80:80 -p 443:443
-p 49152-49407:49152-49407/udp` instead and keep `RUTSTER_MEDIA_ADVERTISED_IP` = the host's
public IP (NAT 1:1, no STUN).
- The two volumes are **non-negotiable**. `/data` in particular: recreating the container
without it re-requests certificates and can hit Let's Encrypt's duplicate-certificate
limit — a week-long total inbound outage ([certificates.md](certificates.md)).
- The four `RUTSTER_TWILIO_*` vars are all-or-none — the engine refuses partial Twilio config
at startup. Omit all four to run WebRTC-only.
- `RUTSTER_DOMAIN` / `RUTSTER_ACME_EMAIL` feed the bundled Caddyfile. Domains that can't do
HTTP-01 (no port 80, wildcard needs) use DNS-01 — [certificates.md](certificates.md).
Verify it's up:
```bash
curl -fsS https://pbx.example.com/healthz && echo OK # liveness
curl -fsS https://pbx.example.com/readyz && echo READY # can accept a new call
```
### First call (browser, WebRTC)
Open `https://pbx.example.com/`, click **Start call**, grant the microphone. You're on a call
with the engine.
### First call (phone, PSTN)
1. Twilio Console → Phone Numbers → your number → **A call comes in** → Webhook,
`POST https://pbx.example.com/v1/trunk/webhook`.
2. Dial your Twilio number from any phone. Twilio hits the webhook, the engine answers with
TwiML pointing Twilio's Media Streams at `wss://pbx.example.com/twilio/media-stream`, and
the call's audio enters the same reflex loop a WebRTC call uses.
## T2 — Modular (compose stack)
```bash
git clone https://git.adlee.work/alee/rutster.git
cd rutster/deploy
cp .env.example .env
$EDITOR .env # set DOMAIN, ACME email, RUTSTER_MEDIA_ADVERTISED_IP, RUTSTER_TWILIO_*
docker compose up -d
```
What comes up: `caddy` (edge, `rutster-edge`), `engine` (`rutster-engine`), `brain`
(`rutster-brain`, sharing the engine's network namespace), `valkey` (upstream image). The
same two first-call paths as T1 apply verbatim.
Operational notes:
- `.env.example` ships `RUTSTER_DRAIN_DEADLINE_SECS=600` and the stack sets
`stop_grace_period: 660s` — grace exceeds drain, so `docker compose down` lets in-flight
calls finish (up to 10 minutes) instead of killing them.
- Upgrade: `docker compose pull && docker compose up -d`.
- Bring your own proxy instead of Caddy: [reverse-proxies.md](reverse-proxies.md).
## Troubleshooting
| Symptom | Cause / fix |
|---|---|
| Twilio error **31910** on calls | Your cert isn't publicly trusted or expired — check `/data` volume survived, see [certificates.md](certificates.md) ([twilio.com/docs/api/errors/31910](https://www.twilio.com/docs/api/errors/31910)) |
| Webhook signature validation fails | The edge isn't forwarding `X-Forwarded-Proto/Host` honestly, or `RUTSTER_TRUSTED_PROXIES` doesn't include your proxy — the engine ignores forwarded headers from unlisted sources |
| Browser call connects, no audio | Media UDP blocked or `RUTSTER_MEDIA_ADVERTISED_IP` wrong — the SDP advertises that IP; callers send UDP straight to it |
| `curl /readyz` returns 503 | Node is draining or at the admission cap (`RUTSTER_MAX_SESSIONS`) — liveness (`/healthz`) stays 200 |
| Let's Encrypt rate-limit errors in Caddy logs | You recreated the container without the `/data` volume — [certificates.md](certificates.md) |
````
- [ ] **Step 3: Re-run the content-check + link check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/quickstart-docker.md \
&& grep -q 'rutster-allinone' docs/deploy/quickstart-docker.md \
&& grep -q 'docker compose up' docs/deploy/quickstart-docker.md \
&& grep -q '/v1/trunk/webhook' docs/deploy/quickstart-docker.md \
&& grep -q 'all-or-none' docs/deploy/quickstart-docker.md \
&& echo CHECK-PASS || echo CHECK-FAIL
grep -oE '\]\([^)#]+\)' docs/deploy/quickstart-docker.md | sed 's/](//;s/)//' \
| grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done
```
Expected: `CHECK-PASS`; BROKEN only for sibling docs not yet written.
- [ ] **Step 4: Cross-check against slice B (if landed)**
```bash
cd /home/alee/Sources/rutster
ls deploy/ 2>/dev/null && { grep -n 'image:' deploy/compose.yaml; grep -n '^[A-Z]' deploy/.env.example; }
```
Reconcile: image names, the `RUTSTER_DOMAIN`/`RUTSTER_ACME_EMAIL` variable names (these two
are artifact-level names owned by slice B's Caddyfile — the spec does not fix them), the
compose service names, and the git clone URL. If `deploy/` does not exist yet, keep the doc as
written and leave the artifact-names note box in place.
- [ ] **Step 5: Commit**
```bash
cd /home/alee/Sources/rutster
git add docs/deploy/quickstart-docker.md
git commit -s -m "docs(deploy): quickstart-docker.md — T1 docker run + T2 compose, zero to first call (slice-G)
The acceptance-bar doc: a copy-paste path from a bare Linux host with a
domain to (1) a browser WebRTC call and (2) a real PSTN call via the
Twilio webhook. Volumes marked non-negotiable with the LE-lockout reason;
all-or-none RUTSTER_TWILIO_* contract stated (matches config.rs); host
networking recommended for direct media UDP."
```
---
### Task 4: `docs/deploy/homelab.md` — the CGNAT truth
**Files:**
- Create: `docs/deploy/homelab.md`
- Test: bash content-check (Steps 1/3)
**Interfaces:**
- Consumes: spec §3.5, brief §3(d) + §2 invariant 9 + §5 risks 3/6, the brief's URLs
(cloudflared #1465/#1282, livekit/agents #3379).
- Produces: the doc `topologies.md` and ADR-0011 link as `homelab.md` /
`../deploy/homelab.md`.
- [ ] **Step 1: Run the failing content-check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/homelab.md \
&& grep -q 'PSTN-only' docs/deploy/homelab.md \
&& grep -q 'WireGuard' docs/deploy/homelab.md \
&& echo CHECK-PASS || echo CHECK-FAIL
```
Expected: `CHECK-FAIL`.
- [ ] **Step 2: Write the complete file**
Write `docs/deploy/homelab.md` with exactly this content:
````markdown
# Homelab & CGNAT — the honest story
**The truth first: no tunnel carries inbound UDP. WebRTC callers are unreachable behind any
tunnel, so a homelab behind CGNAT is PSTN-only, period.** The engine's WebRTC media is
UDP-direct to an advertised IP (no STUN, no TURN); the CPaaS side offers no relay or
rendezvous either, and intermediaries in the media path are a documented frame-dropping hazard
([livekit/agents#3379](https://github.com/livekit/agents/issues/3379)). Nothing on this page
makes CGNAT production-grade for free. Three tiers, worst to best:
## Tier 1 — dev/demo: ngrok (the blessed 5-minute path)
ngrok is the **only** tunnel with a proven Twilio Media Streams record. Do **not** use
Cloudflare Tunnel even for dev: cloudflared has an open Twilio 31920 handshake bug
([cloudflared#1465](https://github.com/cloudflare/cloudflared/issues/1465)), recurring 1006
closures ([cloudflared#1282](https://github.com/cloudflare/cloudflared/issues/1282)), and
Cloudflare documents killing WebSockets on edge code releases.
```bash
# 1. Run the engine, plaintext :8080 (no Caddy needed — ngrok terminates TLS):
docker run -d --name rutster-engine --network host \
-e RUTSTER_TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
-e RUTSTER_TWILIO_AUTH_TOKEN=your_auth_token \
-e RUTSTER_TWILIO_MEDIA_BIND=0.0.0.0:8081 \
-e RUTSTER_TWILIO_WEBHOOK_BASE=https://REPLACE-ME.ngrok-free.app \
-e RUTSTER_TRUSTED_PROXIES=127.0.0.1/32 \
git.adlee.work/alee/rutster-engine:latest
# (or from source: cargo run — see docs/QUICKSTART.md)
# 2. Tunnel it:
ngrok http 8080
# 3. Put the printed https://xxxx.ngrok-free.app URL into RUTSTER_TWILIO_WEBHOOK_BASE
# (restart the container), and into the Twilio number's webhook:
# POST https://xxxx.ngrok-free.app/v1/trunk/webhook
# 4. Dial your Twilio number.
```
**Free-tier arithmetic** (why this is a demo, not a deployment — research: [TLS brief §3(d)](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)): a Media Streams call is
8 kHz µ-law — 8 kB/s of audio per direction, base64-encoded inside a JSON envelope at 50
messages/s, both directions ≈ **~140 MB per call-hour** on the wire. ngrok's free 1 GB/month
data cap ([ngrok pricing](https://ngrok.com/pricing)) therefore buys roughly **seven call-hours a month**. And the audio transits ngrok's
edge **in plaintext** (they terminate TLS) — an unconsented subprocessor, which is a
DPA/BAA/PCI failure. Dev only. Never production.
## Tier 2 — single-user demo: Tailscale Funnel
Privacy-clean variant: TLS terminates **on your node**, so Funnel relays ciphertext it cannot
read.
```bash
tailscale funnel 8080
# webhook base = https://<your-node>.<tailnet>.ts.net
```
Still PSTN-only (no inbound UDP), bandwidth cap undisclosed, one user. Unsizable beyond a
personal demo.
## Tier 3 — production graduation: cheap VPS + WireGuard, TLS at home
The recommended path. The VPS is a **dumb layer-4 forwarder**: TLS terminates at home (your
Caddy, DNS-01 certs), so the forwarder **physically cannot read the audio** — the strongest
privacy topology available. Bonus: forwarding the media UDP range over the same tunnel
restores WebRTC, which no tunnel product can do. Cost: one small VPS (~$5/mo) and one extra
network hop of media latency — pick a VPS region near home.
Point DNS at the VPS: `pbx.example.com A <VPS-public-IP>`.
**VPS `/etc/wireguard/wg0.conf`:**
```ini
[Interface]
Address = 10.88.0.1/24
ListenPort = 51820
PrivateKey = <vps-private-key>
[Peer]
PublicKey = <home-public-key>
AllowedIPs = 10.88.0.2/32
```
**Home box `/etc/wireguard/wg0.conf`** (home initiates — CGNAT-friendly; the keepalive holds
the NAT mapping):
```ini
[Interface]
Address = 10.88.0.2/24
PrivateKey = <home-private-key>
[Peer]
PublicKey = <vps-public-key>
Endpoint = <VPS-public-IP>:51820
AllowedIPs = 10.88.0.0/24
PersistentKeepalive = 25
```
**VPS forwarding** (`sysctl -w net.ipv4.ip_forward=1`, persist it, then nftables):
```nft
table ip rutster-fwd {
chain prerouting {
type nat hook prerouting priority dstnat;
iifname "eth0" tcp dport { 80, 443 } dnat to 10.88.0.2
iifname "eth0" udp dport 49152-49407 dnat to 10.88.0.2
}
chain postrouting {
type nat hook postrouting priority srcnat;
oifname "wg0" masquerade
}
}
```
**Home side:** run T1 or T2 exactly per [quickstart-docker.md](quickstart-docker.md), with:
- `RUTSTER_MEDIA_ADVERTISED_IP=<VPS-public-IP>` — callers send media UDP to the VPS; the DNAT
delivers it home through the tunnel.
- `RUTSTER_MEDIA_PORT_RANGE=49152-49407` (must match the nftables rule).
- Certificates: DNS-01 is the robust choice here ([certificates.md](certificates.md));
HTTP-01 also works since `:80` is forwarded.
First call: same two paths as [quickstart-docker.md](quickstart-docker.md) — browser at
`https://pbx.example.com/` (WebRTC now works — the UDP range rides the tunnel) and the Twilio
webhook at `https://pbx.example.com/v1/trunk/webhook`.
**Or skip the tunnel entirely:** run the engine *on* the VPS (that is just
[T1](quickstart-docker.md) on rented hardware). You trade at-home media for zero forwarding
complexity.
## Explicitly unsupported for production
Cloudflare Tunnel or ngrok in the live audio path: plaintext audio at the vendor edge
(unconsented subprocessor — DPA/BAA/PCI failure), documented mid-call WS terminations, zero
SLA, and Cloudflare's discretionary "disproportionate audio" ToS clause aimed at exactly this
traffic profile (research basis: [TLS brief §4](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)). Ratified in [ADR-0011](../adr/0011-deployment-topology.md).
````
- [ ] **Step 3: Re-run the content-check + link check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/homelab.md \
&& grep -q 'PSTN-only, period' docs/deploy/homelab.md \
&& grep -q 'seven call-hours' docs/deploy/homelab.md \
&& grep -q 'Tailscale Funnel' docs/deploy/homelab.md \
&& grep -q 'physically cannot read the audio' docs/deploy/homelab.md \
&& grep -q 'cloudflared/issues/1465' docs/deploy/homelab.md \
&& echo CHECK-PASS || echo CHECK-FAIL
grep -oE '\]\([^)#]+\)' docs/deploy/homelab.md | sed 's/](//;s/)//' \
| grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done
```
Expected: `CHECK-PASS`; BROKEN only for sibling docs not yet written.
- [ ] **Step 4: Cross-check against slice B (if landed)** — same command as Task 3 Step 4;
reconcile the `rutster-engine` image name and env-var names if `deploy/` exists.
- [ ] **Step 5: Commit**
```bash
cd /home/alee/Sources/rutster
git add docs/deploy/homelab.md
git commit -s -m "docs(deploy): homelab.md — CGNAT truth: PSTN-only behind tunnels, VPS+WireGuard graduation (slice-G)
Acceptance bar honored by honesty: the doc leads with what cannot work
(no tunnel carries inbound UDP -> WebRTC unreachable behind CGNAT). Three
tiers: ngrok blessed dev path with the free-tier arithmetic (~140 MB/
call-hour -> ~7 call-hours/month), Tailscale Funnel single-user demo
(ciphertext-only), and the VPS+WireGuard graduation with TLS terminating
at home (forwarder physically cannot read audio; media UDP DNAT restores
WebRTC). cloudflared explicitly excluded even for dev (#1465, #1282)."
```
---
### Task 5: `docs/deploy/aws.md` — the ALB appendix
**Files:**
- Create: `docs/deploy/aws.md`
- Test: bash content-check (Steps 1/3)
**Interfaces:**
- Consumes: spec §3.5 (AWS appendix bullet), brief §3(c) + §4 (ALB row, NLB rejection).
- Produces: the doc `topologies.md` and ADR-0011 link as `aws.md` / `../deploy/aws.md`.
- [ ] **Step 1: Run the failing content-check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/aws.md \
&& grep -q 'idle_timeout' docs/deploy/aws.md \
&& grep -q '350' docs/deploy/aws.md \
&& echo CHECK-PASS || echo CHECK-FAIL
```
Expected: `CHECK-FAIL`.
- [ ] **Step 2: Write the complete file**
Write `docs/deploy/aws.md` with exactly this content:
````markdown
# AWS appendix — ALB + ACM (docs only, never a shipped artifact)
Scope: operators **already on EC2**. There, an ALB is trust-neutral (the engine is on AWS
infrastructure regardless) and ACM renewal is genuinely zero-touch. This page is an appendix,
not a recommendation to move to AWS — the shipped edges are Caddy
([topologies.md](topologies.md)) and BYO proxies ([reverse-proxies.md](reverse-proxies.md)).
All ALB/NLB facts below are grounded in the [TLS/edge decision brief §3(c)](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md).
## Never NLB-TLS
Do not put the engine behind an NLB TLS listener. Two disqualifiers, ratified in
[ADR-0011](../adr/0011-deployment-topology.md):
1. **Fixed 350-second idle timeout** that silently drops flows — any quiet WebSocket (hold,
parked call, unidirectional stream) dies mid-call with no close frame.
2. **No HTTP context**: an NLB adds no `X-Forwarded-Proto/Host`, so the engine cannot
reconstruct the public URL that `X-Twilio-Signature` is HMAC'd over
([Twilio's SSL-termination guidance](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library))
— genuine traffic fails validation.
## The three mandatory ALB attribute overrides
The defaults kill telephony. All three overrides are **mandatory**, not tuning:
| Attribute | Value | Why |
|---|---|---|
| `idle_timeout.timeout_seconds` | `4000` | The default 60 s kills any quiet WS. 4000 is the ALB maximum — still finite, so the engine's app-level WS pings (`RUTSTER_WS_PING_SECS`, default 20) remain load-bearing here. |
| `client_keep_alive.seconds` | `604800` | The 3600 s default can terminate the client connection under a **>1-hour call**. Set to the 7-day maximum. |
| `routing.http.preserve_host_header.enabled` | `true` | Signature validation reconstructs the URL *as Twilio saw it* — the engine must receive the honest public `Host`, and its `RUTSTER_TRUSTED_PROXIES` must include the ALB's subnet CIDRs. |
## Copy-paste path (CLI)
Assumes: engine node(s) running per [quickstart-docker.md](quickstart-docker.md) with the
`caddy` service disabled (plaintext `:8080`), a VPC, two subnets, and a security group
allowing `:443` from anywhere and `:8080` from the ALB.
```bash
VPC=vpc-0123456789abcdef0
SUBNETS="subnet-0aaa subnet-0bbb"
SG=sg-0ccc
ALB_ARN=$(aws elbv2 create-load-balancer --name rutster-edge --type application \
--subnets $SUBNETS --security-groups $SG \
--query 'LoadBalancers[0].LoadBalancerArn' --output text)
# THE THREE MANDATORY OVERRIDES — do not skip:
aws elbv2 modify-load-balancer-attributes --load-balancer-arn "$ALB_ARN" --attributes \
Key=idle_timeout.timeout_seconds,Value=4000 \
Key=client_keep_alive.seconds,Value=604800 \
Key=routing.http.preserve_host_header.enabled,Value=true
TG_ARN=$(aws elbv2 create-target-group --name rutster-node-1 \
--protocol HTTP --port 8080 --vpc-id "$VPC" \
--health-check-path /readyz \
--query 'TargetGroups[0].TargetGroupArn' --output text)
aws elbv2 register-targets --target-group-arn "$TG_ARN" --targets Id=i-0node1
CERT_ARN=$(aws acm request-certificate --domain-name '*.pbx.example.com' \
--validation-method DNS --query CertificateArn --output text)
# Create the DNS validation CNAME ACM reports, wait for ISSUED, then:
LISTENER_ARN=$(aws elbv2 create-listener --load-balancer-arn "$ALB_ARN" \
--protocol HTTPS --port 443 \
--ssl-policy ELBSecurityPolicy-TLS13-1-2-2021-06 \
--certificates CertificateArn="$CERT_ARN" \
--default-actions Type=forward,TargetGroupArn="$TG_ARN" \
--query 'Listeners[0].ListenerArn' --output text)
# Fleet growth: one target group + one host-header rule per node
# (node-addressed placement — ADR-0011):
aws elbv2 create-rule --listener-arn "$LISTENER_ARN" --priority 10 \
--conditions Field=host-header,Values=node-1.pbx.example.com \
--actions Type=forward,TargetGroupArn="$TG_ARN"
```
Point `*.pbx.example.com` (alias record) at the ALB. First call: identical to
[quickstart-docker.md](quickstart-docker.md) — browser at your domain, Twilio webhook at
`https://pbx.example.com/v1/trunk/webhook`.
The `--ssl-policy` above serves TLS 1.2+1.3 — never pick a 1.3-only policy (Twilio's TLS 1.3
client support is undocumented; see the invariants in
[certificates.md](certificates.md)).
## WebRTC media does not ride the ALB
The ALB carries HTTPS/WSS only. WebRTC media UDP goes **direct to each node**: give every
engine node an Elastic IP, open the media UDP range in the node's security group, and set
`RUTSTER_MEDIA_ADVERTISED_IP` to that EIP.
## Caps and traps
- **Hard cap: 100 target groups per ALB.** With one TG per node (host-header routing), that
is your fleet ceiling per ALB.
- **AWS fleet rotation can still cut a multi-hour call** — ALB infrastructure is replaced
under maintenance without regard for your WS lifetimes. Accepted residual risk; there is no
attribute for it.
- Health checks target `/readyz` (503 while draining or at the admission cap) — scale-in must
be drain-then-terminate: deregister the target, wait out
`RUTSTER_DRAIN_DEADLINE_SECS`, then terminate the instance.
````
- [ ] **Step 3: Re-run the content-check + link check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/aws.md \
&& grep -q 'idle_timeout.timeout_seconds' docs/deploy/aws.md \
&& grep -q '604800' docs/deploy/aws.md \
&& grep -q 'preserve_host_header' docs/deploy/aws.md \
&& grep -q '100 target groups' docs/deploy/aws.md \
&& grep -q '350-second' docs/deploy/aws.md \
&& echo CHECK-PASS || echo CHECK-FAIL
grep -oE '\]\([^)#]+\)' docs/deploy/aws.md | sed 's/](//;s/)//' \
| grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done
```
Expected: `CHECK-PASS`; BROKEN only for sibling docs not yet written.
- [ ] **Step 4: Commit**
```bash
cd /home/alee/Sources/rutster
git add docs/deploy/aws.md
git commit -s -m "docs(deploy): aws.md — ALB appendix: three mandatory overrides, never NLB-TLS (slice-G)
Docs-only appendix per ADR-0011 (never a shipped artifact). The three
mandatory ALB attribute overrides (idle_timeout=4000,
client_keep_alive=604800, preserve_host_header=true) with the telephony
reasons; the never-NLB-TLS rule with the 350s-silent-drop + no-HTTP-
context rationale; 100-target-group fleet ceiling; per-node EIPs for
direct media UDP; copy-paste aws elbv2 CLI path to first call."
```
---
### Task 6: `docs/deploy/reverse-proxies.md` — BYO nginx/HAProxy/Traefik
**Files:**
- Create: `docs/deploy/reverse-proxies.md`
- Test: bash content-check (Steps 1/3)
**Interfaces:**
- Consumes: spec §2.2 (BYO-proxy seam) + §10, brief §3(b) + §4 (runner-up table: HAProxy
`timeout tunnel`/#2280/`hard-stop-after`; Traefik pin-exact-versions #10601 / v2.11.2 /
#11405; nginx not-recommended rationale).
- Produces: the doc `topologies.md`/`quickstart-docker.md` link as `reverse-proxies.md`.
- [ ] **Step 1: Run the failing content-check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/reverse-proxies.md \
&& grep -q 'HAProxy' docs/deploy/reverse-proxies.md \
&& grep -qi 'pin' docs/deploy/reverse-proxies.md \
&& echo CHECK-PASS || echo CHECK-FAIL
```
Expected: `CHECK-FAIL`.
- [ ] **Step 2: Write the complete file**
Write `docs/deploy/reverse-proxies.md` with exactly this content:
````markdown
# Bring your own reverse proxy — tuned-timeout configs
Supported: disable the `caddy` service in the compose stack
([quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack)); the engine keeps its
plaintext `:8080` listener. Everything below terminates TLS in *your* proxy and forwards to
the engine.
## The universal 60-second footgun
**nginx, HAProxy, and Traefik all default their idle/read timeouts to ~60180 seconds — every
one of them will kill a quiet, perfectly healthy call WebSocket.** A Media Streams WS can
legitimately be near-silent in one direction for hours, and neither Twilio nor Telnyx sends
protocol-level keepalives. The engine ships app-level WS pings (`RUTSTER_WS_PING_SECS`,
default 20) as belt-and-braces, but your proxy timeouts must still exceed the maximum call
duration. Every snippet below does that.
## The contract any proxy must honor
1. **Publicly-trusted cert**, auto-renewed — no self-signed path exists
([certificates.md](certificates.md)).
2. **WS upgrade with no frame buffering and no connection-lifetime cap** — 20 ms audio frames,
calls up to 24 h.
3. **Honest `X-Forwarded-Proto` and `X-Forwarded-Host`** (including on the WS upgrade
request): `X-Twilio-Signature` is HMAC over the URL as Twilio saw it
([Twilio's SSL-termination guidance](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library)).
4. **Engine-side trust**: set `RUTSTER_TRUSTED_PROXIES` to your proxy's source IP/CIDR —
forwarded headers from unlisted sources are ignored, and signature validation will fail.
5. **TLS 1.2+1.3, mainstream ECDHE — never 1.3-only** (Twilio's 1.3 client support is
undocumented).
## nginx (works if you insist — not a recommended edge)
Why not recommended: no native DNS-01/wildcard support, and cert renewal requires a reload
whose old workers linger against hours-long WS connections (research basis: [TLS brief §4](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)). rutster ships this snippet and
nothing more for nginx.
```nginx
map $http_upgrade $connection_upgrade { default upgrade; '' close; }
server {
listen 443 ssl;
server_name pbx.example.com;
ssl_certificate /etc/letsencrypt/live/pbx.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/pbx.example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3; # never 1.3-only
location / {
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_buffering off; # 20 ms frames must not be coalesced
proxy_read_timeout 86400s; # the 60s default is the footgun
proxy_send_timeout 86400s;
}
}
```
## HAProxy (3.2 LTS or newer)
The best WS mental model of the classic family: after the upgrade, only `timeout tunnel`
governs the connection — **but only if you set it**; put it in `defaults` or it silently
doesn't apply to the paths you think it does (see
[haproxy#2280](https://github.com/haproxy/haproxy/issues/2280)). Two extra notes: `acme.sh`
can hot-load renewed certs through the stats socket with zero reload; and `hard-stop-after`
kills calls that are still draining — leave it unset or above your drain deadline.
```haproxy
defaults
mode http
timeout connect 5s
timeout client 75s
timeout server 75s
timeout tunnel 24h # governs upgraded WS; unset = the 60s-class footgun
frontend fe_rutster
bind :443 ssl crt /etc/haproxy/certs/ ssl-min-ver TLSv1.2
http-request set-header X-Forwarded-Proto https
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
default_backend be_rutster
backend be_rutster
server fob 127.0.0.1:8080
```
## Traefik v3 — PIN THE EXACT VERSION
Traefik has the best built-in DNS-01/wildcard of the classic family **and a track record of
breaking WebSockets in patch releases**:
[#10601](https://github.com/traefik/traefik/issues/10601), the v2.11.2 timeout-behavior flip,
and [#11405](https://github.com/traefik/traefik/issues/11405). **Pin the exact image tag and
never auto-update the edge.**
```yaml
# traefik.yml (static config)
entryPoints:
websecure:
address: ":443"
transport:
respondingTimeouts:
readTimeout: 0 # 0 disables; the 60s-class default kills quiet WS
idleTimeout: 0
writeTimeout: 0
certificatesResolvers:
le:
acme:
email: you@example.com
storage: /data/acme.json # persist this volume — certificates.md
dnsChallenge:
provider: cloudflare
```
```yaml
# dynamic config
http:
routers:
rutster:
rule: "Host(`pbx.example.com`)"
entryPoints: [websecure]
service: rutster
tls: { certResolver: le }
services:
rutster:
loadBalancer:
servers:
- url: "http://127.0.0.1:8080"
```
Traefik forwards `X-Forwarded-Proto/Host` by default — you still must list its address in
`RUTSTER_TRUSTED_PROXIES`.
## First call
Any of the above in front of the engine, then the same two first-call paths as
[quickstart-docker.md](quickstart-docker.md): browser at `https://pbx.example.com/`, Twilio
webhook at `https://pbx.example.com/v1/trunk/webhook`.
````
- [ ] **Step 3: Re-run the content-check + link check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/reverse-proxies.md \
&& grep -q 'proxy_read_timeout 86400s' docs/deploy/reverse-proxies.md \
&& grep -q 'timeout tunnel' docs/deploy/reverse-proxies.md \
&& grep -q 'PIN THE EXACT VERSION' docs/deploy/reverse-proxies.md \
&& grep -q 'RUTSTER_TRUSTED_PROXIES' docs/deploy/reverse-proxies.md \
&& grep -q '60' docs/deploy/reverse-proxies.md \
&& echo CHECK-PASS || echo CHECK-FAIL
grep -oE '\]\([^)#]+\)' docs/deploy/reverse-proxies.md | sed 's/](//;s/)//' \
| grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done
```
Expected: `CHECK-PASS`; BROKEN only for sibling docs not yet written.
- [ ] **Step 4: Commit**
```bash
cd /home/alee/Sources/rutster
git add docs/deploy/reverse-proxies.md
git commit -s -m "docs(deploy): reverse-proxies.md — BYO nginx/HAProxy/Traefik tuned-timeout configs (slice-G)
The universal 60s idle-timeout footgun stated up front; the five-point
contract any proxy must honor (incl. RUTSTER_TRUSTED_PROXIES + honest
X-Forwarded-* for signature validation); complete snippets: nginx
(explicitly not-recommended, snippet-only support), HAProxy (timeout
tunnel in defaults per #2280; hard-stop-after warning), Traefik (pin the
exact version — #10601 / v2.11.2 / #11405 WS-breaking patch history)."
```
---
### Task 7: `docs/deploy/certificates.md` — ACME paths, wildcard traps, `/data`
**Files:**
- Create: `docs/deploy/certificates.md`
- Test: bash content-check (Steps 1/3)
**Interfaces:**
- Consumes: spec §3.1§3.4, brief §2 invariants 1/7/8, §3(a)/(c), §5 risks 1/2/5; brief URLs
(twilio security page, errors 31910/11237, caddy #6420/#7222).
- Produces: the doc every other tree file links as `certificates.md`.
- [ ] **Step 1: Run the failing content-check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/certificates.md \
&& grep -q 'duplicate' docs/deploy/certificates.md \
&& grep -q '/data' docs/deploy/certificates.md \
&& echo CHECK-PASS || echo CHECK-FAIL
```
Expected: `CHECK-FAIL`.
- [ ] **Step 2: Write the complete file**
Write `docs/deploy/certificates.md` with exactly this content:
````markdown
# Certificates & ACME — the availability-critical subsystem
**A publicly-CA-trusted certificate is a hard CPaaS requirement; no self-signed path exists.**
Twilio refuses self-signed certs for webhooks and Media Streams (error
[31910](https://www.twilio.com/docs/api/errors/31910); see
[twilio.com/docs/usage/security](https://www.twilio.com/docs/usage/security)) — and the
console's cert-validation toggle is account-wide, webhook-only, and dev-only, so it is not an
escape hatch. In a 24/7 product whose calls are non-migratable there is no maintenance window:
**auto-renewal is availability-critical, not a nicety.**
Protocol posture everywhere: TLS 1.2+1.3, mainstream ECDHE suites, **never 1.3-only**
(Twilio's TLS 1.3 client support is undocumented), and never pin Twilio's certificates.
## The `/data` volume — read this before anything else
Caddy keeps its certificates and ACME account state in `/data`. If you recreate the container
without that volume, Caddy re-requests the same certificate — and Let's Encrypt's
**duplicate-certificate limit (5 per week for an identical name set,
[letsencrypt.org/docs/rate-limits](https://letsencrypt.org/docs/rate-limits/))** will lock
your domain out for up to a week. The failure class is **total inbound outage**: 31910 on
Media Streams, [11237](https://www.twilio.com/docs/api/errors/11237) on webhooks, no calls in
or out of the trunk. A container recreate loop (crash-looping deploy, CI that recreates on
every push) burns all five in minutes.
- Always mount `/data` on a persistent volume (`-v rutster-caddy-data:/data`).
- Back it up like state, because it is:
`docker run --rm -v rutster-caddy-data:/data -v "$PWD":/backup debian:stable-slim tar czf /backup/caddy-data.tgz /data`
## ACME challenge paths
| Path | When | Notes |
|---|---|---|
| **HTTP-01** (default) | Port 80 publicly reachable, single hostname | Zero config beyond the domain + email. |
| **DNS-01** | Wildcards (`*.pbx.domain`), CGNAT/behind-NAT hosts, no port 80 | The bundled `rutster-edge` Caddy build ships a curated plugin set: **cloudflare, route53, porkbun, hetzner, desec** (duckdns excluded — no license file). Any other DNS provider requires your own xcaddy build. DNS API credentials live in the container env — scope them to the zone. |
| **BYO-cert** (in-process rustls, Phase 1) | You already have cert distribution (corporate ACME, central wildcard issuance) | Set `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY`; the engine serves TLS itself and **hot-reloads on file change without dropping live WS**. You own renewal delivery. |
In-binary ACME (Phase 2) is deliberately deferred behind named triggers — among them Let's
Encrypt's dns-persist-01 reaching GA, which would dissolve the DNS-plugin matrix entirely. See
[ADR-0011](../adr/0011-deployment-topology.md).
## Caddy reload vs live calls
Routine renewals are safe: Caddy swaps certs **in memory, zero-downtime — the routine periodic
event drops nothing.** Config *reloads* (editing the Caddyfile) are the risk: the mitigation
(`stream_close_delay` above max call duration) has an open upstream bug trail
([caddy#6420](https://github.com/caddyserver/caddy/issues/6420),
[caddy#7222](https://github.com/caddyserver/caddy/issues/7222)), which is why the shipped
artifacts carry a CI e2e test for config-reload-during-live-call. Operator rule: **do not edit
the Caddyfile while calls are live** unless your image version passed that test; drain first
(`/readyz` returns 503 while draining).
## Fleet certificate patterns (T3)
Node-addressed placement means every node needs its own publicly-valid TLS name — this is
CPaaS-imposed (`<Stream>` URLs route on hostname only), not a proxy choice. Two blessed
patterns:
1. **Wildcard `*.pbx.domain`** via DNS-01, issued **centrally** and distributed to nodes.
Understand the trade: the wildcard private key on every node means one compromised node
burns the whole namespace.
2. **Per-node distinct certs** (`node-1.pbx.domain`, `node-2.…`) — no shared key material.
Renewals are exempt from Let's Encrypt's per-domain limits, so this scales with fleet size.
Traps:
- **N nodes independently requesting the identical wildcard** are N duplicate requests — the
5/week duplicate-cert limit locks the fleet out. Central issuance or distinct names only.
- **Caddy on-demand TLS is rejected for node names**: the first TLS handshake to a fresh name
blocks for seconds on issuance, colliding with the sub-second webhook budget (Twilio's hard
cap is 15 s, UX budget sub-second), and its rate-limit knobs are deprecated. Node names are
known at provision time — pre-issue their certs.
## Troubleshooting
| Symptom | Fix |
|---|---|
| Twilio 31910 on Media Streams | Cert not publicly trusted / expired / SNI mismatch. Check `openssl s_client -connect pbx.example.com:443 -servername pbx.example.com`. |
| `rateLimited` / `too many certificates` in Caddy logs | You hit the duplicate-cert limit — almost always a lost `/data` volume. Restore the volume or wait out the window; then fix the volume mount so it never recurs. |
| DNS-01 fails for your provider | Provider not in the bundled plugin set — use a delegated zone on a supported provider, or build your own xcaddy image. |
````
- [ ] **Step 3: Re-run the content-check + link check**
```bash
cd /home/alee/Sources/rutster
test -f docs/deploy/certificates.md \
&& grep -q '5 per week' docs/deploy/certificates.md \
&& grep -q 'errors/31910' docs/deploy/certificates.md \
&& grep -q 'errors/11237' docs/deploy/certificates.md \
&& grep -q 'on-demand TLS is rejected' docs/deploy/certificates.md \
&& grep -q 'RUTSTER_TLS_CERT' docs/deploy/certificates.md \
&& grep -q 'never 1.3-only' docs/deploy/certificates.md \
&& echo CHECK-PASS || echo CHECK-FAIL
grep -oE '\]\([^)#]+\)' docs/deploy/certificates.md | sed 's/](//;s/)//' \
| grep -v '^http' | while read -r p; do [ -e "docs/deploy/$p" ] || echo "BROKEN: $p"; done
```
Expected: `CHECK-PASS`; no BROKEN lines if Tasks 16 have run.
- [ ] **Step 4: Commit**
```bash
cd /home/alee/Sources/rutster
git add docs/deploy/certificates.md
git commit -s -m "docs(deploy): certificates.md — ACME paths, /data persistence, wildcard traps (slice-G)
The availability-critical subsystem: no-self-signed-path (Twilio 31910),
the /data-loss -> LE 5/week duplicate-cert lockout -> total-inbound-outage
chain (31910 streams + 11237 webhooks) with backup command; HTTP-01 vs
DNS-01 (bundled plugin set: cloudflare/route53/porkbun/hetzner/desec,
duckdns excluded) vs BYO-cert rustls Phase 1 (RUTSTER_TLS_CERT/KEY,
hot-reload); Caddy renewal-vs-reload distinction with the #6420/#7222
caveat; fleet patterns (central wildcard vs per-node certs) with the
identical-wildcard trap and the on-demand-TLS rejection."
```
---
### Task 8: `docs/QUICKSTART.md` — point at the Docker fast path (minimal diff)
**Files:**
- Modify: `docs/QUICKSTART.md:13-15` (insert one blockquote between the status blockquote and
the `---` rule; verified against current content — the file currently flows from the status
blockquote ending `…for the active build target's design.` straight into `---` /
`## Prerequisites`)
- Test: bash content-check (Steps 1/3)
**Interfaces:**
- Consumes: `docs/deploy/quickstart-docker.md` (Task 3 — must exist first).
- Produces: nothing downstream.
- [ ] **Step 1: Run the failing content-check**
```bash
cd /home/alee/Sources/rutster
grep -q 'deploy/quickstart-docker.md' docs/QUICKSTART.md && echo CHECK-PASS || echo CHECK-FAIL
```
Expected: `CHECK-FAIL`.
- [ ] **Step 2: Apply the minimal edit**
In `docs/QUICKSTART.md`, find this exact text (end of the status blockquote, lines 1215):
```markdown
> [`docs/superpowers/specs/2026-07-05-slice-5-rented-transport-design.md`](superpowers/specs/2026-07-05-slice-5-rented-transport-design.md)
> for the active build target's design.
---
```
Replace with:
```markdown
> [`docs/superpowers/specs/2026-07-05-slice-5-rented-transport-design.md`](superpowers/specs/2026-07-05-slice-5-rented-transport-design.md)
> for the active build target's design.
> **Fastest path:** if you want a *running deployment* — TLS, a domain, a phone
> number — skip the source build and use the Docker quickstart:
> [`docs/deploy/quickstart-docker.md`](deploy/quickstart-docker.md). This page
> is the from-source developer path.
---
```
No other changes to the file.
- [ ] **Step 3: Re-run the content-check + link check**
```bash
cd /home/alee/Sources/rutster
grep -q 'deploy/quickstart-docker.md' docs/QUICKSTART.md \
&& test -e docs/deploy/quickstart-docker.md \
&& echo CHECK-PASS || echo CHECK-FAIL
git diff --stat docs/QUICKSTART.md # expect: 1 file changed, ~6 insertions(+), 0 deletions(-)
```
Expected: `CHECK-PASS`; the diffstat shows insertions only (minimal diff honored).
- [ ] **Step 4: Commit**
```bash
cd /home/alee/Sources/rutster
git add docs/QUICKSTART.md
git commit -s -m "docs: QUICKSTART points at deploy/quickstart-docker.md as the fast path (slice-G)
Minimal diff: one blockquote after the status note. The from-source page
stays the developer path; the Docker quickstart is now the operator fast
path (slice-G docs tree)."
```
---
## Final acceptance checklist
After all 8 tasks:
- [ ] **Spec §10 acceptance bar, checked doc by doc:** `quickstart-docker.md` (copy-paste T1 +
T2 to first browser call and first PSTN call), `homelab.md` (copy-paste ngrok dev path +
VPS/WireGuard graduation, **plus** the honest CGNAT-WebRTC impossibility statement),
`aws.md` (copy-paste `aws elbv2` path), `reverse-proxies.md` (three complete configs +
first-call pointer), `topologies.md` (delegates copy-paste to quickstart; states T3
cannot be deployed today), `certificates.md` (supporting doc; backup + troubleshooting
commands are copy-paste).
- [ ] **All relative links resolve across the whole set** (no BROKEN output):
```bash
cd /home/alee/Sources/rutster
for f in docs/adr/0011-deployment-topology.md docs/deploy/*.md docs/QUICKSTART.md; do
d=$(dirname "$f")
grep -oE '\]\([^)#]+\)' "$f" | sed 's/](//;s/)//' | grep -v '^http' \
| while read -r p; do [ -e "$d/$p" ] || echo "BROKEN in $f: $p"; done
done
```
- [ ] **ADR-0011 format audit:** header bullet-list (Status/Date/Closes/Origin/Related) →
Context → Decision → Consequences → References, matching ADR-0007/0008; Status is
`Proposed` (ratification is the maintainer's move, not this slice's).
- [ ] **Slice-B reconciliation done or explicitly deferred:** if `deploy/` exists, image
names, service names, `.env.example` var names, and the registry namespace in the docs
match it; if not, the artifact-names note box in `quickstart-docker.md` is present.
- [ ] **Repo gates still green** (docs can't break them; prove it anyway):
```bash
cd /home/alee/Sources/rutster
cargo fmt --all --check && cargo clippy --all --all-targets -- -D warnings && cargo test --all && cargo deny check
git diff main --name-only | grep -E 'crates/rutster-media/src/(loop_driver|rtc_session)\.rs' && echo "SEAM VIOLATION" || echo "seam intact"
```
- [ ] Eight commits, each signed off (`git log --format='%h %s' -8` shows the slice-G
series; `git log -8 | grep -c 'Signed-off-by'` = 8).
- [ ] PR opened via `tea` per house workflow; the PR description notes: **ADR-0011 lands as
Proposed — ratification discussion is a separate step**, and names the plan-B
reconciliation status.