From 0c307acdb5ed7f37bb0203d983ae9526156d98a2 Mon Sep 17 00:00:00 2001 From: "Aaron D. Lee" Date: Mon, 6 Jul 2026 04:07:15 +0000 Subject: [PATCH] deploy slice G (docs + ADR-0011): docs/deploy/ tree + ADR-0011 Proposed (#23) Co-authored-by: Aaron D. Lee Co-committed-by: Aaron D. Lee --- docs/QUICKSTART.md | 5 + docs/adr/0011-deployment-topology.md | 249 +++++++++++++++++++++++++++ docs/deploy/aws.md | 99 +++++++++++ docs/deploy/certificates.md | 79 +++++++++ docs/deploy/homelab.md | 132 ++++++++++++++ docs/deploy/quickstart-docker.md | 104 +++++++++++ docs/deploy/reverse-proxies.md | 139 +++++++++++++++ docs/deploy/topologies.md | 105 +++++++++++ 8 files changed, 912 insertions(+) create mode 100644 docs/adr/0011-deployment-topology.md create mode 100644 docs/deploy/aws.md create mode 100644 docs/deploy/certificates.md create mode 100644 docs/deploy/homelab.md create mode 100644 docs/deploy/quickstart-docker.md create mode 100644 docs/deploy/reverse-proxies.md create mode 100644 docs/deploy/topologies.md diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index 282075d..1527ab0 100644 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -12,6 +12,11 @@ Get Rutster running and hear your own voice echoed back in under 5 minutes. > [`docs/superpowers/specs/2026-07-05-slice-5-rented-transport-design.md`](superpowers/specs/2026-07-05-slice-5-rented-transport-design.md) > for the active build target's design. +> **Fastest path:** if you want a *running deployment* — TLS, a domain, a phone +> number — skip the source build and use the Docker quickstart: +> [`docs/deploy/quickstart-docker.md`](deploy/quickstart-docker.md). This page +> is the from-source developer path. + --- ## Prerequisites diff --git a/docs/adr/0011-deployment-topology.md b/docs/adr/0011-deployment-topology.md new file mode 100644 index 0000000..6dec96f --- /dev/null +++ b/docs/adr/0011-deployment-topology.md @@ -0,0 +1,249 @@ +# ADR-0011 — Deployment topology: one binary, three blessed shapes + +- **Status:** Proposed +- **Date:** 2026-07-05 +- **Closes:** the deployment-topology ADR reserved by [ADR-0008](0008-fob-and-green-zone.md)'s + worked example ("the deployment topology itself may get its own ADR once that design closes"). +- **Origin:** [2026-07-05 deployment-topology spec](../superpowers/specs/2026-07-05-deployment-topology-design.md) + §2–§4. The TLS/edge fork was researched via a six-family survey — see the companion + [TLS/edge decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md). +- **Related:** [ADR-0002](0002-north-star-and-fused-core.md) (the fused per-call vertical no + topology may split), [ADR-0005](0005-event-bus.md) (Valkey = bus + KV + presence; media never + rides the bus; the bus is never billing source-of-truth), [ADR-0006](0006-ingress-posture.md) + (two inbound protocols only; the tap is egress-only), [ADR-0007](0007-trunk-rented-transport.md) + (rented transport ⇒ public HTTPS/WSS reachability is non-negotiable), + [ADR-0008](0008-fob-and-green-zone.md) (the FOB/green-zone rule this ADR applies to + deployment), [ADR-0009](0009-spend-gate-honest-rescope.md) (ledger trait: in-memory + single-node, Valkey-backed fleets), [ADR-0010](0010-spearhead-benchmark-sim-harness.md) + (sequencing this ADR must not reorder), README pillar 6 ("one binary, one bus, one deploy"). + +## Context + +[ADR-0008](0008-fob-and-green-zone.md) made the build-vs-reuse doctrine physical in a worked +example — the all-in-one container — and reserved the topology ADR for when the design closed. +Two things have since closed it: the rented transport +([ADR-0007](0007-trunk-rented-transport.md)) is merged, so every deployment must now present a +public HTTPS/WSS edge to a CPaaS; and slice-5/seams landed the operational primitives a +deployment story needs (drain lifecycle, `/readyz`, admission cap, `MediaAddressConfig`, +`EventSink`). + +The CPaaS side imposes a hard external spec on any edge rutster ships (researched in the +[decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md); restated here +because these invariants are load-bearing for every decision below): + +1. Publicly-CA-trusted cert; **no self-signed path exists** (Twilio error 31910). Auto-renewal + is availability-critical. +2. wss:// on 443, open to the whole internet: Twilio publishes no egress IPs and offers no + mTLS. Auth is application-layer only (HMAC signatures). +3. One WS per call, up to 24 h, 50 msg/s/direction at 20 ms cadence; zero tolerance for frame + buffering, connection-lifetime caps, or write-side idle timers. Neither Twilio nor Telnyx + documents WS keepalive — keepalive is entirely our job. +4. `` URLs carry **no query strings** — wss routing keys on hostname/path only. +5. `X-Twilio-Signature` is HMAC over the URL *as Twilio saw it*: the edge must forward + `X-Forwarded-Proto/Host` honestly and the FOB must reconstruct the public URL. +6. Webhook budget: sub-second for UX, 15 s hard cap — no blocking issuance in the handshake + path. +7. TLS 1.2+1.3, mainstream ECDHE, never 1.3-only; never pin Twilio certs. + +Design for Twilio; Telnyx/Vonage come for free (strictly less demanding, plus optional +source-IP allowlisting as extra hardening Twilio can't use). + +On the other side, the FOB's own physics constrain every shape: per-call state is in-process +and non-migratable (the fused vertical, [ADR-0002](0002-north-star-and-fused-core.md)), WebRTC +media is per-session UDP direct to the process, and calls run for hours. Any topology that +assumes migratable, short-lived, connectionless workloads is disqualified before it starts. + +## Decision + +**The FOB never decomposes; topology is a configuration property, not a code property.** The +same binary runs in every blessed shape; only `RUTSTER_*` env and what is deployed alongside it +differ. "Microservices" in rutster means *green-zone services get their own +processes/containers* — never that the per-call vertical splits +([ADR-0002](0002-north-star-and-fused-core.md)). + +### The three blessed shapes + +#### T1 — Solo (all-in-one container) — shipped artifact + +One image (`rutster-allinone`), s6-overlay as PID 1 supervising four processes: + +| Process | Role | Binding | +|---|---|---| +| `caddy` | Edge: ACME, TLS termination, WS proxy to FOB | :443/:80 public | +| `rutster` (FOB) | Engine | 127.0.0.1:8080 (behind Caddy); media UDP direct | +| `rutster-brain-realtime` | Brain | 127.0.0.1:8082 loopback tap (as today) | +| `valkey-server` | Dark on day one except `EventSink` | 127.0.0.1:6379 | + +Two volumes, both non-negotiable: `/data` (Caddy cert/ACME state — loss risks the Let's +Encrypt duplicate-cert lockout, a total-inbound-outage class of failure) and `/var/lib/valkey` +(stream/state persistence). WebRTC media UDP goes **direct to the FOB**, never through Caddy: +host networking recommended; otherwise published `RUTSTER_MEDIA_PORT_RANGE` + +`RUTSTER_MEDIA_ADVERTISED_IP` (NAT 1:1 model, no STUN — unchanged from slice-5/seams). + +Valkey ships bundled from v1 **before most consumers exist**: the operator contract (volumes, +ports, upgrade shape) is stable from day one, so the spend ledger +([ADR-0009](0009-spend-gate-honest-rescope.md)) and the fleet directory land later without +changing the deployment shape. Cost accepted: ~15 MB and one mostly-idle process. + +#### T2 — Modular (compose stack) — shipped artifact, the reference deployment + +Services: `caddy` (image `rutster-edge`), `engine` (`rutster-engine`), `brain` +(`rutster-brain`), `valkey` (upstream `valkey/valkey`). + +- **The brain shares the engine's network namespace** (`network_mode: "service:engine"`) so + the loopback-only tap posture (`resolve_tap_url` rejects non-loopback until step 6) survives + unchanged. The brain graduates to its own netns when the wss:// tap lands (step 6). +- Valkey is its own service on the compose network — network-near, satisfying + [ADR-0005](0005-event-bus.md)'s sub-ms rule. +- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` (grace > drain). +- BYO-proxy supported: disable the `caddy` service; the engine keeps its plaintext `:8080` + mode; tuned-timeout snippets for nginx/HAProxy/Traefik ship in + [`docs/deploy/reverse-proxies.md`](../deploy/reverse-proxies.md). + +#### T3 — Fleet — ADR chapter only (paper; no code in this epoch) + +N **symmetric** nodes, each running the T1 process set *minus* valkey, plus a shared green +zone: one Valkey (presence + directory + the [ADR-0009](0009-spend-gate-honest-rescope.md) +spend ledger) and object storage (named as required, still deferred). Wildcard DNS +`*.pbx.domain`; per-node certs (see the fleet cert story below). The per-node admission cap is +the [ADR-0010](0010-spearhead-benchmark-sim-harness.md) benchmark number (replacing the +`RUTSTER_MAX_SESSIONS=64` placeholder). Scale-in is drain-then-terminate only; spot instances +are rejected for the engine tier (2026-07-04 review — a reclaimed node kills every +non-migratable call on it). + +### Edge doctrine + +#### Bundled edge: Caddy (both shipped artifacts, same build, same Caddyfile) + +Custom xcaddy build (the `rutster-edge` image): Caddy core (Apache-2.0) + a curated DNS-01 +plugin set — cloudflare, route53, porkbun, hetzner, desec (duckdns excluded: no license file). +Zero-downtime in-memory cert renewal; verified protocol-unaware WS tunneling (no frame +buffering). Config is a ~6-line Caddyfile with tuned timeouts, honest `X-Forwarded-*`, and +`stream_close_delay` above max call duration — **plus a CI e2e case for +config-reload-during-live-call, because the upstream mitigation has an open bug trail +(caddy #6420/#7222) and is not trusted untested.** + +#### In-process TLS: staged — the ratified end-state, not adopted wholesale now + +[ADR-0008](0008-fob-and-green-zone.md) claims the trust-domain edge as security-constitutive +FOB territory, and str0m already terminates DTLS-SRTP in-process on the same aws-lc-rs +provider — "keep TLS out of the FOB" is not the objection. The objection is economics: owning +an ACME renewal state machine forever, in a 45-day-cert world. + +- **Phase 1 (this epoch):** `axum-server` + rustls with operator-supplied certs + (`RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY`), hot-reload without dropping live WS. + Plaintext-behind-Caddy stays the artifact default. +- **Phase 2 (in-binary ACME) triggers, named:** (a) Let's Encrypt dns-persist-01 confirmed GA + (closes the wildcard/CGNAT issuance gap; instant-acme already parses it); (b) a field + incident where Caddy reload/`stream_close_delay` drops live calls; (c) fleet or + VPS-forwarder topology making encrypted-to-the-binary a selling point; (d) maintenance + fatigue with the curated xcaddy plugin build. Until a trigger fires, Phase 2 is deliberately + not built — no turnkey ACME crate fits the tree today (rustls-acme conflicts with the + deny.toml rcgen/x509-parser posture). + +#### Fleet cert story + +Per-node public TLS names are CPaaS-imposed (invariant 4 + node-addressed placement). Two +blessed patterns: + +1. **Wildcard `*.pbx.domain` via DNS-01** with central issuance/distribution. +2. **Per-node distinct certs** (`node-N.domain`) where the wildcard private key must not live + on every node — one compromised node otherwise burns the whole namespace. + +Traps, stated so nobody re-discovers them in production: N nodes independently requesting the +*identical* wildcard hit the Let's Encrypt duplicate-cert limit (5/week) — distinct names or +central distribution only. **Caddy on-demand TLS is rejected for node names**: the first +handshake to a fresh name blocks seconds on issuance, colliding with the sub-second webhook +budget (invariant 6), and its rate-limit knobs are deprecated. Node names known at provision +time get pre-issued certs. + +### Fleet placement paradigm (the trunk webhook handler and session API grow *toward* this; no fleet code in this epoch) + +**Node-addressed URLs + Valkey presence; no router tier.** (Decision 2026-07-05 over the two +rejected alternatives below.) + +- Nodes heartbeat capacity into Valkey (TTL'd presence keys — the + [ADR-0005](0005-event-bus.md) presence layer). +- Placement happens **at answer time** by whichever node the request lands on (`api.domain`, + DNS round-robin): read fleet capacity → pick owner → return TwiML whose Stream URL is + `wss://node-K.pbx.domain/...`. The media WS dials the owning node directly — zero extra + hops, consistent with connection-follows-ownership. Twilio's answer-time Stream URL is + purpose-shaped for this: invariant 4 permits hostname routing. +- WebRTC: the create-session response carries the owner's node-addressed base URL + advertised + media IP (UDP is already direct). +- Misdirected control ops (e.g. a DELETE hitting a non-owner) consult the Valkey directory and + **307-redirect** to the owner. +- **Degraded mode (Valkey dark): the answering node self-places if it has capacity**; presence + heals via TTL on recovery. Fail-safe degradation, consistent with + [ADR-0009](0009-spend-gate-honest-rescope.md)'s posture. +- Single-node is the degenerate case of the same code: a one-entry directory. + +**Rejected: a routing tier that proxies everything.** It would put every 20 ms media frame +through an extra userspace hop and re-create exactly the SPOF the symmetric design avoids — +and it buys nothing, because the CPaaS already lets the answer-time URL do the routing. + +**Rejected: a distinguished control node.** It breaks node symmetry (two artifacts, two +failure stories, a promotion protocol) to solve a problem the presence directory already +solves; placement needs fleet state either way, and Valkey is already the fleet-state home +([ADR-0005](0005-event-bus.md)). + +### Rejected shapes + +- **Decomposing the FOB into services** — rejected **permanently**, not deferred. + Media/reflex/tap/spend stay one process, one trust domain + ([ADR-0002](0002-north-star-and-fused-core.md)). +- **Serverless.** Lambdas are categorically unfit for the engine: non-migratable multi-hour + calls, per-session UDP sockets, in-process media state. Webhook-on-Lambda buys nothing + because placement needs fleet state anyway. +- **Tunnels in production.** Cloudflare Tunnel and ngrok see plaintext audio at the vendor + edge (an unconsented subprocessor — DPA/BAA/PCI failure), document mid-call WS kills, and + cloudflared has an open Twilio handshake bug (cloudflared #1465). ngrok is the blessed + 5-minute demo path (the only tunnel with a proven Media Streams record); Tailscale Funnel is + the privacy-clean single-user demo (TLS terminates on-node), unsizable beyond that. **No + tunnel carries inbound UDP → homelab behind CGNAT is PSTN-only, period.** The production + graduation path is a cheap VPS + WireGuard as a dumb TCP forwarder with TLS terminating at + home, or running the engine on the VPS — see + [`docs/deploy/homelab.md`](../deploy/homelab.md). +- **NLB TLS listeners — never.** Fixed 350 s idle timeout that silently drops flows, and no + HTTP context, so `X-Forwarded-*` headers for signature validation (invariant 5) don't exist. + The ALB path is documented as a docs-only appendix + ([`docs/deploy/aws.md`](../deploy/aws.md)), never a shipped artifact. +- **k8s manifests/Helm.** Compose-first (README pillar 6). k8s notes live in docs only (drain + vs `terminationGracePeriodSeconds`); manifests are a later rung if demand appears. + +### Deferred, named (so the boundary is deliberate) + +- **T3 fleet implementation** (presence heartbeats, directory redirects, placement code). +- **The zero-egress / air-gap profile** (SBC layer-2 ingress per + [ADR-0007](0007-trunk-rented-transport.md) layer 2 + a self-hosted brain): a defined + profile, no code this epoch. T1 is about one-command simplicity, not isolation. +- **In-binary ACME (rustls Phase 2)** — behind the four named triggers above. +- **Durable CDR pipeline**: `ValkeyEventSink` is evidence preservation on a capped stream — + explicitly NOT the billing ledger ([ADR-0005](0005-event-bus.md) source-of-truth rule). + CDR → object storage stays deferred. +- **Multi-tenancy** (needs its own ADR before schemas ossify); wss:// tap + brain fleet + + resume tokens (step 6); media-thread sharding graduation; arm64 images. + +## Consequences + +- **Positive:** one binary and one mental model across all three shapes — an operator's + knowledge transfers from `docker run` to a fleet; the CPaaS invariants are satisfied by a + decade-hardened ACME loop (Caddy) rather than a DIY renewal state machine; the fleet + paradigm needs zero new infrastructure beyond what [ADR-0005](0005-event-bus.md) already + bundles; the rejected shapes are written down, so nobody burns a week discovering that + Lambda or NLB-TLS can't work. +- **Negative:** the curated xcaddy DNS-plugin set is a permanent maintenance surface until + dns-persist-01 GA; Caddy config-reload behavior vs live calls is mitigated + (`stream_close_delay` + CI e2e) but has an open upstream bug trail; homelab WebRTC behind + CGNAT is unsolved **by design** — only the VPS graduation fixes it, and the docs must say so + loudly or operators will read "homelab supported" as including browser calls. +- **Mitigation:** the Phase-2 triggers convert both negatives into a planned exit; the + `docs/deploy/` tree ships with a copy-paste-to-first-call acceptance bar so the honest + limits are in the operator's face, not in a footnote. + +## References + +- [2026-07-05 deployment-topology spec](../superpowers/specs/2026-07-05-deployment-topology-design.md) — §2–§4 are this ADR's source +- [TLS/edge decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md) — six-family survey + the CPaaS ground truth (external citations live there) +- [ADR-0002](0002-north-star-and-fused-core.md) · [ADR-0005](0005-event-bus.md) · [ADR-0006](0006-ingress-posture.md) · [ADR-0007](0007-trunk-rented-transport.md) · [ADR-0008](0008-fob-and-green-zone.md) · [ADR-0009](0009-spend-gate-honest-rescope.md) · [ADR-0010](0010-spearhead-benchmark-sim-harness.md) +- The `docs/deploy/` tree: [topologies](../deploy/topologies.md) · [quickstart-docker](../deploy/quickstart-docker.md) · [homelab](../deploy/homelab.md) · [aws](../deploy/aws.md) · [reverse-proxies](../deploy/reverse-proxies.md) · [certificates](../deploy/certificates.md) diff --git a/docs/deploy/aws.md b/docs/deploy/aws.md new file mode 100644 index 0000000..b9c16c3 --- /dev/null +++ b/docs/deploy/aws.md @@ -0,0 +1,99 @@ +# AWS appendix — ALB + ACM (docs only, never a shipped artifact) + +Scope: operators **already on EC2**. There, an ALB is trust-neutral (the engine is on AWS +infrastructure regardless) and ACM renewal is genuinely zero-touch. This page is an appendix, +not a recommendation to move to AWS — the shipped edges are Caddy +([topologies.md](topologies.md)) and BYO proxies ([reverse-proxies.md](reverse-proxies.md)). +All ALB/NLB facts below are grounded in the [TLS/edge decision brief §3(c)](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md). + +## Never NLB-TLS + +Do not put the engine behind an NLB TLS listener. Two disqualifiers, ratified in +[ADR-0011](../adr/0011-deployment-topology.md): + +1. **Fixed 350-second idle timeout** that silently drops flows — any quiet WebSocket (hold, + parked call, unidirectional stream) dies mid-call with no close frame. +2. **No HTTP context**: an NLB adds no `X-Forwarded-Proto/Host`, so the engine cannot + reconstruct the public URL that `X-Twilio-Signature` is HMAC'd over + ([Twilio's SSL-termination guidance](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library)) + — genuine traffic fails validation. + +## The three mandatory ALB attribute overrides + +The defaults kill telephony. All three overrides are **mandatory**, not tuning: + +| Attribute | Value | Why | +|---|---|---| +| `idle_timeout.timeout_seconds` | `4000` | The default 60 s kills any quiet WS. 4000 is the ALB maximum — still finite, so the engine's app-level WS pings (`RUTSTER_WS_PING_SECS`, default 20) remain load-bearing here. | +| `client_keep_alive.seconds` | `604800` | The 3600 s default can terminate the client connection under a **>1-hour call**. Set to the 7-day maximum. | +| `routing.http.preserve_host_header.enabled` | `true` | Signature validation reconstructs the URL *as Twilio saw it* — the engine must receive the honest public `Host`, and its `RUTSTER_TRUSTED_PROXIES` must include the ALB's subnet CIDRs. | + +## Copy-paste path (CLI) + +Assumes: engine node(s) running per [quickstart-docker.md](quickstart-docker.md) with the +`caddy` service disabled (plaintext `:8080`), a VPC, two subnets, and a security group +allowing `:443` from anywhere and `:8080` from the ALB. + +```bash +VPC=vpc-0123456789abcdef0 +SUBNETS="subnet-0aaa subnet-0bbb" +SG=sg-0ccc + +ALB_ARN=$(aws elbv2 create-load-balancer --name rutster-edge --type application \ + --subnets $SUBNETS --security-groups $SG \ + --query 'LoadBalancers[0].LoadBalancerArn' --output text) + +# THE THREE MANDATORY OVERRIDES — do not skip: +aws elbv2 modify-load-balancer-attributes --load-balancer-arn "$ALB_ARN" --attributes \ + Key=idle_timeout.timeout_seconds,Value=4000 \ + Key=client_keep_alive.seconds,Value=604800 \ + Key=routing.http.preserve_host_header.enabled,Value=true + +TG_ARN=$(aws elbv2 create-target-group --name rutster-node-1 \ + --protocol HTTP --port 8080 --vpc-id "$VPC" \ + --health-check-path /readyz \ + --query 'TargetGroups[0].TargetGroupArn' --output text) +aws elbv2 register-targets --target-group-arn "$TG_ARN" --targets Id=i-0node1 + +CERT_ARN=$(aws acm request-certificate --domain-name '*.pbx.example.com' \ + --validation-method DNS --query CertificateArn --output text) +# Create the DNS validation CNAME ACM reports, wait for ISSUED, then: + +LISTENER_ARN=$(aws elbv2 create-listener --load-balancer-arn "$ALB_ARN" \ + --protocol HTTPS --port 443 \ + --ssl-policy ELBSecurityPolicy-TLS13-1-2-2021-06 \ + --certificates CertificateArn="$CERT_ARN" \ + --default-actions Type=forward,TargetGroupArn="$TG_ARN" \ + --query 'Listeners[0].ListenerArn' --output text) + +# Fleet growth: one target group + one host-header rule per node +# (node-addressed placement — ADR-0011): +aws elbv2 create-rule --listener-arn "$LISTENER_ARN" --priority 10 \ + --conditions Field=host-header,Values=node-1.pbx.example.com \ + --actions Type=forward,TargetGroupArn="$TG_ARN" +``` + +Point `*.pbx.example.com` (alias record) at the ALB. First call: identical to +[quickstart-docker.md](quickstart-docker.md) — browser at your domain, Twilio webhook at +`https://pbx.example.com/v1/trunk/webhook`. + +The `--ssl-policy` above serves TLS 1.2+1.3 — never pick a 1.3-only policy (Twilio's TLS 1.3 +client support is undocumented; see the invariants in +[certificates.md](certificates.md)). + +## WebRTC media does not ride the ALB + +The ALB carries HTTPS/WSS only. WebRTC media UDP goes **direct to each node**: give every +engine node an Elastic IP, open the media UDP range in the node's security group, and set +`RUTSTER_MEDIA_ADVERTISED_IP` to that EIP. + +## Caps and traps + +- **Hard cap: 100 target groups per ALB.** With one TG per node (host-header routing), that + is your fleet ceiling per ALB. +- **AWS fleet rotation can still cut a multi-hour call** — ALB infrastructure is replaced + under maintenance without regard for your WS lifetimes. Accepted residual risk; there is no + attribute for it. +- Health checks target `/readyz` (503 while draining or at the admission cap) — scale-in must + be drain-then-terminate: deregister the target, wait out + `RUTSTER_DRAIN_DEADLINE_SECS`, then terminate the instance. diff --git a/docs/deploy/certificates.md b/docs/deploy/certificates.md new file mode 100644 index 0000000..8d5d15a --- /dev/null +++ b/docs/deploy/certificates.md @@ -0,0 +1,79 @@ +# Certificates & ACME — the availability-critical subsystem + +**A publicly-CA-trusted certificate is a hard CPaaS requirement; no self-signed path exists.** +Twilio refuses self-signed certs for webhooks and Media Streams (error +[31910](https://www.twilio.com/docs/api/errors/31910); see +[twilio.com/docs/usage/security](https://www.twilio.com/docs/usage/security)) — and the +console's cert-validation toggle is account-wide, webhook-only, and dev-only, so it is not an +escape hatch. In a 24/7 product whose calls are non-migratable there is no maintenance window: +**auto-renewal is availability-critical, not a nicety.** + +Protocol posture everywhere: TLS 1.2+1.3, mainstream ECDHE suites, **never 1.3-only** +(Twilio's TLS 1.3 client support is undocumented), and never pin Twilio's certificates. + +## The `/data` volume — read this before anything else + +Caddy keeps its certificates and ACME account state in `/data`. If you recreate the container +without that volume, Caddy re-requests the same certificate — and Let's Encrypt's +**duplicate-certificate limit (5 per week for an identical name set, +[letsencrypt.org/docs/rate-limits](https://letsencrypt.org/docs/rate-limits/))** will lock +your domain out for up to a week. The failure class is **total inbound outage**: 31910 on +Media Streams, [11237](https://www.twilio.com/docs/api/errors/11237) on webhooks, no calls in +or out of the trunk. A container recreate loop (crash-looping deploy, CI that recreates on +every push) burns all five in minutes. + +- Always mount `/data` on a persistent volume (`-v rutster-caddy-data:/data`). +- Back it up like state, because it is: + `docker run --rm -v rutster-caddy-data:/data -v "$PWD":/backup debian:stable-slim tar czf /backup/caddy-data.tgz /data` + +## ACME challenge paths + +| Path | When | Notes | +|---|---|---| +| **HTTP-01** (default) | Port 80 publicly reachable, single hostname | Zero config beyond the domain + email. | +| **DNS-01** | Wildcards (`*.pbx.domain`), CGNAT/behind-NAT hosts, no port 80 | The bundled `rutster-edge` Caddy build ships a curated plugin set: **cloudflare, route53, porkbun, hetzner, desec** (duckdns excluded — no license file). Any other DNS provider requires your own xcaddy build. DNS API credentials live in the container env — scope them to the zone. | +| **BYO-cert** (in-process rustls, Phase 1) | You already have cert distribution (corporate ACME, central wildcard issuance) | Set `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY`; the engine serves TLS itself and **hot-reloads on file change without dropping live WS**. You own renewal delivery. | + +In-binary ACME (Phase 2) is deliberately deferred behind named triggers — among them Let's +Encrypt's dns-persist-01 reaching GA, which would dissolve the DNS-plugin matrix entirely. See +[ADR-0011](../adr/0011-deployment-topology.md). + +## Caddy reload vs live calls + +Routine renewals are safe: Caddy swaps certs **in memory, zero-downtime — the routine periodic +event drops nothing.** Config *reloads* (editing the Caddyfile) are the risk: the mitigation +(`stream_close_delay` above max call duration) has an open upstream bug trail +([caddy#6420](https://github.com/caddyserver/caddy/issues/6420), +[caddy#7222](https://github.com/caddyserver/caddy/issues/7222)), which is why the shipped +artifacts carry a CI e2e test for config-reload-during-live-call. Operator rule: **do not edit +the Caddyfile while calls are live** unless your image version passed that test; drain first +(`/readyz` returns 503 while draining). + +## Fleet certificate patterns (T3) + +Node-addressed placement means every node needs its own publicly-valid TLS name — this is +CPaaS-imposed (`` URLs route on hostname only), not a proxy choice. Two blessed +patterns: + +1. **Wildcard `*.pbx.domain`** via DNS-01, issued **centrally** and distributed to nodes. + Understand the trade: the wildcard private key on every node means one compromised node + burns the whole namespace. +2. **Per-node distinct certs** (`node-1.pbx.domain`, `node-2.…`) — no shared key material. + Renewals are exempt from Let's Encrypt's per-domain limits, so this scales with fleet size. + +Traps: + +- **N nodes independently requesting the identical wildcard** are N duplicate requests — the + 5/week duplicate-cert limit locks the fleet out. Central issuance or distinct names only. +- **Caddy on-demand TLS is rejected for node names**: the first TLS handshake to a fresh name + blocks for seconds on issuance, colliding with the sub-second webhook budget (Twilio's hard + cap is 15 s, UX budget sub-second), and its rate-limit knobs are deprecated. Node names are + known at provision time — pre-issue their certs. + +## Troubleshooting + +| Symptom | Fix | +|---|---| +| Twilio 31910 on Media Streams | Cert not publicly trusted / expired / SNI mismatch. Check `openssl s_client -connect pbx.example.com:443 -servername pbx.example.com`. | +| `rateLimited` / `too many certificates` in Caddy logs | You hit the duplicate-cert limit — almost always a lost `/data` volume. Restore the volume or wait out the window; then fix the volume mount so it never recurs. | +| DNS-01 fails for your provider | Provider not in the bundled plugin set — use a delegated zone on a supported provider, or build your own xcaddy image. | diff --git a/docs/deploy/homelab.md b/docs/deploy/homelab.md new file mode 100644 index 0000000..b56cc29 --- /dev/null +++ b/docs/deploy/homelab.md @@ -0,0 +1,132 @@ +# Homelab & CGNAT — the honest story + +**The truth first: no tunnel carries inbound UDP. WebRTC callers are unreachable behind any + tunnel, so a homelab behind CGNAT is PSTN-only, period.** The engine's WebRTC media is +UDP-direct to an advertised IP (no STUN, no TURN); the CPaaS side offers no relay or +rendezvous either, and intermediaries in the media path are a documented frame-dropping hazard +([livekit/agents#3379](https://github.com/livekit/agents/issues/3379)). Nothing on this page +makes CGNAT production-grade for free. Three tiers, worst to best: + +## Tier 1 — dev/demo: ngrok (the blessed 5-minute path) + +ngrok is the **only** tunnel with a proven Twilio Media Streams record. Do **not** use +Cloudflare Tunnel even for dev: cloudflared has an open Twilio 31920 handshake bug +([cloudflared#1465](https://github.com/cloudflare/cloudflared/issues/1465)), recurring 1006 +closures ([cloudflared#1282](https://github.com/cloudflare/cloudflared/issues/1282)), and +Cloudflare documents killing WebSockets on edge code releases. + +```bash +# 1. Run the engine, plaintext :8080 (no Caddy needed — ngrok terminates TLS): +docker run -d --name rutster-engine --network host \ + -e RUTSTER_TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \ + -e RUTSTER_TWILIO_AUTH_TOKEN=your_auth_token \ + -e RUTSTER_TWILIO_MEDIA_BIND=0.0.0.0:8081 \ + -e RUTSTER_TWILIO_WEBHOOK_BASE=https://REPLACE-ME.ngrok-free.app \ + -e RUTSTER_TRUSTED_PROXIES=127.0.0.1/32 \ + git.adlee.work/alee/rutster-engine:latest +# (or from source: cargo run — see docs/QUICKSTART.md) + +# 2. Tunnel it: +ngrok http 8080 +# 3. Put the printed https://xxxx.ngrok-free.app URL into RUTSTER_TWILIO_WEBHOOK_BASE +# (restart the container), and into the Twilio number's webhook: +# POST https://xxxx.ngrok-free.app/v1/trunk/webhook +# 4. Dial your Twilio number. +``` + +**Free-tier arithmetic** (why this is a demo, not a deployment — research: [TLS brief §3(d)](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)): a Media Streams call is +8 kHz µ-law — 8 kB/s of audio per direction, base64-encoded inside a JSON envelope at 50 +messages/s, both directions ≈ **~140 MB per call-hour** on the wire. ngrok's free 1 GB/month +data cap ([ngrok pricing](https://ngrok.com/pricing)) therefore buys roughly **seven call-hours a month**. And the audio transits ngrok's +edge **in plaintext** (they terminate TLS) — an unconsented subprocessor, which is a +DPA/BAA/PCI failure. Dev only. Never production. + +## Tier 2 — single-user demo: Tailscale Funnel + +Privacy-clean variant: TLS terminates **on your node**, so Funnel relays ciphertext it cannot +read. + +```bash +tailscale funnel 8080 +# webhook base = https://..ts.net +``` + +Still PSTN-only (no inbound UDP), bandwidth cap undisclosed, one user. Unsizable beyond a +personal demo. + +## Tier 3 — production graduation: cheap VPS + WireGuard, TLS at home + +The recommended path. The VPS is a **dumb layer-4 forwarder**: TLS terminates at home (your +Caddy, DNS-01 certs), so the forwarder **physically cannot read the audio** — the strongest +privacy topology available. Bonus: forwarding the media UDP range over the same tunnel +restores WebRTC, which no tunnel product can do. Cost: one small VPS (~$5/mo) and one extra +network hop of media latency — pick a VPS region near home. + +Point DNS at the VPS: `pbx.example.com A `. + +**VPS `/etc/wireguard/wg0.conf`:** + +```ini +[Interface] +Address = 10.88.0.1/24 +ListenPort = 51820 +PrivateKey = + +[Peer] +PublicKey = +AllowedIPs = 10.88.0.2/32 +``` + +**Home box `/etc/wireguard/wg0.conf`** (home initiates — CGNAT-friendly; the keepalive holds +the NAT mapping): + +```ini +[Interface] +Address = 10.88.0.2/24 +PrivateKey = + +[Peer] +PublicKey = +Endpoint = :51820 +AllowedIPs = 10.88.0.0/24 +PersistentKeepalive = 25 +``` + +**VPS forwarding** (`sysctl -w net.ipv4.ip_forward=1`, persist it, then nftables): + +```nft +table ip rutster-fwd { + chain prerouting { + type nat hook prerouting priority dstnat; + iifname "eth0" tcp dport { 80, 443 } dnat to 10.88.0.2 + iifname "eth0" udp dport 49152-49407 dnat to 10.88.0.2 + } + chain postrouting { + type nat hook postrouting priority srcnat; + oifname "wg0" masquerade + } +} +``` + +**Home side:** run T1 or T2 exactly per [quickstart-docker.md](quickstart-docker.md), with: + +- `RUTSTER_MEDIA_ADVERTISED_IP=` — callers send media UDP to the VPS; the DNAT + delivers it home through the tunnel. +- `RUTSTER_MEDIA_PORT_RANGE=49152-49407` (must match the nftables rule). +- Certificates: DNS-01 is the robust choice here ([certificates.md](certificates.md)); + HTTP-01 also works since `:80` is forwarded. + +First call: same two paths as [quickstart-docker.md](quickstart-docker.md) — browser at +`https://pbx.example.com/` (WebRTC now works — the UDP range rides the tunnel) and the Twilio +webhook at `https://pbx.example.com/v1/trunk/webhook`. + +**Or skip the tunnel entirely:** run the engine *on* the VPS (that is just +[T1](quickstart-docker.md) on rented hardware). You trade at-home media for zero forwarding +complexity. + +## Explicitly unsupported for production + +Cloudflare Tunnel or ngrok in the live audio path: plaintext audio at the vendor edge +(unconsented subprocessor — DPA/BAA/PCI failure), documented mid-call WS terminations, zero +SLA, and Cloudflare's discretionary "disproportionate audio" ToS clause aimed at exactly this +traffic profile (research basis: [TLS brief §4](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)). Ratified in [ADR-0011](../adr/0011-deployment-topology.md). diff --git a/docs/deploy/quickstart-docker.md b/docs/deploy/quickstart-docker.md new file mode 100644 index 0000000..61da1d8 --- /dev/null +++ b/docs/deploy/quickstart-docker.md @@ -0,0 +1,104 @@ +# Docker quickstart — zero to first call + +Two shapes, one binary ([topologies.md](topologies.md)). T1 is one `docker run`; T2 is the +reference `compose up`. Both end the same way: a browser call at your domain, and (with Twilio +credentials) a real phone call. + +> **Artifact names note:** images are published to the `git.adlee.work/alee/` registry +> namespace by the release CI (plan B / slice F settled this). If your Gitea registry uses a +> different owner segment, substitute accordingly — check `deploy/compose.yaml` in your +> checkout, which is authoritative. + +## Prerequisites (both shapes) + +1. A Linux host with Docker, a **public IPv4**, and inbound `80/tcp`, `443/tcp`, and your + chosen media UDP range open. (No public IP? → [homelab.md](homelab.md).) +2. A domain with an A record pointing at that host, e.g. `pbx.example.com`. Certificates are + issued automatically via ACME — no self-signed path exists, the CPaaS refuses it + ([certificates.md](certificates.md)). +3. For the phone call: a [Twilio](https://www.twilio.com/) account + a Voice-capable number + (trial works). + +## T1 — Solo (all-in-one) + +```bash +docker run -d --name rutster --restart unless-stopped \ + --network host \ + -v rutster-caddy-data:/data \ + -v rutster-valkey:/var/lib/valkey \ + -e RUTSTER_DOMAIN=pbx.example.com \ + -e RUTSTER_ACME_EMAIL=you@example.com \ + -e RUTSTER_MEDIA_ADVERTISED_IP=203.0.113.7 \ + -e RUTSTER_MEDIA_PORT_RANGE=49152-49407 \ + -e RUTSTER_TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \ + -e RUTSTER_TWILIO_AUTH_TOKEN=your_auth_token \ + -e RUTSTER_TWILIO_MEDIA_BIND=0.0.0.0:8081 \ + -e RUTSTER_TWILIO_WEBHOOK_BASE=https://pbx.example.com \ + git.adlee.work/alee/rutster-allinone:latest +``` + +Notes: + +- `--network host` is the recommended mode: WebRTC media UDP goes **direct to the engine**, + never through Caddy. If you can't use host networking, publish `-p 80:80 -p 443:443 + -p 49152-49407:49152-49407/udp` instead and keep `RUTSTER_MEDIA_ADVERTISED_IP` = the host's + public IP (NAT 1:1, no STUN). +- The two volumes are **non-negotiable**. `/data` in particular: recreating the container + without it re-requests certificates and can hit Let's Encrypt's duplicate-certificate + limit — a week-long total inbound outage ([certificates.md](certificates.md)). +- The four `RUTSTER_TWILIO_*` vars are all-or-none — the engine refuses partial Twilio config + at startup. Omit all four to run WebRTC-only. +- `RUTSTER_DOMAIN` / `RUTSTER_ACME_EMAIL` feed the bundled Caddyfile. Domains that can't do + HTTP-01 (no port 80, wildcard needs) use DNS-01 — [certificates.md](certificates.md). + +Verify it's up: + +```bash +curl -fsS https://pbx.example.com/healthz && echo OK # liveness +curl -fsS https://pbx.example.com/readyz && echo READY # can accept a new call +``` + +### First call (browser, WebRTC) + +Open `https://pbx.example.com/`, click **Start call**, grant the microphone. You're on a call +with the engine. + +### First call (phone, PSTN) + +1. Twilio Console → Phone Numbers → your number → **A call comes in** → Webhook, + `POST https://pbx.example.com/v1/trunk/webhook`. +2. Dial your Twilio number from any phone. Twilio hits the webhook, the engine answers with + TwiML pointing Twilio's Media Streams at `wss://pbx.example.com/twilio/media-stream`, and + the call's audio enters the same reflex loop a WebRTC call uses. + +## T2 — Modular (compose stack) + +```bash +git clone https://git.adlee.work/alee/rutster.git +cd rutster/deploy +cp .env.example .env +$EDITOR .env # set DOMAIN, ACME email, RUTSTER_MEDIA_ADVERTISED_IP, RUTSTER_TWILIO_* +docker compose up -d +``` + +What comes up: `caddy` (edge, `rutster-edge`), `engine` (`rutster-engine`), `brain` +(`rutster-brain`, sharing the engine's network namespace), `valkey` (upstream image). The +same two first-call paths as T1 apply verbatim. + +Operational notes: + +- `.env.example` ships `RUTSTER_DRAIN_DEADLINE_SECS=600` and the stack sets + `stop_grace_period: 660s` — grace exceeds drain, so `docker compose down` lets in-flight + calls finish (up to 10 minutes) instead of killing them. +- Upgrade: `docker compose pull && docker compose up -d`. +- Bring your own proxy instead of Caddy: [reverse-proxies.md](reverse-proxies.md). + +## Troubleshooting + +| Symptom | Cause / fix | +|---|---| +| Twilio error **31910** on calls | Your cert isn't publicly trusted or expired — check `/data` volume survived, see [certificates.md](certificates.md) ([twilio.com/docs/api/errors/31910](https://www.twilio.com/docs/api/errors/31910)) | +| Webhook signature validation fails | The edge isn't forwarding `X-Forwarded-Proto/Host` honestly, or `RUTSTER_TRUSTED_PROXIES` doesn't include your proxy — the engine ignores forwarded headers from unlisted sources | +| Browser call connects, no audio | Media UDP blocked or `RUTSTER_MEDIA_ADVERTISED_IP` wrong — the SDP advertises that IP; callers send UDP straight to it | +| `curl /readyz` returns 503 | Node is draining or at the admission cap (`RUTSTER_MAX_SESSIONS`) — liveness (`/healthz`) stays 200 | +| Let's Encrypt rate-limit errors in Caddy logs | You recreated the container without the `/data` volume — [certificates.md](certificates.md) | diff --git a/docs/deploy/reverse-proxies.md b/docs/deploy/reverse-proxies.md new file mode 100644 index 0000000..5c05a3f --- /dev/null +++ b/docs/deploy/reverse-proxies.md @@ -0,0 +1,139 @@ +# Bring your own reverse proxy — tuned-timeout configs + +Supported: disable the `caddy` service in the compose stack +([quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack)); the engine keeps its +plaintext `:8080` listener. Everything below terminates TLS in *your* proxy and forwards to +the engine. + +## The universal 60-second footgun + +**nginx, HAProxy, and Traefik all default their idle/read timeouts to ~60–180 seconds — every +one of them will kill a quiet, perfectly healthy call WebSocket.** A Media Streams WS can +legitimately be near-silent in one direction for hours, and neither Twilio nor Telnyx sends +protocol-level keepalives. The engine ships app-level WS pings (`RUTSTER_WS_PING_SECS`, +default 20) as belt-and-braces, but your proxy timeouts must still exceed the maximum call +duration. Every snippet below does that. + +## The contract any proxy must honor + +1. **Publicly-trusted cert**, auto-renewed — no self-signed path exists + ([certificates.md](certificates.md)). +2. **WS upgrade with no frame buffering and no connection-lifetime cap** — 20 ms audio frames, + calls up to 24 h. +3. **Honest `X-Forwarded-Proto` and `X-Forwarded-Host`** (including on the WS upgrade + request): `X-Twilio-Signature` is HMAC over the URL as Twilio saw it + ([Twilio's SSL-termination guidance](https://www.twilio.com/en-us/blog/developers/tutorials/building-blocks/handle-ssl-termination-twilio-node-js-helper-library)). +4. **Engine-side trust**: set `RUTSTER_TRUSTED_PROXIES` to your proxy's source IP/CIDR — + forwarded headers from unlisted sources are ignored, and signature validation will fail. +5. **TLS 1.2+1.3, mainstream ECDHE — never 1.3-only** (Twilio's 1.3 client support is + undocumented). + +## nginx (works if you insist — not a recommended edge) + +Why not recommended: no native DNS-01/wildcard support, and cert renewal requires a reload +whose old workers linger against hours-long WS connections (research basis: [TLS brief §4](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md)). rutster ships this snippet and +nothing more for nginx. + +```nginx +map $http_upgrade $connection_upgrade { default upgrade; '' close; } + +server { + listen 443 ssl; + server_name pbx.example.com; + ssl_certificate /etc/letsencrypt/live/pbx.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/pbx.example.com/privkey.pem; + ssl_protocols TLSv1.2 TLSv1.3; # never 1.3-only + + location / { + proxy_pass http://127.0.0.1:8080; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_buffering off; # 20 ms frames must not be coalesced + proxy_read_timeout 86400s; # the 60s default is the footgun + proxy_send_timeout 86400s; + } +} +``` + +## HAProxy (3.2 LTS or newer) + +The best WS mental model of the classic family: after the upgrade, only `timeout tunnel` +governs the connection — **but only if you set it**; put it in `defaults` or it silently +doesn't apply to the paths you think it does (see +[haproxy#2280](https://github.com/haproxy/haproxy/issues/2280)). Two extra notes: `acme.sh` +can hot-load renewed certs through the stats socket with zero reload; and `hard-stop-after` +kills calls that are still draining — leave it unset or above your drain deadline. + +```haproxy +defaults + mode http + timeout connect 5s + timeout client 75s + timeout server 75s + timeout tunnel 24h # governs upgraded WS; unset = the 60s-class footgun + +frontend fe_rutster + bind :443 ssl crt /etc/haproxy/certs/ ssl-min-ver TLSv1.2 + http-request set-header X-Forwarded-Proto https + http-request set-header X-Forwarded-Host %[req.hdr(Host)] + default_backend be_rutster + +backend be_rutster + server fob 127.0.0.1:8080 +``` + +## Traefik v3 — PIN THE EXACT VERSION + +Traefik has the best built-in DNS-01/wildcard of the classic family **and a track record of +breaking WebSockets in patch releases**: +[#10601](https://github.com/traefik/traefik/issues/10601), the v2.11.2 timeout-behavior flip, +and [#11405](https://github.com/traefik/traefik/issues/11405). **Pin the exact image tag and +never auto-update the edge.** + +```yaml +# traefik.yml (static config) +entryPoints: + websecure: + address: ":443" + transport: + respondingTimeouts: + readTimeout: 0 # 0 disables; the 60s-class default kills quiet WS + idleTimeout: 0 + writeTimeout: 0 +certificatesResolvers: + le: + acme: + email: you@example.com + storage: /data/acme.json # persist this volume — certificates.md + dnsChallenge: + provider: cloudflare +``` + +```yaml +# dynamic config +http: + routers: + rutster: + rule: "Host(`pbx.example.com`)" + entryPoints: [websecure] + service: rutster + tls: { certResolver: le } + services: + rutster: + loadBalancer: + servers: + - url: "http://127.0.0.1:8080" +``` + +Traefik forwards `X-Forwarded-Proto/Host` by default — you still must list its address in +`RUTSTER_TRUSTED_PROXIES`. + +## First call + +Any of the above in front of the engine, then the same two first-call paths as +[quickstart-docker.md](quickstart-docker.md): browser at `https://pbx.example.com/`, Twilio +webhook at `https://pbx.example.com/v1/trunk/webhook`. diff --git a/docs/deploy/topologies.md b/docs/deploy/topologies.md new file mode 100644 index 0000000..a69e550 --- /dev/null +++ b/docs/deploy/topologies.md @@ -0,0 +1,105 @@ +# Deployment topologies — one binary, three blessed shapes + +The FOB never decomposes; topology is a configuration property, not a code property. The same +binary runs in every shape below; only `RUTSTER_*` env and what is deployed alongside it +differ. Ratified (Proposed) in [ADR-0011](../adr/0011-deployment-topology.md). + +## Decision guide + +| You are… | Use | Doc | +|---|---|---| +| Trying rutster for the first time; one box, one domain, a public IP | **T1 — Solo** | [quickstart-docker.md](quickstart-docker.md) | +| Running production and want independently upgradable parts | **T2 — Modular** (the reference deployment) | [quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack) | +| Behind CGNAT / no public IP (homelab) | T1/T2 + tunnel (dev, **PSTN-only**) or VPS forwarder (production) | [homelab.md](homelab.md) | +| Already running nginx / HAProxy / Traefik | T2 minus the `caddy` service | [reverse-proxies.md](reverse-proxies.md) | +| On EC2 and want ALB + ACM to terminate TLS | T2/T3 behind an ALB | [aws.md](aws.md) | +| Outgrowing one node | **T3 — Fleet** — **paper only today** | [ADR-0011](../adr/0011-deployment-topology.md) | + +## T1 — Solo (all-in-one container) + +One image, `rutster-allinone`, with s6-overlay supervising four processes: + +| Process | Role | Binding | +|---|---|---| +| `caddy` | Edge: ACME issuance/renewal, TLS termination, WS proxy | `:443` / `:80` public | +| `rutster` (the FOB) | Engine — signaling, trunk webhook, media, reflexes | `127.0.0.1:8080` behind Caddy; **media UDP direct** | +| `rutster-brain-realtime` | Brain | `127.0.0.1:8082` loopback tap | +| `valkey-server` | Event stream (`EventSink`); ledger/directory later | `127.0.0.1:6379` | + +Two volumes, **both non-negotiable**: + +- `/data` — Caddy's cert/ACME state. Losing it can lock your domain out of Let's Encrypt for + up to a week (duplicate-certificate limit) — a **total inbound outage**. See + [certificates.md](certificates.md). +- `/var/lib/valkey` — stream/state persistence. + +WebRTC media (UDP) goes **direct to the FOB, never through Caddy**: run with host networking, +or publish `RUTSTER_MEDIA_PORT_RANGE` and set `RUTSTER_MEDIA_ADVERTISED_IP` to your public IP +(NAT 1:1 model — no STUN). + +Copy-paste path from zero to first call: [quickstart-docker.md](quickstart-docker.md). + +## T2 — Modular (compose stack) — the reference deployment + +Four services in `deploy/compose.yaml`: + +| Service | Image | Notes | +|---|---|---| +| `caddy` | `rutster-edge` | Same custom Caddy build + same Caddyfile as T1 | +| `engine` | `rutster-engine` | The FOB; plaintext `:8080` on the compose network; media UDP published | +| `brain` | `rutster-brain` | `network_mode: "service:engine"` — shares the engine's netns so the loopback-only tap posture survives unchanged | +| `valkey` | `valkey/valkey` (upstream) | Own service; network-near | + +Operational contract: + +- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` — grace **must** + exceed drain, or the orchestrator kills calls the engine was gracefully draining. +- Bring-your-own-proxy: disable the `caddy` service; the engine keeps its plaintext `:8080` + mode. Tuned-timeout configs: [reverse-proxies.md](reverse-proxies.md). +- Upgrades: `docker compose pull && docker compose up -d` — the drain window lets in-flight + calls finish (calls are non-migratable; a killed engine kills its calls, honestly stated). + +Copy-paste path from zero to first call: +[quickstart-docker.md](quickstart-docker.md#t2--modular-compose-stack). + +## T3 — Fleet (paper only — no shipped artifacts in this epoch) + +**Honest statement: you cannot deploy T3 today by copy-paste; there are no fleet artifacts +yet.** The design is ratified on paper in +[ADR-0011](../adr/0011-deployment-topology.md) so single-node deployments grow toward it, not +away from it: + +- N symmetric nodes (the T1 process set minus valkey) + a shared green zone: one Valkey + (presence + directory + spend ledger), object storage. +- Wildcard DNS `*.pbx.domain`; per-node certs — the traps live in + [certificates.md](certificates.md). +- Answer-time placement via Valkey presence; node-addressed `wss://node-K.pbx.domain/...` + Stream URLs; no router tier. +- Per-node admission cap = the benchmark number (`RUTSTER_MAX_SESSIONS`, default 64, is a + placeholder until then). +- Scale-in is drain-then-terminate only. No spot instances for the engine tier. + +## What is deliberately not supported + +- **Splitting the FOB into services** — permanent rejection, not a roadmap item. +- **Serverless** — non-migratable multi-hour calls, per-session UDP sockets, in-process media + state. Categorically unfit. +- **Tunnels in production** — plaintext audio at the tunnel vendor's edge + documented + mid-call WS kills. Dev/demo only; the whole story is in [homelab.md](homelab.md). +- **NLB TLS listeners** — see [aws.md](aws.md) for the 350-second reason. +- **k8s manifests/Helm** — compose-first. If you run k8s anyway: set + `terminationGracePeriodSeconds` above `RUTSTER_DRAIN_DEADLINE_SECS`, same grace-exceeds-drain + rule as compose. + +## Ports & volumes reference (all shapes) + +| Surface | Where | Notes | +|---|---|---| +| `:80` / `:443` TCP | Caddy, public | ACME HTTP-01 + TLS + all HTTPS/WSS traffic | +| `:8080` TCP | FOB, behind the edge | Signaling + `/v1/trunk/webhook` + `/twilio/media-stream` WS — one listener (`RUTSTER_HTTP_BIND`) | +| `RUTSTER_MEDIA_PORT_RANGE` UDP | FOB, **public, direct** | WebRTC media; never proxied | +| `:8082` TCP | Brain, loopback only | The tap; non-loopback rejected until step 6 | +| `:6379` TCP | Valkey, private | Loopback (T1) / compose network (T2) / green zone (T3) | +| `:9090` TCP | FOB `/metrics`, internal | `RUTSTER_METRICS_BIND`; never routed through Caddy | +| `/data` volume | Caddy | Cert/ACME state — loss class: total inbound outage | +| `/var/lib/valkey` volume | Valkey | Event stream persistence |