# Rutster deployment topology — one binary, three blessed shapes

- **Status:** Draft (pending review)
- **Date:** 2026-07-05
- **Origin:** [ADR-0008](../../adr/0008-fob-and-green-zone.md) explicitly reserved a deployment
  topology ADR "once that design closes." This spec closes it. Brainstormed 2026-07-05; the
  TLS/edge fork was researched via a six-family survey — see the companion
  [TLS/edge decision brief](2026-07-05-tls-edge-decision-brief.md).
- **Produces:** **ADR-0011 (deployment topology)** — drafted from §2–§4 of this spec, Proposed,
  ratified per repo process — plus the artifacts and code slices in §5–§6.
- **Depends on (already merged):** slice-5/seams (drain lifecycle, `/readyz`, admission cap,
  `EventSink`, `MediaAddressConfig`, `config.rs` env-parser pattern); slice-4½ benchmark/sim
  (S1–S8, CI-regressed — the future source of the real per-node admission cap).
- **Related:** [ADR-0002](../../adr/0002-north-star-and-fused-core.md) (fused per-call vertical),
  [ADR-0005](../../adr/0005-event-bus.md) (Valkey = bus + KV + presence; media never rides the
  bus; bus is never billing source-of-truth), [ADR-0006](../../adr/0006-ingress-posture.md)
  (two inbound protocols only; tap is egress-only), [ADR-0007](../../adr/0007-trunk-rented-transport.md)
  (rented transport ⇒ public HTTPS/WSS reachability), [ADR-0009](../../adr/0009-spend-gate-honest-rescope.md)
  (ledger trait: in-memory single-node, Valkey-backed fleets), [ADR-0010](../../adr/0010-spearhead-benchmark-sim-harness.md)
  (sequencing this spec must not reorder), README pillar 6 ("one binary, one bus, one deploy").

---

## TL;DR

**The FOB never decomposes; topology is a configuration property, not a code property.** The same
binary runs in every blessed shape; only `RUTSTER_*` env and what is deployed alongside it differ.
"Microservices" in rutster means *green-zone services get their own processes/containers* — never
that the per-call vertical splits (ADR-0002).

Three blessed topologies:

| | Shape | Status after this work |
|---|---|---|
| **T1 — Solo** | One all-in-one container: s6 supervises Caddy + FOB + brain + valkey-server (EventSink consumer day one; ledger/directory later). `docker run` with a domain + Twilio creds → taking calls. | Shipped artifact |
| **T2 — Modular** | Compose stack: `caddy`, `engine`, `brain` (shared netns with engine), `valkey` as separate services. `compose up` is the pillar. | Shipped artifact (reference deployment) |
| **T3 — Fleet** | N symmetric nodes + shared green zone (one Valkey, object storage). Node-addressed URLs, answer-time placement via Valkey presence. | ADR chapter only (paper) |

Riders shipping with the artifacts: TCP_NODELAY + app-level WS-ping hygiene, trunk config fixes,
in-process TLS **Phase 1** (BYO-cert, hot-reload), a hand-rolled Prometheus `/metrics` endpoint,
`ValkeyEventSink` (first Valkey consumer), image-publish CI, and a `docs/deploy/` documentation
tree with a copy-paste-to-first-call acceptance bar.

**Six things this design deliberately is NOT:**

- **Not a decomposition of the FOB into services.** Media/reflex/tap/spend stay one process,
  one trust domain, per ADR-0002. Rejected permanently, not deferred.
- **Not serverless.** Lambdas are categorically unfit for the engine: non-migratable multi-hour
  calls, per-session UDP sockets, in-process media state. Webhook-on-Lambda buys nothing because
  placement needs fleet state anyway. Stated as a rejected shape in ADR-0011.
- **Not k8s manifests/Helm.** Compose-first (pillar 6). k8s notes live in docs only (drain vs
  `terminationGracePeriodSeconds`); manifests are a later rung if demand appears.
- **Not the durable CDR pipeline.** `ValkeyEventSink` is evidence preservation on a capped
  stream — explicitly NOT the billing ledger. CDR → object storage stays deferred
  (ADR-0005 source-of-truth rule).
- **Not the zero-egress / air-gap profile.** T1 is about one-command simplicity, not isolation.
  Zero-egress (SBC layer-2 ingress + self-hosted brain) is named as a defined-but-deferred
  profile in ADR-0011, one footnote section, no code.
- **Not in-binary ACME (Phase 2).** Deferred behind named triggers (§3.4). No turnkey crate fits
  the tree today (rustls-acme conflicts with the deny.toml rcgen/x509-parser posture); owning a
  renewal state machine in a 45-day-cert world is a deliberate, triggered decision.

---

## 1. Scope

### 1.1 In scope

- **ADR-0011** (deployment topology): the three shapes, the edge doctrine, the fleet placement
  paradigm, the rejected shapes.
- **Artifacts** under `deploy/`: multi-stage `Dockerfile` (four first-party images), `compose.yaml`,
  `Caddyfile`, `.env.example`.
- **Code:** TCP_NODELAY, app-level WS pings, trunk config fixes (webhook base URL, credential
  wiring, trusted-proxy header posture), rustls Phase 1 (BYO-cert TLS), `/metrics`,
  `ValkeyEventSink`.
- **CI:** containerized smoke suite (incl. Caddy-reload-during-live-call), image publish on tag.
- **Docs:** `docs/deploy/` tree (§10).

### 1.2 Out of scope (named in ADR-0011 as deferred, so the boundary is deliberate)

- T3 fleet **implementation** (presence heartbeats, directory redirects, placement code).
- Object-storage CDR durability; multi-tenancy (needs its own ADR before schemas ossify);
  wss:// tap + brain fleet + resume tokens (step 6); media-thread sharding graduation;
  in-binary ACME (Phase 2); SBC layer-2 wire contract; arm64 images.

---

## 2. Topologies

### 2.1 T1 — Solo (all-in-one container)

One image (`rutster-allinone`), tini/s6-overlay as PID 1 supervising four processes:

| Process | Role | Binding |
|---|---|---|
| `caddy` | Edge: ACME, TLS termination, WS proxy to FOB | :443/:80 public |
| `rutster` (FOB) | Engine | 127.0.0.1:8080 (behind Caddy); media UDP direct |
| `rutster-brain-realtime` | Brain | 127.0.0.1:8082 loopback tap (as today) |
| `valkey-server` | **Dark on day one** except EventSink (§5.6) | 127.0.0.1:6379 |

Volumes (both non-negotiable, documented loudly):
- `/data` — Caddy cert/ACME state. Loss risks Let's Encrypt duplicate-cert lockout (5/week)
  → total inbound outage class.
- `/var/lib/valkey` — stream/state persistence.

WebRTC media UDP goes **direct to the FOB**, never through Caddy: host networking recommended;
otherwise published `RUTSTER_MEDIA_PORT_RANGE` + `RUTSTER_MEDIA_ADVERTISED_IP` (NAT 1:1 model,
no STUN — unchanged from slice-5/seams).

Valkey ships bundled from v1 **before most consumers exist** (decision 2026-07-05): the operator
contract (volumes, ports, upgrade shape) is stable from day one, so the spend ledger and fleet
directory land later without changing the deployment shape. Cost accepted: ~15 MB and one
mostly-idle process.

### 2.2 T2 — Modular (compose stack) — the reference deployment

Services: `caddy` (image `rutster-edge`), `engine` (`rutster-engine`), `brain` (`rutster-brain`),
`valkey` (upstream `valkey/valkey`).

- **Brain shares the engine's network namespace** (`network_mode: "service:engine"`) so the
  loopback-only tap posture (`resolve_tap_url` rejects non-loopback until step 6) survives
  unchanged. ADR-0011 notes the brain graduates to its own netns when the wss:// tap lands.
- Valkey is its own service on the compose network — network-near, satisfies ADR-0005's sub-ms
  rule.
- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` (§8).
- BYO-proxy supported: disable the `caddy` service; the engine keeps its plaintext `:8080` mode;
  tuned-timeout snippets for nginx/HAProxy/Traefik ship in docs (§10).

### 2.3 T3 — Fleet (ADR chapter only; no code in this epoch)

N **symmetric** nodes, each running the T1 process set *minus* valkey, plus a shared green zone:
one Valkey (presence + directory + ADR-0009 spend ledger), object storage (named as required,
still deferred). Wildcard DNS `*.pbx.domain`; per-node certs (§3.3). Per-node admission cap =
the ADR-0010 benchmark number (replacing the `RUTSTER_MAX_SESSIONS=64` placeholder). Scale-in is
drain-then-terminate only; spot instances rejected for the engine tier (2026-07-04 review).
Placement paradigm: §4.

---

## 3. Edge doctrine

### 3.1 CPaaS invariants (the spec any edge must satisfy — research §2)

1. Publicly-CA-trusted cert; **no self-signed path exists** (Twilio error 31910). Auto-renewal is
   availability-critical.
2. wss:// on 443, open to the internet: Twilio publishes no egress IPs, offers no mTLS. Auth is
   application-layer only (HMAC signatures).
3. One WS per call, up to 24 h, 50 msg/s/direction at 20 ms cadence; zero tolerance for frame
   buffering, lifetime caps, or write-side idle timers. Neither Twilio nor Telnyx documents WS
   keepalive — **keepalive is entirely our job** (§5.2).
4. `<Stream>` URLs carry **no query strings** — wss routing keys on hostname/path only
   (constrains §4).
5. `X-Twilio-Signature` is HMAC over the URL *as Twilio saw it*: the edge must forward
   `X-Forwarded-Proto/Host` honestly and the FOB must reconstruct the public URL (§5.3).
6. Webhook budget: sub-second for UX, 15 s hard cap — no blocking issuance in the handshake path.
7. TLS 1.2+1.3, mainstream ECDHE, never 1.3-only; never pin Twilio certs.

Design for Twilio; Telnyx/Vonage come for free (they are strictly less demanding, plus optional
source-IP allowlisting as extra hardening Twilio can't use).

### 3.2 Bundled edge: Caddy (both shipped artifacts, same build, same Caddyfile)

Custom xcaddy build (`rutster-edge` image): Caddy core (Apache-2.0) + curated DNS-01 plugin set —
cloudflare, route53, porkbun, hetzner, desec (duckdns excluded: no license file). Zero-downtime
in-memory cert renewal; verified protocol-unaware WS tunneling (no frame buffering). Config:
~6-line Caddyfile with tuned timeouts, honest `X-Forwarded-*`, `stream_close_delay` above max
call duration — **and a CI e2e case for config-reload-during-live-call, because the upstream
mitigation has an open bug trail (caddy #6420/#7222) and is not trusted untested.**

### 3.3 Fleet cert story (ADR chapter)

Per-node public TLS names are CPaaS-imposed (invariant 4 + node-addressed placement). Two blessed
patterns: wildcard `*.pbx.domain` via DNS-01 with central issuance/distribution, or per-node
distinct certs (`node-N.domain`) where the wildcard private key must not live on every node.
Traps stated in the ADR: N nodes independently requesting the identical wildcard hit the LE
duplicate-cert limit; **Caddy on-demand TLS is rejected for node names** (blocking first
handshake vs the webhook budget).

### 3.4 In-process TLS: staged (the ratified end-state, not adopted wholesale now)

ADR-0008 claims the trust-domain edge as security-constitutive FOB territory, and str0m already
terminates DTLS-SRTP in-process on the same aws-lc-rs provider — "keep TLS out of the FOB" is not
the objection. The objection is economics: owning an ACME renewal state machine forever.

- **Phase 1 (this epoch, §5.4):** `axum-server` + rustls with operator-supplied certs, hot-reload
  without dropping live WS. Plaintext-behind-Caddy stays the default.
- **Phase 2 (in-binary ACME) triggers, named in ADR-0011:** (a) Let's Encrypt dns-persist-01
  confirmed GA (closes the wildcard/CGNAT issuance gap; instant-acme already parses it);
  (b) a field incident where Caddy reload/`stream_close_delay` drops live calls; (c) fleet or
  VPS-forwarder topology making encrypted-to-the-binary a selling point; (d) maintenance fatigue
  with the curated xcaddy plugin build.

### 3.5 Tunnels and cloud LBs

- **Tunnels are dev/demo-only, stated bluntly.** Cloudflare Tunnel and ngrok see plaintext audio
  at the vendor edge (unconsented subprocessor — DPA/BAA/PCI failure), document mid-call WS kills,
  and cloudflared has an open Twilio handshake bug (#1465). ngrok is the blessed 5-minute demo
  path (only tunnel with a proven Media Streams record). Tailscale Funnel: privacy-clean
  single-user demo (TLS terminates on-node), unsizable beyond that.
- **No tunnel carries inbound UDP → homelab behind CGNAT is PSTN-only, period.** Production
  graduation path: cheap VPS + WireGuard as a dumb TCP forwarder with TLS terminating at home
  (forwarder physically cannot read audio), or run the engine on the VPS.
- **AWS appendix (docs only, never a shipped artifact):** ALB + wildcard ACM + host-header rules;
  mandatory attribute overrides `idle_timeout=4000`, `client_keep_alive=604800`, header
  forwarding. **Never NLB-TLS** (fixed 350 s silent drop, no HTTP context for signatures).

---

## 4. Fleet placement paradigm (ADR chapter; the trunk webhook handler and session API grow
*toward* this, no fleet code in this epoch)

**Node-addressed URLs + Valkey presence; no router tier.** (Decision 2026-07-05 over
routing-tier-proxies-everything and distinguished-control-node.)

- Nodes heartbeat capacity into Valkey (TTL'd presence keys — the ADR-0005 presence layer).
- Placement happens at answer time by whichever node the request lands on (`api.domain`,
  DNS round-robin): read fleet capacity → pick owner → return TwiML whose Stream URL is
  `wss://node-K.pbx.domain/...`. The media WS dials the owning node directly — zero extra hops,
  consistent with connection-follows-ownership.
- WebRTC: create-session response carries the owner's node-addressed base URL + advertised media
  IP (UDP is already direct).
- Misdirected control ops (e.g. DELETE hitting a non-owner) consult the Valkey directory and
  **307-redirect** to the owner.
- Degraded mode (Valkey dark): the answering node self-places if it has capacity; presence heals
  via TTL on recovery. Fail-safe degradation, consistent with ADR-0009's posture.
- Single-node is the degenerate case of the same code: a one-entry directory.

Rationale: Twilio's answer-time Stream URL is purpose-shaped for this (invariant 4 permits
hostname routing); a proxying router tier would put every 20 ms frame through an extra userspace
hop and re-create the SPOF the symmetric design avoids.

---

## 5. Code changes (workspace)

### 5.1 TCP_NODELAY (hygiene, unconditional)

Set on every accepted socket (axum #2521 — Nagle can coalesce 20 ms WS frames on the plaintext
listener behind *any* proxy). Latency assertion joins the CI-regressed sim-bench.

### 5.2 App-level WS pings (hygiene, unconditional)

Engine-originated ping/pong on the trunk media-stream WS (and any future long-lived WS surface;
WebRTC signaling is plain HTTP today, so the media stream is the only such socket), interval
configurable (`RUTSTER_WS_PING_SECS`, default 20). Belt-and-braces against every idle timer
between the FOB and the CPaaS (invariant 3).

### 5.3 Trunk config fixes + trusted-proxy posture

- TwiML Stream URL derived from `RUTSTER_TWILIO_WEBHOOK_BASE` (kills the hardcoded loopback at
  `routes.rs:258`).
- `config::twilio_credentials` wired into startup validation (parsed-but-never-called today).
- Trusted-proxy: `X-Forwarded-Proto/Host` honored **only** from the configured edge
  (`RUTSTER_TRUSTED_PROXIES`, CIDR list; empty = headers ignored); signature validation
  reconstructs the public URL from them (invariant 5).

### 5.4 rustls Phase 1 — BYO-cert in-process TLS

`axum-server` + rustls (already in-tree transitively): `RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY`
enable a TLS listener; hot-reload on file change without dropping live WS (arc-swap resolver).
Always compiled, runtime-gated; plaintext `:8080` behind Caddy remains the artifact default.

### 5.5 `/metrics` — hand-rolled Prometheus text endpoint

Reads the *existing* atomics: active sessions, `tick_overruns`, `last_tick_micros`, admission
rejects, EventSink drops (§5.6). Zero new deps; node-level cardinality only. Own internal bind
`RUTSTER_METRICS_BIND` (default `127.0.0.1:9090`) — never routed through Caddy.

### 5.6 `ValkeyEventSink` — first Valkey consumer

Implements the existing `EventSink` trait: `XADD` of current lifecycle events as JSON to a capped
stream (`XADD MAXLEN ~`), strictly fire-and-forget — Valkey down ⇒ count-and-drop (counter
surfaces in `/metrics`), never blocks the call path. `RUTSTER_VALKEY_URL` unset ⇒ today's
`TracingEventSink`. Client crate: redis-rs (BSD-3; verify against deny.toml in the slice).
Closes the "OOM-killed node erases all evidence of its calls" gap. **Evidence preservation, not
the billing ledger** (ADR-0005 rule restated in code docs).

### 5.7 Env additions (all follow the `config.rs` fail-fast parser pattern)

| Var | Default | Used by |
|---|---|---|
| `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY` | unset (plaintext) | §5.4 |
| `RUTSTER_TRUSTED_PROXIES` | empty (ignore forwarded headers) | §5.3 |
| `RUTSTER_WS_PING_SECS` | 20 | §5.2 |
| `RUTSTER_METRICS_BIND` | 127.0.0.1:9090 | §5.5 |
| `RUTSTER_VALKEY_URL` | unset (tracing sink) | §5.6 |
| `RUTSTER_DRAIN_DEADLINE_SECS` | unchanged code default 0; **600 in `.env.example`** | §8 |

---

## 6. Artifacts

### 6.1 Images (four first-party, published to the git.adlee.work registry)

| Image | Contents | Base |
|---|---|---|
| `rutster-allinone` | s6-overlay: Caddy + FOB + brain + valkey-server | debian-slim + `libopus0` |
| `rutster-engine` | FOB only | debian-slim + `libopus0` |
| `rutster-brain` | brain only | debian-slim |
| `rutster-edge` | custom xcaddy build (§3.2) | scratch/alpine |

Multi-stage `deploy/Dockerfile`, pinned 1.85 toolchain per `rust-toolchain.toml`. Distro
`libopus0` over static-bundled (matches dev; static noted as future size work). linux/amd64
first; arm64 deferred. Bundled-binary licenses are aggregation-clean vs GPL-3 (Caddy Apache-2.0,
valkey BSD-3, s6 ISC) — noted for the cargo-deny-adjacent image license scan in CI.

### 6.2 `deploy/` tree

`Dockerfile`, `compose.yaml`, `Caddyfile`, `.env.example` (documented defaults incl. drain 600 s),
plus the smoke-test harness consumed by CI (§9).

### 6.3 Publish CI

Gitea Actions job on tag push: build all four images, push to the registry. Runs after (and
gated on) the existing fmt/clippy/test/deny/sim-bench pipeline.

---

## 7. Data flow (T1/T2)

**PSTN:** Twilio → HTTPS webhook → Caddy → FOB `/v1/trunk/webhook` (signature validated against
URL reconstructed from trusted `X-Forwarded-*`) → TwiML with derived wss URL → Twilio dials
media stream → Caddy (unbuffered WS tunnel) → FOB WS upgrade → `MediaCmd::RegisterTrunk` →
media-thread tick (G.711 ↔ PCM) ↔ brain tap (loopback). **WebRTC:** signaling via Caddy; SDP
answer advertises media IP; DTLS-SRTP UDP direct to FOB. **Events:** EventSink → Valkey stream
(loopback T1 / compose network T2). **Metrics:** Prometheus scrapes the internal bind.
**T3 flow** as §4.

---

## 8. Failure modes (worst first)

| Failure | Handling |
|---|---|
| Caddy `/data` loss | Persistent volume mandatory; LE duplicate-cert lockout documented as total-inbound-outage class |
| Caddy reload vs live calls | `stream_close_delay` > max call duration **+ CI reload-during-call e2e** (§9); documented reload procedure |
| Valkey down | EventSink counts-and-drops; calls unaffected. T3: degraded self-placement (§4) |
| Node/FOB death | Calls on it die (non-migratable — stated honestly); s6 restarts in T1; Valkey stream preserves evidence; T3 presence TTL stops new placements |
| Brain death | FOB reflexes continue (the FOB point); amnesiac reconnect documented; resume tokens are step 6 |
| Drain vs orchestrator | `.env.example` drain 600 s; compose `stop_grace_period` 660 s (grace > drain); systemd `TimeoutStopSec`/k8s equivalents in docs |
| Quiet WS killed by any middle timer | §5.2 pings + tuned edge timeouts |

---

## 9. Testing

- **Code:** NODELAY latency assertion in sim-bench; webhook URL derivation; signature validation
  behind forwarded headers (trusted and untrusted source cases); WS ping emission; TLS hot-reload
  (self-signed test certs); `ValkeyEventSink` XADD + drop-on-down (real valkey service container
  in CI).
- **Artifacts (new CI muscle):** build all images; **all-in-one smoke** — boots, Caddy terminates
  TLS via its internal CA (no ACME in CI), full sim call through the real edge→FOB WS path;
  **compose smoke** — all services healthy, a lifecycle event lands in the Valkey stream;
  **Caddy config reload during a live simulated call, asserting zero drops.**
- Existing seam-gates (loop_driver/rtc_session blob hashes), MSRV matrix, cargo-deny (now
  covering redis-rs), sim-bench single-threaded — all unchanged and must stay green.

## 10. Documentation (`docs/deploy/`)

`topologies.md` (three shapes + decision guide), `quickstart-docker.md`, `homelab.md` (CGNAT
truth: PSTN-only behind tunnels; ngrok dev path; VPS+WireGuard graduation), `aws.md` (ALB
appendix, three mandatory overrides, never-NLB-TLS), `reverse-proxies.md` (tuned-timeout snippets
for BYO nginx/HAProxy/Traefik incl. the universal 60 s idle-timeout footgun), `certificates.md`
(ACME paths, wildcard traps, duplicate-cert lockout).

**Acceptance bar: every topology doc contains a copy-paste path from zero to first call** — or
the honest statement of what cannot work there (e.g. CGNAT WebRTC).

## 11. Sequencing — slices

| Slice | Contents | Depends |
|---|---|---|
| **A** | Hygiene: §5.1 NODELAY + §5.2 pings + §5.3 trunk fixes/trusted-proxy | — |
| **B** | Images + compose + Caddyfile + smoke CI (§6, §9) | A |
| **C** | rustls Phase 1 (§5.4) | A |
| **D** | `/metrics` (§5.5) | — |
| **E** | `ValkeyEventSink` (§5.6) | — |
| **F** | Publish CI (§6.3) | B |
| **G** | `docs/deploy/` tree (§10) + ADR-0011 drafting/ratification | B for accuracy |

C–F order-independent after their deps. Additive to the ADR-0010 sequence (benchmark →
escalation → trunk → spend) — not a reordering. The artifacts ship with WebRTC calling live now;
PSTN lights up end-to-end the moment the remaining trunk work lands (trunk registration is a
placeholder NACK today; the container wiring will be ready).

## 12. Open questions deliberately left open (tracked, not blocking)

dns-persist-01 GA timeline (gates rustls Phase 2 *and* dissolves the xcaddy plugin burden);
object-storage CDR shape; multi-tenancy ADR (before CDR/config schemas ossify); brain fleet +
tap auth/resume (step 6); media-thread sharding graduation; SBC layer-2 wire contract;
the real admission-cap number from the ADR-0010 benchmark.