docs(adr): ADR-0011 deployment topology — one binary, three blessed shapes (Proposed) (slice-G)
Drafted from the 2026-07-05 deployment-topology spec §2–§4: T1 solo / T2 modular / T3 fleet (paper), the edge doctrine (bundled Caddy; rustls Phase 1 now, Phase 2 behind four named triggers), the fleet cert story (wildcard vs per-node; on-demand TLS rejected), node-addressed placement via Valkey presence (routing tier + distinguished control node rejected), and the rejected shapes (FOB decomposition, serverless, tunnels-in-prod, NLB-TLS, k8s manifests). Status Proposed — ratification per repo process. Closes the topology ADR reserved by ADR-0008's worked example. Signed-off-by: Aaron D. Lee <himself@adlee.work>
This commit is contained in:
249
docs/adr/0011-deployment-topology.md
Normal file
249
docs/adr/0011-deployment-topology.md
Normal file
@@ -0,0 +1,249 @@
|
|||||||
|
# ADR-0011 — Deployment topology: one binary, three blessed shapes
|
||||||
|
|
||||||
|
- **Status:** Proposed
|
||||||
|
- **Date:** 2026-07-05
|
||||||
|
- **Closes:** the deployment-topology ADR reserved by [ADR-0008](0008-fob-and-green-zone.md)'s
|
||||||
|
worked example ("the deployment topology itself may get its own ADR once that design closes").
|
||||||
|
- **Origin:** [2026-07-05 deployment-topology spec](../superpowers/specs/2026-07-05-deployment-topology-design.md)
|
||||||
|
§2–§4. The TLS/edge fork was researched via a six-family survey — see the companion
|
||||||
|
[TLS/edge decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md).
|
||||||
|
- **Related:** [ADR-0002](0002-north-star-and-fused-core.md) (the fused per-call vertical no
|
||||||
|
topology may split), [ADR-0005](0005-event-bus.md) (Valkey = bus + KV + presence; media never
|
||||||
|
rides the bus; the bus is never billing source-of-truth), [ADR-0006](0006-ingress-posture.md)
|
||||||
|
(two inbound protocols only; the tap is egress-only), [ADR-0007](0007-trunk-rented-transport.md)
|
||||||
|
(rented transport ⇒ public HTTPS/WSS reachability is non-negotiable),
|
||||||
|
[ADR-0008](0008-fob-and-green-zone.md) (the FOB/green-zone rule this ADR applies to
|
||||||
|
deployment), [ADR-0009](0009-spend-gate-honest-rescope.md) (ledger trait: in-memory
|
||||||
|
single-node, Valkey-backed fleets), [ADR-0010](0010-spearhead-benchmark-sim-harness.md)
|
||||||
|
(sequencing this ADR must not reorder), README pillar 6 ("one binary, one bus, one deploy").
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
[ADR-0008](0008-fob-and-green-zone.md) made the build-vs-reuse doctrine physical in a worked
|
||||||
|
example — the all-in-one container — and reserved the topology ADR for when the design closed.
|
||||||
|
Two things have since closed it: the rented transport
|
||||||
|
([ADR-0007](0007-trunk-rented-transport.md)) is merged, so every deployment must now present a
|
||||||
|
public HTTPS/WSS edge to a CPaaS; and slice-5/seams landed the operational primitives a
|
||||||
|
deployment story needs (drain lifecycle, `/readyz`, admission cap, `MediaAddressConfig`,
|
||||||
|
`EventSink`).
|
||||||
|
|
||||||
|
The CPaaS side imposes a hard external spec on any edge rutster ships (researched in the
|
||||||
|
[decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md); restated here
|
||||||
|
because these invariants are load-bearing for every decision below):
|
||||||
|
|
||||||
|
1. Publicly-CA-trusted cert; **no self-signed path exists** (Twilio error 31910). Auto-renewal
|
||||||
|
is availability-critical.
|
||||||
|
2. wss:// on 443, open to the whole internet: Twilio publishes no egress IPs and offers no
|
||||||
|
mTLS. Auth is application-layer only (HMAC signatures).
|
||||||
|
3. One WS per call, up to 24 h, 50 msg/s/direction at 20 ms cadence; zero tolerance for frame
|
||||||
|
buffering, connection-lifetime caps, or write-side idle timers. Neither Twilio nor Telnyx
|
||||||
|
documents WS keepalive — keepalive is entirely our job.
|
||||||
|
4. `<Stream>` URLs carry **no query strings** — wss routing keys on hostname/path only.
|
||||||
|
5. `X-Twilio-Signature` is HMAC over the URL *as Twilio saw it*: the edge must forward
|
||||||
|
`X-Forwarded-Proto/Host` honestly and the FOB must reconstruct the public URL.
|
||||||
|
6. Webhook budget: sub-second for UX, 15 s hard cap — no blocking issuance in the handshake
|
||||||
|
path.
|
||||||
|
7. TLS 1.2+1.3, mainstream ECDHE, never 1.3-only; never pin Twilio certs.
|
||||||
|
|
||||||
|
Design for Twilio; Telnyx/Vonage come for free (strictly less demanding, plus optional
|
||||||
|
source-IP allowlisting as extra hardening Twilio can't use).
|
||||||
|
|
||||||
|
On the other side, the FOB's own physics constrain every shape: per-call state is in-process
|
||||||
|
and non-migratable (the fused vertical, [ADR-0002](0002-north-star-and-fused-core.md)), WebRTC
|
||||||
|
media is per-session UDP direct to the process, and calls run for hours. Any topology that
|
||||||
|
assumes migratable, short-lived, connectionless workloads is disqualified before it starts.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
**The FOB never decomposes; topology is a configuration property, not a code property.** The
|
||||||
|
same binary runs in every blessed shape; only `RUTSTER_*` env and what is deployed alongside it
|
||||||
|
differ. "Microservices" in rutster means *green-zone services get their own
|
||||||
|
processes/containers* — never that the per-call vertical splits
|
||||||
|
([ADR-0002](0002-north-star-and-fused-core.md)).
|
||||||
|
|
||||||
|
### The three blessed shapes
|
||||||
|
|
||||||
|
#### T1 — Solo (all-in-one container) — shipped artifact
|
||||||
|
|
||||||
|
One image (`rutster-allinone`), s6-overlay as PID 1 supervising four processes:
|
||||||
|
|
||||||
|
| Process | Role | Binding |
|
||||||
|
|---|---|---|
|
||||||
|
| `caddy` | Edge: ACME, TLS termination, WS proxy to FOB | :443/:80 public |
|
||||||
|
| `rutster` (FOB) | Engine | 127.0.0.1:8080 (behind Caddy); media UDP direct |
|
||||||
|
| `rutster-brain-realtime` | Brain | 127.0.0.1:8082 loopback tap (as today) |
|
||||||
|
| `valkey-server` | Dark on day one except `EventSink` | 127.0.0.1:6379 |
|
||||||
|
|
||||||
|
Two volumes, both non-negotiable: `/data` (Caddy cert/ACME state — loss risks the Let's
|
||||||
|
Encrypt duplicate-cert lockout, a total-inbound-outage class of failure) and `/var/lib/valkey`
|
||||||
|
(stream/state persistence). WebRTC media UDP goes **direct to the FOB**, never through Caddy:
|
||||||
|
host networking recommended; otherwise published `RUTSTER_MEDIA_PORT_RANGE` +
|
||||||
|
`RUTSTER_MEDIA_ADVERTISED_IP` (NAT 1:1 model, no STUN — unchanged from slice-5/seams).
|
||||||
|
|
||||||
|
Valkey ships bundled from v1 **before most consumers exist**: the operator contract (volumes,
|
||||||
|
ports, upgrade shape) is stable from day one, so the spend ledger
|
||||||
|
([ADR-0009](0009-spend-gate-honest-rescope.md)) and the fleet directory land later without
|
||||||
|
changing the deployment shape. Cost accepted: ~15 MB and one mostly-idle process.
|
||||||
|
|
||||||
|
#### T2 — Modular (compose stack) — shipped artifact, the reference deployment
|
||||||
|
|
||||||
|
Services: `caddy` (image `rutster-edge`), `engine` (`rutster-engine`), `brain`
|
||||||
|
(`rutster-brain`), `valkey` (upstream `valkey/valkey`).
|
||||||
|
|
||||||
|
- **The brain shares the engine's network namespace** (`network_mode: "service:engine"`) so
|
||||||
|
the loopback-only tap posture (`resolve_tap_url` rejects non-loopback until step 6) survives
|
||||||
|
unchanged. The brain graduates to its own netns when the wss:// tap lands (step 6).
|
||||||
|
- Valkey is its own service on the compose network — network-near, satisfying
|
||||||
|
[ADR-0005](0005-event-bus.md)'s sub-ms rule.
|
||||||
|
- `stop_grace_period: 660s` paired with `RUTSTER_DRAIN_DEADLINE_SECS=600` (grace > drain).
|
||||||
|
- BYO-proxy supported: disable the `caddy` service; the engine keeps its plaintext `:8080`
|
||||||
|
mode; tuned-timeout snippets for nginx/HAProxy/Traefik ship in
|
||||||
|
[`docs/deploy/reverse-proxies.md`](../deploy/reverse-proxies.md).
|
||||||
|
|
||||||
|
#### T3 — Fleet — ADR chapter only (paper; no code in this epoch)
|
||||||
|
|
||||||
|
N **symmetric** nodes, each running the T1 process set *minus* valkey, plus a shared green
|
||||||
|
zone: one Valkey (presence + directory + the [ADR-0009](0009-spend-gate-honest-rescope.md)
|
||||||
|
spend ledger) and object storage (named as required, still deferred). Wildcard DNS
|
||||||
|
`*.pbx.domain`; per-node certs (see the fleet cert story below). The per-node admission cap is
|
||||||
|
the [ADR-0010](0010-spearhead-benchmark-sim-harness.md) benchmark number (replacing the
|
||||||
|
`RUTSTER_MAX_SESSIONS=64` placeholder). Scale-in is drain-then-terminate only; spot instances
|
||||||
|
are rejected for the engine tier (2026-07-04 review — a reclaimed node kills every
|
||||||
|
non-migratable call on it).
|
||||||
|
|
||||||
|
### Edge doctrine
|
||||||
|
|
||||||
|
#### Bundled edge: Caddy (both shipped artifacts, same build, same Caddyfile)
|
||||||
|
|
||||||
|
Custom xcaddy build (the `rutster-edge` image): Caddy core (Apache-2.0) + a curated DNS-01
|
||||||
|
plugin set — cloudflare, route53, porkbun, hetzner, desec (duckdns excluded: no license file).
|
||||||
|
Zero-downtime in-memory cert renewal; verified protocol-unaware WS tunneling (no frame
|
||||||
|
buffering). Config is a ~6-line Caddyfile with tuned timeouts, honest `X-Forwarded-*`, and
|
||||||
|
`stream_close_delay` above max call duration — **plus a CI e2e case for
|
||||||
|
config-reload-during-live-call, because the upstream mitigation has an open bug trail
|
||||||
|
(caddy #6420/#7222) and is not trusted untested.**
|
||||||
|
|
||||||
|
#### In-process TLS: staged — the ratified end-state, not adopted wholesale now
|
||||||
|
|
||||||
|
[ADR-0008](0008-fob-and-green-zone.md) claims the trust-domain edge as security-constitutive
|
||||||
|
FOB territory, and str0m already terminates DTLS-SRTP in-process on the same aws-lc-rs
|
||||||
|
provider — "keep TLS out of the FOB" is not the objection. The objection is economics: owning
|
||||||
|
an ACME renewal state machine forever, in a 45-day-cert world.
|
||||||
|
|
||||||
|
- **Phase 1 (this epoch):** `axum-server` + rustls with operator-supplied certs
|
||||||
|
(`RUTSTER_TLS_CERT`/`RUTSTER_TLS_KEY`), hot-reload without dropping live WS.
|
||||||
|
Plaintext-behind-Caddy stays the artifact default.
|
||||||
|
- **Phase 2 (in-binary ACME) triggers, named:** (a) Let's Encrypt dns-persist-01 confirmed GA
|
||||||
|
(closes the wildcard/CGNAT issuance gap; instant-acme already parses it); (b) a field
|
||||||
|
incident where Caddy reload/`stream_close_delay` drops live calls; (c) fleet or
|
||||||
|
VPS-forwarder topology making encrypted-to-the-binary a selling point; (d) maintenance
|
||||||
|
fatigue with the curated xcaddy plugin build. Until a trigger fires, Phase 2 is deliberately
|
||||||
|
not built — no turnkey ACME crate fits the tree today (rustls-acme conflicts with the
|
||||||
|
deny.toml rcgen/x509-parser posture).
|
||||||
|
|
||||||
|
#### Fleet cert story
|
||||||
|
|
||||||
|
Per-node public TLS names are CPaaS-imposed (invariant 4 + node-addressed placement). Two
|
||||||
|
blessed patterns:
|
||||||
|
|
||||||
|
1. **Wildcard `*.pbx.domain` via DNS-01** with central issuance/distribution.
|
||||||
|
2. **Per-node distinct certs** (`node-N.domain`) where the wildcard private key must not live
|
||||||
|
on every node — one compromised node otherwise burns the whole namespace.
|
||||||
|
|
||||||
|
Traps, stated so nobody re-discovers them in production: N nodes independently requesting the
|
||||||
|
*identical* wildcard hit the Let's Encrypt duplicate-cert limit (5/week) — distinct names or
|
||||||
|
central distribution only. **Caddy on-demand TLS is rejected for node names**: the first
|
||||||
|
handshake to a fresh name blocks seconds on issuance, colliding with the sub-second webhook
|
||||||
|
budget (invariant 6), and its rate-limit knobs are deprecated. Node names known at provision
|
||||||
|
time get pre-issued certs.
|
||||||
|
|
||||||
|
### Fleet placement paradigm (the trunk webhook handler and session API grow *toward* this; no fleet code in this epoch)
|
||||||
|
|
||||||
|
**Node-addressed URLs + Valkey presence; no router tier.** (Decision 2026-07-05 over the two
|
||||||
|
rejected alternatives below.)
|
||||||
|
|
||||||
|
- Nodes heartbeat capacity into Valkey (TTL'd presence keys — the
|
||||||
|
[ADR-0005](0005-event-bus.md) presence layer).
|
||||||
|
- Placement happens **at answer time** by whichever node the request lands on (`api.domain`,
|
||||||
|
DNS round-robin): read fleet capacity → pick owner → return TwiML whose Stream URL is
|
||||||
|
`wss://node-K.pbx.domain/...`. The media WS dials the owning node directly — zero extra
|
||||||
|
hops, consistent with connection-follows-ownership. Twilio's answer-time Stream URL is
|
||||||
|
purpose-shaped for this: invariant 4 permits hostname routing.
|
||||||
|
- WebRTC: the create-session response carries the owner's node-addressed base URL + advertised
|
||||||
|
media IP (UDP is already direct).
|
||||||
|
- Misdirected control ops (e.g. a DELETE hitting a non-owner) consult the Valkey directory and
|
||||||
|
**307-redirect** to the owner.
|
||||||
|
- **Degraded mode (Valkey dark): the answering node self-places if it has capacity**; presence
|
||||||
|
heals via TTL on recovery. Fail-safe degradation, consistent with
|
||||||
|
[ADR-0009](0009-spend-gate-honest-rescope.md)'s posture.
|
||||||
|
- Single-node is the degenerate case of the same code: a one-entry directory.
|
||||||
|
|
||||||
|
**Rejected: a routing tier that proxies everything.** It would put every 20 ms media frame
|
||||||
|
through an extra userspace hop and re-create exactly the SPOF the symmetric design avoids —
|
||||||
|
and it buys nothing, because the CPaaS already lets the answer-time URL do the routing.
|
||||||
|
|
||||||
|
**Rejected: a distinguished control node.** It breaks node symmetry (two artifacts, two
|
||||||
|
failure stories, a promotion protocol) to solve a problem the presence directory already
|
||||||
|
solves; placement needs fleet state either way, and Valkey is already the fleet-state home
|
||||||
|
([ADR-0005](0005-event-bus.md)).
|
||||||
|
|
||||||
|
### Rejected shapes
|
||||||
|
|
||||||
|
- **Decomposing the FOB into services** — rejected **permanently**, not deferred.
|
||||||
|
Media/reflex/tap/spend stay one process, one trust domain
|
||||||
|
([ADR-0002](0002-north-star-and-fused-core.md)).
|
||||||
|
- **Serverless.** Lambdas are categorically unfit for the engine: non-migratable multi-hour
|
||||||
|
calls, per-session UDP sockets, in-process media state. Webhook-on-Lambda buys nothing
|
||||||
|
because placement needs fleet state anyway.
|
||||||
|
- **Tunnels in production.** Cloudflare Tunnel and ngrok see plaintext audio at the vendor
|
||||||
|
edge (an unconsented subprocessor — DPA/BAA/PCI failure), document mid-call WS kills, and
|
||||||
|
cloudflared has an open Twilio handshake bug (cloudflared #1465). ngrok is the blessed
|
||||||
|
5-minute demo path (the only tunnel with a proven Media Streams record); Tailscale Funnel is
|
||||||
|
the privacy-clean single-user demo (TLS terminates on-node), unsizable beyond that. **No
|
||||||
|
tunnel carries inbound UDP → homelab behind CGNAT is PSTN-only, period.** The production
|
||||||
|
graduation path is a cheap VPS + WireGuard as a dumb TCP forwarder with TLS terminating at
|
||||||
|
home, or running the engine on the VPS — see
|
||||||
|
[`docs/deploy/homelab.md`](../deploy/homelab.md).
|
||||||
|
- **NLB TLS listeners — never.** Fixed 350 s idle timeout that silently drops flows, and no
|
||||||
|
HTTP context, so `X-Forwarded-*` headers for signature validation (invariant 5) don't exist.
|
||||||
|
The ALB path is documented as a docs-only appendix
|
||||||
|
([`docs/deploy/aws.md`](../deploy/aws.md)), never a shipped artifact.
|
||||||
|
- **k8s manifests/Helm.** Compose-first (README pillar 6). k8s notes live in docs only (drain
|
||||||
|
vs `terminationGracePeriodSeconds`); manifests are a later rung if demand appears.
|
||||||
|
|
||||||
|
### Deferred, named (so the boundary is deliberate)
|
||||||
|
|
||||||
|
- **T3 fleet implementation** (presence heartbeats, directory redirects, placement code).
|
||||||
|
- **The zero-egress / air-gap profile** (SBC layer-2 ingress per
|
||||||
|
[ADR-0007](0007-trunk-rented-transport.md) layer 2 + a self-hosted brain): a defined
|
||||||
|
profile, no code this epoch. T1 is about one-command simplicity, not isolation.
|
||||||
|
- **In-binary ACME (rustls Phase 2)** — behind the four named triggers above.
|
||||||
|
- **Durable CDR pipeline**: `ValkeyEventSink` is evidence preservation on a capped stream —
|
||||||
|
explicitly NOT the billing ledger ([ADR-0005](0005-event-bus.md) source-of-truth rule).
|
||||||
|
CDR → object storage stays deferred.
|
||||||
|
- **Multi-tenancy** (needs its own ADR before schemas ossify); wss:// tap + brain fleet +
|
||||||
|
resume tokens (step 6); media-thread sharding graduation; arm64 images.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- **Positive:** one binary and one mental model across all three shapes — an operator's
|
||||||
|
knowledge transfers from `docker run` to a fleet; the CPaaS invariants are satisfied by a
|
||||||
|
decade-hardened ACME loop (Caddy) rather than a DIY renewal state machine; the fleet
|
||||||
|
paradigm needs zero new infrastructure beyond what [ADR-0005](0005-event-bus.md) already
|
||||||
|
bundles; the rejected shapes are written down, so nobody burns a week discovering that
|
||||||
|
Lambda or NLB-TLS can't work.
|
||||||
|
- **Negative:** the curated xcaddy DNS-plugin set is a permanent maintenance surface until
|
||||||
|
dns-persist-01 GA; Caddy config-reload behavior vs live calls is mitigated
|
||||||
|
(`stream_close_delay` + CI e2e) but has an open upstream bug trail; homelab WebRTC behind
|
||||||
|
CGNAT is unsolved **by design** — only the VPS graduation fixes it, and the docs must say so
|
||||||
|
loudly or operators will read "homelab supported" as including browser calls.
|
||||||
|
- **Mitigation:** the Phase-2 triggers convert both negatives into a planned exit; the
|
||||||
|
`docs/deploy/` tree ships with a copy-paste-to-first-call acceptance bar so the honest
|
||||||
|
limits are in the operator's face, not in a footnote.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [2026-07-05 deployment-topology spec](../superpowers/specs/2026-07-05-deployment-topology-design.md) — §2–§4 are this ADR's source
|
||||||
|
- [TLS/edge decision brief](../superpowers/specs/2026-07-05-tls-edge-decision-brief.md) — six-family survey + the CPaaS ground truth (external citations live there)
|
||||||
|
- [ADR-0002](0002-north-star-and-fused-core.md) · [ADR-0005](0005-event-bus.md) · [ADR-0006](0006-ingress-posture.md) · [ADR-0007](0007-trunk-rented-transport.md) · [ADR-0008](0008-fob-and-green-zone.md) · [ADR-0009](0009-spend-gate-honest-rescope.md) · [ADR-0010](0010-spearhead-benchmark-sim-harness.md)
|
||||||
|
- The `docs/deploy/` tree: [topologies](../deploy/topologies.md) · [quickstart-docker](../deploy/quickstart-docker.md) · [homelab](../deploy/homelab.md) · [aws](../deploy/aws.md) · [reverse-proxies](../deploy/reverse-proxies.md) · [certificates](../deploy/certificates.md)
|
||||||
Reference in New Issue
Block a user