From 0e415c748085438a29f81e90da7bab522a6c0921 Mon Sep 17 00:00:00 2001 From: "Aaron D. Lee" Date: Sun, 5 Jul 2026 23:04:57 -0400 Subject: [PATCH] =?UTF-8?q?docs(deploy):=20certificates.md=20=E2=80=94=20A?= =?UTF-8?q?CME=20paths,=20/data=20persistence,=20wildcard=20traps=20(slice?= =?UTF-8?q?-G)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The availability-critical subsystem: no-self-signed-path (Twilio 31910), the /data-loss -> LE 5/week duplicate-cert lockout -> total-inbound-outage chain (31910 streams + 11237 webhooks) with backup command; HTTP-01 vs DNS-01 (bundled plugin set: cloudflare/route53/porkbun/hetzner/desec, duckdns excluded) vs BYO-cert rustls Phase 1 (RUTSTER_TLS_CERT/KEY, hot-reload); Caddy renewal-vs-reload distinction with the #6420/#7222 caveat; fleet patterns (central wildcard vs per-node certs) with the identical-wildcard trap and the on-demand-TLS rejection. Signed-off-by: Aaron D. Lee --- docs/deploy/certificates.md | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 docs/deploy/certificates.md diff --git a/docs/deploy/certificates.md b/docs/deploy/certificates.md new file mode 100644 index 0000000..8d5d15a --- /dev/null +++ b/docs/deploy/certificates.md @@ -0,0 +1,79 @@ +# Certificates & ACME — the availability-critical subsystem + +**A publicly-CA-trusted certificate is a hard CPaaS requirement; no self-signed path exists.** +Twilio refuses self-signed certs for webhooks and Media Streams (error +[31910](https://www.twilio.com/docs/api/errors/31910); see +[twilio.com/docs/usage/security](https://www.twilio.com/docs/usage/security)) — and the +console's cert-validation toggle is account-wide, webhook-only, and dev-only, so it is not an +escape hatch. In a 24/7 product whose calls are non-migratable there is no maintenance window: +**auto-renewal is availability-critical, not a nicety.** + +Protocol posture everywhere: TLS 1.2+1.3, mainstream ECDHE suites, **never 1.3-only** +(Twilio's TLS 1.3 client support is undocumented), and never pin Twilio's certificates. + +## The `/data` volume — read this before anything else + +Caddy keeps its certificates and ACME account state in `/data`. If you recreate the container +without that volume, Caddy re-requests the same certificate — and Let's Encrypt's +**duplicate-certificate limit (5 per week for an identical name set, +[letsencrypt.org/docs/rate-limits](https://letsencrypt.org/docs/rate-limits/))** will lock +your domain out for up to a week. The failure class is **total inbound outage**: 31910 on +Media Streams, [11237](https://www.twilio.com/docs/api/errors/11237) on webhooks, no calls in +or out of the trunk. A container recreate loop (crash-looping deploy, CI that recreates on +every push) burns all five in minutes. + +- Always mount `/data` on a persistent volume (`-v rutster-caddy-data:/data`). +- Back it up like state, because it is: + `docker run --rm -v rutster-caddy-data:/data -v "$PWD":/backup debian:stable-slim tar czf /backup/caddy-data.tgz /data` + +## ACME challenge paths + +| Path | When | Notes | +|---|---|---| +| **HTTP-01** (default) | Port 80 publicly reachable, single hostname | Zero config beyond the domain + email. | +| **DNS-01** | Wildcards (`*.pbx.domain`), CGNAT/behind-NAT hosts, no port 80 | The bundled `rutster-edge` Caddy build ships a curated plugin set: **cloudflare, route53, porkbun, hetzner, desec** (duckdns excluded — no license file). Any other DNS provider requires your own xcaddy build. DNS API credentials live in the container env — scope them to the zone. | +| **BYO-cert** (in-process rustls, Phase 1) | You already have cert distribution (corporate ACME, central wildcard issuance) | Set `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY`; the engine serves TLS itself and **hot-reloads on file change without dropping live WS**. You own renewal delivery. | + +In-binary ACME (Phase 2) is deliberately deferred behind named triggers — among them Let's +Encrypt's dns-persist-01 reaching GA, which would dissolve the DNS-plugin matrix entirely. See +[ADR-0011](../adr/0011-deployment-topology.md). + +## Caddy reload vs live calls + +Routine renewals are safe: Caddy swaps certs **in memory, zero-downtime — the routine periodic +event drops nothing.** Config *reloads* (editing the Caddyfile) are the risk: the mitigation +(`stream_close_delay` above max call duration) has an open upstream bug trail +([caddy#6420](https://github.com/caddyserver/caddy/issues/6420), +[caddy#7222](https://github.com/caddyserver/caddy/issues/7222)), which is why the shipped +artifacts carry a CI e2e test for config-reload-during-live-call. Operator rule: **do not edit +the Caddyfile while calls are live** unless your image version passed that test; drain first +(`/readyz` returns 503 while draining). + +## Fleet certificate patterns (T3) + +Node-addressed placement means every node needs its own publicly-valid TLS name — this is +CPaaS-imposed (`` URLs route on hostname only), not a proxy choice. Two blessed +patterns: + +1. **Wildcard `*.pbx.domain`** via DNS-01, issued **centrally** and distributed to nodes. + Understand the trade: the wildcard private key on every node means one compromised node + burns the whole namespace. +2. **Per-node distinct certs** (`node-1.pbx.domain`, `node-2.…`) — no shared key material. + Renewals are exempt from Let's Encrypt's per-domain limits, so this scales with fleet size. + +Traps: + +- **N nodes independently requesting the identical wildcard** are N duplicate requests — the + 5/week duplicate-cert limit locks the fleet out. Central issuance or distinct names only. +- **Caddy on-demand TLS is rejected for node names**: the first TLS handshake to a fresh name + blocks for seconds on issuance, colliding with the sub-second webhook budget (Twilio's hard + cap is 15 s, UX budget sub-second), and its rate-limit knobs are deprecated. Node names are + known at provision time — pre-issue their certs. + +## Troubleshooting + +| Symptom | Fix | +|---|---| +| Twilio 31910 on Media Streams | Cert not publicly trusted / expired / SNI mismatch. Check `openssl s_client -connect pbx.example.com:443 -servername pbx.example.com`. | +| `rateLimited` / `too many certificates` in Caddy logs | You hit the duplicate-cert limit — almost always a lost `/data` volume. Restore the volume or wait out the window; then fix the volume mount so it never recurs. | +| DNS-01 fails for your provider | Provider not in the bundled plugin set — use a delegated zone on a supported provider, or build your own xcaddy image. |