docs(deploy): certificates.md — ACME paths, /data persistence, wildcard traps (slice-G)

The availability-critical subsystem: no-self-signed-path (Twilio 31910),
the /data-loss -> LE 5/week duplicate-cert lockout -> total-inbound-outage
chain (31910 streams + 11237 webhooks) with backup command; HTTP-01 vs
DNS-01 (bundled plugin set: cloudflare/route53/porkbun/hetzner/desec,
duckdns excluded) vs BYO-cert rustls Phase 1 (RUTSTER_TLS_CERT/KEY,
hot-reload); Caddy renewal-vs-reload distinction with the #6420/#7222
caveat; fleet patterns (central wildcard vs per-node certs) with the
identical-wildcard trap and the on-demand-TLS rejection.

Signed-off-by: Aaron D. Lee <himself@adlee.work>

docs/deploy/certificates.md

@@ -0,0 +1,79 @@
+# Certificates & ACME — the availability-critical subsystem
+
+**A publicly-CA-trusted certificate is a hard CPaaS requirement; no self-signed path exists.**
+Twilio refuses self-signed certs for webhooks and Media Streams (error
+[31910](https://www.twilio.com/docs/api/errors/31910); see
+[twilio.com/docs/usage/security](https://www.twilio.com/docs/usage/security)) — and the
+console's cert-validation toggle is account-wide, webhook-only, and dev-only, so it is not an
+escape hatch. In a 24/7 product whose calls are non-migratable there is no maintenance window:
+**auto-renewal is availability-critical, not a nicety.**
+
+Protocol posture everywhere: TLS 1.2+1.3, mainstream ECDHE suites, **never 1.3-only**
+(Twilio's TLS 1.3 client support is undocumented), and never pin Twilio's certificates.
+
+## The `/data` volume — read this before anything else
+
+Caddy keeps its certificates and ACME account state in `/data`. If you recreate the container
+without that volume, Caddy re-requests the same certificate — and Let's Encrypt's
+**duplicate-certificate limit (5 per week for an identical name set,
+[letsencrypt.org/docs/rate-limits](https://letsencrypt.org/docs/rate-limits/))** will lock
+your domain out for up to a week. The failure class is **total inbound outage**: 31910 on
+Media Streams, [11237](https://www.twilio.com/docs/api/errors/11237) on webhooks, no calls in
+or out of the trunk. A container recreate loop (crash-looping deploy, CI that recreates on
+every push) burns all five in minutes.
+
+- Always mount `/data` on a persistent volume (`-v rutster-caddy-data:/data`).
+- Back it up like state, because it is:
+  `docker run --rm -v rutster-caddy-data:/data -v "$PWD":/backup debian:stable-slim tar czf /backup/caddy-data.tgz /data`
+
+## ACME challenge paths
+
+| Path | When | Notes |
+|---|---|---|
+| **HTTP-01** (default) | Port 80 publicly reachable, single hostname | Zero config beyond the domain + email. |
+| **DNS-01** | Wildcards (`*.pbx.domain`), CGNAT/behind-NAT hosts, no port 80 | The bundled `rutster-edge` Caddy build ships a curated plugin set: **cloudflare, route53, porkbun, hetzner, desec** (duckdns excluded — no license file). Any other DNS provider requires your own xcaddy build. DNS API credentials live in the container env — scope them to the zone. |
+| **BYO-cert** (in-process rustls, Phase 1) | You already have cert distribution (corporate ACME, central wildcard issuance) | Set `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY`; the engine serves TLS itself and **hot-reloads on file change without dropping live WS**. You own renewal delivery. |
+
+In-binary ACME (Phase 2) is deliberately deferred behind named triggers — among them Let's
+Encrypt's dns-persist-01 reaching GA, which would dissolve the DNS-plugin matrix entirely. See
+[ADR-0011](../adr/0011-deployment-topology.md).
+
+## Caddy reload vs live calls
+
+Routine renewals are safe: Caddy swaps certs **in memory, zero-downtime — the routine periodic
+event drops nothing.** Config *reloads* (editing the Caddyfile) are the risk: the mitigation
+(`stream_close_delay` above max call duration) has an open upstream bug trail
+([caddy#6420](https://github.com/caddyserver/caddy/issues/6420),
+[caddy#7222](https://github.com/caddyserver/caddy/issues/7222)), which is why the shipped
+artifacts carry a CI e2e test for config-reload-during-live-call. Operator rule: **do not edit
+the Caddyfile while calls are live** unless your image version passed that test; drain first
+(`/readyz` returns 503 while draining).
+
+## Fleet certificate patterns (T3)
+
+Node-addressed placement means every node needs its own publicly-valid TLS name — this is
+CPaaS-imposed (`<Stream>` URLs route on hostname only), not a proxy choice. Two blessed
+patterns:
+
+1. **Wildcard `*.pbx.domain`** via DNS-01, issued **centrally** and distributed to nodes.
+   Understand the trade: the wildcard private key on every node means one compromised node
+   burns the whole namespace.
+2. **Per-node distinct certs** (`node-1.pbx.domain`, `node-2.…`) — no shared key material.
+   Renewals are exempt from Let's Encrypt's per-domain limits, so this scales with fleet size.
+
+Traps:
+
+- **N nodes independently requesting the identical wildcard** are N duplicate requests — the
+  5/week duplicate-cert limit locks the fleet out. Central issuance or distinct names only.
+- **Caddy on-demand TLS is rejected for node names**: the first TLS handshake to a fresh name
+  blocks for seconds on issuance, colliding with the sub-second webhook budget (Twilio's hard
+  cap is 15 s, UX budget sub-second), and its rate-limit knobs are deprecated. Node names are
+  known at provision time — pre-issue their certs.
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| Twilio 31910 on Media Streams | Cert not publicly trusted / expired / SNI mismatch. Check `openssl s_client -connect pbx.example.com:443 -servername pbx.example.com`. |
+| `rateLimited` / `too many certificates` in Caddy logs | You hit the duplicate-cert limit — almost always a lost `/data` volume. Restore the volume or wait out the window; then fix the volume mount so it never recurs. |
+| DNS-01 fails for your provider | Provider not in the bundled plugin set — use a delegated zone on a supported provider, or build your own xcaddy image. |