rutster/docs/deploy/certificates.md

Certificates & ACME — the availability-critical subsystem

A publicly-CA-trusted certificate is a hard CPaaS requirement; no self-signed path exists. Twilio refuses self-signed certs for webhooks and Media Streams (error 31910; see twilio.com/docs/usage/security) — and the console's cert-validation toggle is account-wide, webhook-only, and dev-only, so it is not an escape hatch. In a 24/7 product whose calls are non-migratable there is no maintenance window: auto-renewal is availability-critical, not a nicety.

Protocol posture everywhere: TLS 1.2+1.3, mainstream ECDHE suites, never 1.3-only (Twilio's TLS 1.3 client support is undocumented), and never pin Twilio's certificates.

The /data volume — read this before anything else

Caddy keeps its certificates and ACME account state in /data. If you recreate the container without that volume, Caddy re-requests the same certificate — and Let's Encrypt's duplicate-certificate limit (5 per week for an identical name set, letsencrypt.org/docs/rate-limits) will lock your domain out for up to a week. The failure class is total inbound outage: 31910 on Media Streams, 11237 on webhooks, no calls in or out of the trunk. A container recreate loop (crash-looping deploy, CI that recreates on every push) burns all five in minutes.

• Always mount /data on a persistent volume (-v rutster-caddy-data:/data).
• Back it up like state, because it is: docker run --rm -v rutster-caddy-data:/data -v "$PWD":/backup debian:stable-slim tar czf /backup/caddy-data.tgz /data

ACME challenge paths

Path	When	Notes
HTTP-01 (default)	Port 80 publicly reachable, single hostname	Zero config beyond the domain + email.
DNS-01	Wildcards (*.pbx.domain), CGNAT/behind-NAT hosts, no port 80	The bundled rutster-edge Caddy build ships a curated plugin set: cloudflare, route53, porkbun, hetzner, desec (duckdns excluded — no license file). Any other DNS provider requires your own xcaddy build. DNS API credentials live in the container env — scope them to the zone.
BYO-cert (in-process rustls, Phase 1)	You already have cert distribution (corporate ACME, central wildcard issuance)	Set RUTSTER_TLS_CERT / RUTSTER_TLS_KEY; the engine serves TLS itself and hot-reloads on file change without dropping live WS. You own renewal delivery.

In-binary ACME (Phase 2) is deliberately deferred behind named triggers — among them Let's Encrypt's dns-persist-01 reaching GA, which would dissolve the DNS-plugin matrix entirely. See ADR-0011.

Caddy reload vs live calls

Routine renewals are safe: Caddy swaps certs in memory, zero-downtime — the routine periodic event drops nothing. Config reloads (editing the Caddyfile) are the risk: the mitigation (stream_close_delay above max call duration) has an open upstream bug trail (caddy#6420, caddy#7222), which is why the shipped artifacts carry a CI e2e test for config-reload-during-live-call. Operator rule: do not edit the Caddyfile while calls are live unless your image version passed that test; drain first (/readyz returns 503 while draining).

Fleet certificate patterns (T3)

Node-addressed placement means every node needs its own publicly-valid TLS name — this is CPaaS-imposed (<Stream> URLs route on hostname only), not a proxy choice. Two blessed patterns:

1. Wildcard *.pbx.domain via DNS-01, issued centrally and distributed to nodes. Understand the trade: the wildcard private key on every node means one compromised node burns the whole namespace.
2. Per-node distinct certs (node-1.pbx.domain, node-2.…) — no shared key material. Renewals are exempt from Let's Encrypt's per-domain limits, so this scales with fleet size.

Traps:

• N nodes independently requesting the identical wildcard are N duplicate requests — the 5/week duplicate-cert limit locks the fleet out. Central issuance or distinct names only.
• Caddy on-demand TLS is rejected for node names: the first TLS handshake to a fresh name blocks for seconds on issuance, colliding with the sub-second webhook budget (Twilio's hard cap is 15 s, UX budget sub-second), and its rate-limit knobs are deprecated. Node names are known at provision time — pre-issue their certs.

Troubleshooting

Symptom	Fix
Twilio 31910 on Media Streams	Cert not publicly trusted / expired / SNI mismatch. Check openssl s_client -connect pbx.example.com:443 -servername pbx.example.com.
rateLimited / too many certificates in Caddy logs	You hit the duplicate-cert limit — almost always a lost /data volume. Restore the volume or wait out the window; then fix the volume mount so it never recurs.
DNS-01 fails for your provider	Provider not in the bundled plugin set — use a delegated zone on a supported provider, or build your own xcaddy image.