rutster/docs/deploy/certificates.md

# Certificates & ACME — the availability-critical subsystem

**A publicly-CA-trusted certificate is a hard CPaaS requirement; no self-signed path exists.**
Twilio refuses self-signed certs for webhooks and Media Streams (error
[31910](https://www.twilio.com/docs/api/errors/31910); see
[twilio.com/docs/usage/security](https://www.twilio.com/docs/usage/security)) — and the
console's cert-validation toggle is account-wide, webhook-only, and dev-only, so it is not an
escape hatch. In a 24/7 product whose calls are non-migratable there is no maintenance window:
**auto-renewal is availability-critical, not a nicety.**

Protocol posture everywhere: TLS 1.2+1.3, mainstream ECDHE suites, **never 1.3-only**
(Twilio's TLS 1.3 client support is undocumented), and never pin Twilio's certificates.

## The `/data` volume — read this before anything else

Caddy keeps its certificates and ACME account state in `/data`. If you recreate the container
without that volume, Caddy re-requests the same certificate — and Let's Encrypt's
**duplicate-certificate limit (5 per week for an identical name set,
[letsencrypt.org/docs/rate-limits](https://letsencrypt.org/docs/rate-limits/))** will lock
your domain out for up to a week. The failure class is **total inbound outage**: 31910 on
Media Streams, [11237](https://www.twilio.com/docs/api/errors/11237) on webhooks, no calls in
or out of the trunk. A container recreate loop (crash-looping deploy, CI that recreates on
every push) burns all five in minutes.

- Always mount `/data` on a persistent volume (`-v rutster-caddy-data:/data`).
- Back it up like state, because it is:
  `docker run --rm -v rutster-caddy-data:/data -v "$PWD":/backup debian:stable-slim tar czf /backup/caddy-data.tgz /data`

## ACME challenge paths

| Path | When | Notes |
|---|---|---|
| **HTTP-01** (default) | Port 80 publicly reachable, single hostname | Zero config beyond the domain + email. |
| **DNS-01** | Wildcards (`*.pbx.domain`), CGNAT/behind-NAT hosts, no port 80 | The bundled `rutster-edge` Caddy build ships a curated plugin set: **cloudflare, route53, porkbun, hetzner, desec** (duckdns excluded — no license file). Any other DNS provider requires your own xcaddy build. DNS API credentials live in the container env — scope them to the zone. |
| **BYO-cert** (in-process rustls, Phase 1) | You already have cert distribution (corporate ACME, central wildcard issuance) | Set `RUTSTER_TLS_CERT` / `RUTSTER_TLS_KEY`; the engine serves TLS itself and **hot-reloads on file change without dropping live WS**. You own renewal delivery. |

In-binary ACME (Phase 2) is deliberately deferred behind named triggers — among them Let's
Encrypt's dns-persist-01 reaching GA, which would dissolve the DNS-plugin matrix entirely. See
[ADR-0011](../adr/0011-deployment-topology.md).

## Caddy reload vs live calls

Routine renewals are safe: Caddy swaps certs **in memory, zero-downtime — the routine periodic
event drops nothing.** Config *reloads* (editing the Caddyfile) are the risk: the mitigation
(`stream_close_delay` above max call duration) has an open upstream bug trail
([caddy#6420](https://github.com/caddyserver/caddy/issues/6420),
[caddy#7222](https://github.com/caddyserver/caddy/issues/7222)), which is why the shipped
artifacts carry a CI e2e test for config-reload-during-live-call. Operator rule: **do not edit
the Caddyfile while calls are live** unless your image version passed that test; drain first
(`/readyz` returns 503 while draining).

## Fleet certificate patterns (T3)

Node-addressed placement means every node needs its own publicly-valid TLS name — this is
CPaaS-imposed (`<Stream>` URLs route on hostname only), not a proxy choice. Two blessed
patterns:

1. **Wildcard `*.pbx.domain`** via DNS-01, issued **centrally** and distributed to nodes.
   Understand the trade: the wildcard private key on every node means one compromised node
   burns the whole namespace.
2. **Per-node distinct certs** (`node-1.pbx.domain`, `node-2.…`) — no shared key material.
   Renewals are exempt from Let's Encrypt's per-domain limits, so this scales with fleet size.

Traps:

- **N nodes independently requesting the identical wildcard** are N duplicate requests — the
  5/week duplicate-cert limit locks the fleet out. Central issuance or distinct names only.
- **Caddy on-demand TLS is rejected for node names**: the first TLS handshake to a fresh name
  blocks for seconds on issuance, colliding with the sub-second webhook budget (Twilio's hard
  cap is 15 s, UX budget sub-second), and its rate-limit knobs are deprecated. Node names are
  known at provision time — pre-issue their certs.

## Troubleshooting

| Symptom | Fix |
|---|---|
| Twilio 31910 on Media Streams | Cert not publicly trusted / expired / SNI mismatch. Check `openssl s_client -connect pbx.example.com:443 -servername pbx.example.com`. |
| `rateLimited` / `too many certificates` in Caddy logs | You hit the duplicate-cert limit — almost always a lost `/data` volume. Restore the volume or wait out the window; then fix the volume mount so it never recurs. |
| DNS-01 fails for your provider | Provider not in the bundled plugin set — use a delegated zone on a supported provider, or build your own xcaddy image. |