Slice 3 — OpenAI Realtime brain: swap echo for the brain (#4)

docs/superpowers/specs/2026-06-30-slice-3-realtime-brain-design.md

@@ -0,0 +1,767 @@
+# Rutster slice 3 — Swap echo for OpenAI Realtime: the brain lands
+
+- **Status:** Draft (pending review)
+- **Date:** 2026-06-30
+- **Spearhead step:** 3 of 6 (vision-revision §10 / PORT_PLAN "Phasing")
+- **Origin:** brainstorming session 2026-06-30, resumed after the 2026-06-29
+  strategic pivot (ADR-0007 + ADR-0008).
+- **Depends on:** [slice 2 — the agent tap](2026-06-28-slice-2-agent-tap-design.md)
+  (must be implemented + green; the tap protocol and `TapAudioPipe` seam are
+  the foundation this slice swaps content into).
+- **Related:**
+  [ADR-0002](../../adr/0002-north-star-and-fused-core.md) (fused vertical),
+  [ADR-0004](../../adr/0004-license.md) (GPL-3.0-or-later),
+  [ADR-0008](../../adr/0008-fob-and-green-zone.md) (the brain is **green-zone**;
+  the reflex loop is **FOB** — load-bearing for the S4 turn-ownership
+  decision below),
+  [ARCHITECTURE.md §"Agent tap"](../../ARCHITECTURE.md).
+
+---
+
+## TL;DR
+
+Stand up spearhead step 3: swap slice-2's echo brain for a real
+**speech-to-speech brain — OpenAI Realtime** — reached through slice-2's
+existing tap. The core dials out (core-as-client; brain-as-server; still
+**no inbound tap port on the core**), slice-2's tap protocol carries
+audio + new event types (additive, v1, forwards-compatible), and the
+brain process translates between our protocol and OpenAI Realtime's event
+taxonomy.
+
+Slice 3 proves **agent integration**: the same WSS plumbing that today
+echoes will, in step 4, carry VAD/barge-in signals back from the brain to
+the core's reflex loop (the FOB). It deliberately defers real barge-in /
+VAD-driven playout kill (step 4), the PSTN trunk (step 5), and spend
+control (step 6) — but it *pre-paves* the `speech_started` /
+`speech_stopped` event seam so step 4 lands cleanly.
+
+The seam slice-2 pre-paved (`AudioSource` / `AudioSink` traits + the
+`TapAudioPipe` shape + the `TapEngine` task) is the **test of this
+slice**: `RtcSession`'s media-loop path changes by zero lines; the
+TapEngine's spawn / reconnect / teardown logic is untouched; only the
+*tap protocol module* (`rutster-tap/src/protocol.rs`) grows new event
+types and a new tool-registry module (`crates/rutster/src/tool_registry.rs`)
+lands in the binary.
+
+---
+
+## 1. Scope
+
+### 1.1 In scope
+
+- Implementation of spearhead step 3: WebRTC peer → core terminates
+  DTLS-SRTP, decodes Opus → canonical PCM @ 24 kHz mono, ships PCM
+  **over WSS** to an external OpenAI Realtime brain process, receives PCM
+  back, encodes + plays out via str0m. The user speaks and hears the AI
+  reply through an end-to-end speech-to-speech loop within ~700 ms
+  (slice-1's 200 ms + tap round-trip + OpenAI latency + 100 ms playout
+  buffer headroom).
+- **`crates/rutster-brain-realtime`** — a new workspace member; library +
+  binary. Dual purpose like slice-2's `rutster-tap-echo`: a standalone
+  dev-loop binary (`cargo run -p rutster-brain-realtime`) and an in-process
+  `MockRealtimeBrain` for integration tests (no network calls to OpenAI).
+  Default port `ws://127.0.0.1:8082/realtime` (the slice-2 echo brain
+  defaults to `:8081/echo`; the two coexist).
+- **Tap protocol extension (additive, v1, forwards-compatible):**
+  - `speech_started`, `speech_stopped` (brain → core, advisory).
+  - `function_call` (brain → core: tool name + args).
+  - `function_call_output` (core → brain: status + result).
+  - `tools.update` (brain → core: tool catalog so the core can validate).
+  Old echo brains ignore the new types per slice-2's "unknown type → log +
+  count + drop" rule (§3.4 of the slice-2 spec).
+- **In-boundary tool registry** in the brown binary
+  (`crates/rutster/src/tool_registry.rs`). `hangup` is the only wired tool
+  — fires the existing `Channel: Connected → Closing` path. Other tool
+  names reply `status: "not_implemented"`. The brain's `tools.update`
+  event declares the catalog; the registry validates function_call events
+  against it before dispatch.
+- **OpenAI Realtime translation layer** in `rutster-brain-realtime`: tap
+  events ↔ OpenAI Realtime events. Audio is 24 kHz mono PCM inside base64
+  LE i16 — **matching slice-1's canonical tap format exactly**, no resample.
+- **S4 turn-ownership decision (load-bearing per ADR-0008):** OpenAI
+  Realtime's `session.update` is sent with `turn_detection: null`
+  (disabled). The core drives turn-taking through its own (FOB) reflex
+  loop in step 4; OpenAI is treated as a speech-to-speech transducer.
+  The `speech_started` / `speech_stopped` events are caught and forwarded
+  through the tap protocol as advisory signals so step 4 can use them,
+  but OpenAI does **not** auto-bar the brain's `audio_out` — the
+  core-authoritative playout buffer (slice 2 §4.1) is the only thing
+  that gates playout.
+- **Two-source config** for the brain process: `OPENAI_API_KEY` env
+  default + `OPENAI_API_KEY_FILE` path override. `OPENAI_REALTIME_MODEL`
+  env (default: `gpt-4o-realtime` or current equivalent; documented).
+- **`--features=mock` dev mode** on the brain binary: runs the brain
+  process with an in-process mocked Realtime (no API key, no network
+  calls to OpenAI) — for offline dev loop + integration tests.
+- New workspace deps: `tokio-tungstenite` (already pulled by slice-2),
+  `reqwest` or `serde_json` (already pulled), and **no new** workspace
+  member-deps beyond what slice-2 already pinned. Reuse slice-2's
+  protocol types from `rutster-tap`.
+
+### 1.2 Out of scope (with scheduled return)
+
+| Deferred item | Returns in | Why deferred |
+|---|---|---|
+| Real barge-in / VAD-driven playout kill | Step 4 | Slice-3 pre-paves the `speech_started` / `speech_stopped` advisory event seam; step 4 wires the FOB reflex loop to act on them. The core-authoritative playout buffer from slice-2 §4.1 is already in place. |
+| PSTN trunk / rented-transport integration | Step 5 | ADR-0007's rented CPaaS raw-media fork lands the phone number; the brain is unaffected. |
+| Spend cap / abuse gate | Step 6 | The brain has no spend surface yet — OpenAI bills the operator directly. In-boundary spend pacing lands with `rutster-spend` in step 6. |
+| Multi-brain routing / per-tenant brain selection | Step 6 | One tap URL per call in slice-3 (env + per-call override). The slice-3 brain process is OpenAI-Realtime-only. A Deepgram+LLM+TTS composite adapter or a self-hosted open-weights brain is a future-rung concern; the tap protocol is brain-agnostic by design. |
+| API-key rotation / KMS integration | Step 6 + later | `OPENAI_API_KEY` (env) + `OPENAI_API_KEY_FILE` (path) is the dev posture. KMS / Vault integration lands with the real trust boundary (step 6). |
+| TLS on the tap and on the OpenAI leg | Step 6 | Slice-2 rejects `wss://` URLs at session-create (deferred to step 6); the OpenAI Realtime leg is `wss://` and uses rustls-native roots (no cert pinning in slice-3 — defer to step 6). |
+| Authentication / authorization on the `tap_url` override | Step 6 | Inherits slice-2's "no auth yet" posture. |
+| Re-INVITE / session migration / resumability | Later | Refresh the page → new session, same as slice-1/2. OpenAI Realtime session is per-call; no in-flight call preservation across server restart. |
+| CDR / event bus / OTel beyond per-Channel `tracing` spans | Step 5 | Single peer + single brain; no fanout yet. Tool-call events go to logs + counters only. |
+| Quality dashboard (containment, escalation reasons) | Capability ladder rung 3 | The current build target proves agent integration, not analytics. |
+| Audio resampling for brains using 16 kHz | Future | OpenAI Realtime uses 24 kHz mono, matching slice-1's canonical tap format. Will be needed if a future brain uses 16 kHz; the translator resamples at that time. |
+| `response.audio.delta` batching optimization | Future | OpenAI sends many small delta events; the translator MAY batch into the slice-2 `audio_out` 20 ms frame. Optimization optional for v1; tracked. |
+
+### 1.3 What this slice does NOT prove
+
+It does **not** prove: a real barge-in reflex (only the event seam),
+latency determinism under reflex timing, PSTN trunking, spending controls,
+multi-tenancy on the tap URL, API-key rotation, or wss:// TLS posture in
+production. It proves **only** agent integration: the OpenAI Realtime
+adapter as green-zone (per ADR-0008), the tap protocol extension carrying
+audio + interruption signals + function-call plumbing, the in-boundary
+tool registry dispatching `hangup` cleanly, and the seam test (slice-2's
+`RtcSession` accepts the new brain with zero internal change).
+
+---
+
+## 2. Workspace layout (delta on slice-2)
+
+One new workspace member; one new module in the binary; additive
+extension to `rutster-tap`'s protocol module.
+
+```
+rutster/
+├── Cargo.toml                          # +rutster-brain-realtime in members
+├── crates/
+│   ├── rutster/                         # binary: +tool_registry module
+│   │   ├── src/main.rs
+│   │   ├── src/session_map.rs          # +tool-call side-channel drain
+│   │   ├── src/routes.rs               # unchanged shape
+│   │   ├── src/tap_engine.rs           # UNCHANGED — the seam test
+│   │   ├── src/tool_registry.rs        # NEW: Tool trait, hangup tool
+│   │   └── static/index.html           # minor: surface brain connection status
+│   ├── rutster-media/                   # UNCHANGED — the seam test
+│   │   └── src/loop_driver.rs          # UNCHANGED
+│   ├── rutster-call-model/              # UNCHANGED
+│   ├── rutster-tap/                     # +additive protocol event types
+│   │   └── src/protocol.rs             # +speech_started/stopped, function_call/output, tools.update
+│   ├── rutster-tap-echo/                # UNCHANGED (still works against extended protocol)
+│   ├── rutster-brain-realtime/          # NEW crate
+│   │   ├── Cargo.toml                  # deps: rutster-tap, tokio, tokio-tungstenite, serde_json, tracing, url
+│   │   ├── src/lib.rs                  # lib + MockRealtimeBrain for tests
+│   │   ├── src/main.rs                 # standalone binary: ws://127.0.0.1:8082/realtime + OpenAI WS client
+│   │   ├── src/translator.rs           # tap ⇄ OpenAI event translation
+│   │   └── src/openai_client.rs         # wss://api.openai.com/v1/realtime client
+│   ├── rutster-trunk/                   # STUB (unchanged)
+│   └── rutster-spend/                   # STUB (unchanged)
+└── examples/
+    └── echo_brain/                      # unchanged (Python reference; ignores new events)
+```
+
+### 2.1 Dependency direction
+
+- `rutster-brain-realtime` → `rutster-tap` (for protocol types — same
+  re-export pattern slice-2 established for `rutster-tap-echo`).
+- `rutster-brain-realtime` is its own workspace member that's both a
+  binary (dev loop) and a library (test fixture). The library re-exports
+  a `MockRealtimeBrain` for use by integration tests in the binary crate.
+- `rutster` (binary) gains `tool_registry.rs` as a sibling of
+  `tap_engine.rs` — the new module dispatches tool-call events the
+  TapClient observes via the new side-channel mpsc.
+- `rutster-tap`'s protocol module is the contract both the brown core
+  and the new brain process share; new event types added here are re-used
+  by `rutster-brain-realtime` (the contract test that the wire types are
+  reusable from outside the core, exactly as `rutster-tap-echo` did).
+- `rutster-media` and `rutster-call-model` are untouched. `loop_driver.rs`
+  and `rtc_session.rs` are byte-identical to slice-2 (post-review-fix)
+  baseline.
+
+### 2.2 Why one new crate for the OpenAI Realtime brain (not an `examples/` file)
+
+Mirrors slice-2's dual-purpose pattern for `rutster-tap-echo`: a real
+workspace member that runs `cargo fmt`, `cargo clippy -D warnings`,
+`cargo test`, and `cargo deny check` like every other crate. It reuses
+`rutster-tap`'s protocol types — the contract test that the wire types
+are reusable from **outside the core**, which is what makes the tap a
+real extension point and not a closed system.
+
+A Python reference brain (`examples/openai_realtime_brain/openai_realtime_brain.py`)
+is the canonical foreign-language brain demo for **OpenAI Realtime
+specifically** (slice-2 already shipped the echo brain in Python). This
+file is **not in CI** (Python would violate the zero-non-Rust-dev-deps
+dev loop). README-documented runnable: `pip install websockets openai`
+and run.
+
+### 2.3 Why the brain is green-zone (per ADR-0008)
+
+ADR-0008 classifies "the agent brain" explicitly under green-zone: it's
+not hot-path inside the boundary (its round-trip is hundreds of ms; the
+timing-critical work happens in the FOB reflex loop), not
+security-constitutive (the core-authoritative playout buffer structurally
+prevents the brain from flooding the wire), and not differentiating (the
+tap-as-open-protocol is the differentiator; any brain speaking it works).
+So the OpenAI Realtime adapter lives **outside** the FOB trust boundary —
+its own process, its own API key, its own failure domain. The brain is
+**reached by** the FOB; it does not live in it. The only FOB-side
+additions in slice-3 are:
+
+- `rutster-tap/src/protocol.rs` — additive event types (protocol
+  extension, not new behavior — the existing TapClient + TapAudioPipe
+  surface is unchanged).
+- `crates/rutster/src/tool_registry.rs` — in-boundary tool dispatch (a
+  security-constitutive capability: the brain proposes tool calls; the
+  FOB disposes. Per ADR-0007's spend-gate posture: "rutster mediates both
+  the provider call-control API and the brain tap, so the brain never
+  holds the wire").
+
+---
+
+## 3. Tap wire protocol extension (`rutster-tap/src/protocol.rs`)
+
+The slice-2 v1 protocol (envelope: `{v, type, seq, ts}`). The new event
+types are additive — slice-2's §3.4 "unknown type → log + count + drop"
+rule means an old brain (slice-2's `rutster-tap-echo`) ignores them.
+Slice-2's envelope invariants (per-direction `seq`, advisory `ts`,
+`samples: 480` for audio frames, explicit LE byte order) carry over
+unchanged.
+
+### 3.1 Envelope (slice-2's, unchanged)
+
+```jsonc
+{
+  "v": 1,
+  "type": "<event_name>",
+  "seq": <uint>,
+  "ts": <uint>
+}
+```
+
+### 3.2 New events — brain → core (advisory)
+
+| `type` | payload fields | when |
+|---|---|---|
+| `speech_started` | `{}` | brain detected user speech started (translated from OpenAI `input_audio_buffer.speech_started`; advisory — the core’s FOB reflex loop, when wired in step 4, may use it for barge-in) |
+| `speech_stopped` | `{}` | brain detected user speech stopped (translated from OpenAI `input_audio_buffer.speech_stopped`; advisory) |
+| `tools.update` | `{ "tools": [{ "name": "<slug>", "description": "...", "parameters": {...} }] }` | brain declares its tool catalog (sent on `hello` ack, re-sent on tool catalog changes). The core's tool registry uses this to validate function_call events. |
+| `function_call` | `{ "id": "<uuid>", "name": "<slug>", "args": { ... } }` | brain wants the core to execute a tool (translated from OpenAI `response.function_call_arguments.done`). The core dispatches via the tool registry and replies via `function_call_output`. |
+
+### 3.3 New events — core → brain
+
+| `type` | payload fields | when |
+|---|---|---|
+| `function_call_output` | `{ "id": "<uuid>", "status": "ok"\|"error"\|"not_implemented", "result": { ... } }` | core's tool-registry reply. Translated by the brain process into an OpenAI `conversation.item.create` with `type: function_call_output` so OpenAI continues the conversation appropriately. |
+
+### 3.4 Invariants
+
+- **Tool-call id:** `function_call` and `function_call_output` carry the
+  same `id` (a UUID minted by the brain; OpenAI calls it `call_id`, we
+  translate verbatim). Mismatches are logged + counted; the tool-registry
+  dispatch keys off the id.
+- **Tool validation:** the core's tool registry validates `function_call`
+  events against the most recent `tools.update` catalog. An unknown tool
+  name → `function_call_output` with `status: "not_implemented"` (not
+  `error`; distinguishing "the FOB doesn't know this tool" from "the tool
+  failed"). The catalog is brain-authoritative — the brain declares it;
+  the core merely checks dispatch.
+- **Advisory interruption events:** `speech_started` / `speech_stopped`
+  are **advisory only** in slice-3. The core logs + counts them; the FOB
+  reflex loop lands in step 4 and will use them. No slice-3 code path
+  acts on them in the hot media loop — `loop_driver.rs` is byte-identical
+  to slice-2 (post-review-fix) baseline.
+- **Forward-compat:** unknown `type` values continue to be logged +
+  counted + dropped (slice-2 §3.4). The echo brain from slice-2 ignores
+  every new type this slice introduces; it echoes audio unchanged.
+
+### 3.5 Byte-order + audio format (unchanged from slice-2)
+
+PCM inside the base64 payload is explicit little-endian `i16` bytes (v1
+wire contract, slice-2 §3.4). OpenAI Realtime's audio events also use
+24 kHz mono PCM inside base64 LE i16 — **no transcoding, no resample,
+no endianness swap.** The translator passes the audio payload through
+to the WS frame verbatim. This is a happy convergence that simplifies
+the translator; it is not a coincidence (slice-1's choice of 24 kHz mono
+was made with this ecosystem in mind).
+
+---
+
+## 4. The OpenAI Realtime translation layer (`rutster-brain-realtime/src/translator.rs`)
+
+Pure (no schema of its own beyond the new tap event types); maps event
+names + payload shapes between the two wire protocols. Owns no call
+state beyond the OpenAI Realtime session/connection state.
+
+### 4.1 The two-leg brain process
+
+```
+                WS server side                        WS client side
+        (core-as-client dials here)         (brain-as-client dials OpenAI)
+                  ┌──────────────────┐                              ┌──────────┐
+   tap protocol   │  rutster-brain-  │     translation layer         │  OpenAI  │
+   core ◄────────┤  realtime         ├────────────────────────────►  │ Realtime │
+  (audio_in/out,  │                  │   tap event ⇄ OpenAI event   │  API WS  │
+  speech_started, │                  │                              │ (wss://api.openai.com)│
+  func_call, ...) └──────────────────┘                              └──────────┘
+                        │
+                        │ brain process also owns:
+                        │  • OpenAI session.update (turn_detection: null — S4)
+                        │  • interruption-signal forwarding (OpenAI → tap)
+                        │  • function_call forwarding (bidirectional)
+                        │  • tools.update on startup (catalog to core)
+```
+
+The translation layer is pure (no schema of its own beyond the new tap
+event types); it maps event names + payload shapes between the two wire
+protocols. It owns no call state beyond OpenAI's session/connection state.
+
+### 4.2 Event mapping table
+
+| Tap protocol | OpenAI Realtime | Notes |
+|---|---|---|
+| `hello` | `session.update` (modalities: ["audio", "text"], audio config, **turn_detection: null**) | Sent on handshake + reconnect; the S4 decision is encoded here |
+| `audio_in` (base64 PCM LE i16) | `input_audio_buffer.append` (base64, same wire shape) | Pass-through; no transcoding |
+| `audio_out` (from brain → core, advisory) | `response.audio.delta` events | The brain receives, formats, sends to core via tap_audio_out mpsc |
+| `speech_started` (brain → core advisory) | `input_audio_buffer.speech_started` | Caught and forwarded as advisory |
+| `speech_stopped` (brain → core advisory) | `input_audio_buffer.speech_stopped` | Caught and forwarded as advisory |
+| `function_call` (brain → core) | `response.function_call_arguments.done` (OpenAI emits) → translator formats as tap function_call | Brain's translator extracts `call_id`, `name`, `arguments`; the core dispatches via tool registry |
+| `function_call_output` (core → brain) | `conversation.item.create` with `item.type: function_call_output`, `call_id: <id>`, `output: <result-json>` | Closes the tool-call loop |
+| `bye` / `session_end` | `session.delete` + WS close | Graceful teardown |
+
+### 4.3 The S4 turn-ownership decision (load-bearing per ADR-0008)
+
+> ARCHITECTURE.md is right that the core should be authoritative on
+> VAD/barge-in ("the tap carries the *results* of reflexes, not the
+> responsibility"; "`AudioOut` advisory / core-authoritative"). But the
+> most likely first brain — OpenAI Realtime — does its own server-side
+> VAD and turn detection by default. Integrating step 3–4 means either:
+> disable the brain's turn detection and feed it clean, locally-detected
+> turns (preserves the reflex-authoritative principle, but is more
+> integration work and fights the API's grain), or
+> accept split-brain turn-taking where local VAD and the brain's VAD can
+> disagree (double-triggers, dropped barge-ins).
+>
+> **Decision implied:** make "who owns turn detection" an explicit decision
+> in the step-3 (brain) design doc, defaulting to core-authoritative,
+> brain VAD off, with the integration cost budgeted. Don't let it be
+> discovered at wiring time.
+> — Claude, vision-sanity-check S4
+
+ADR-0008's FOB/green-zone split makes this decidable without ambiguity:
+
+- **The reflex loop is FOB** (hot-path + differentiating — turn-taking,
+  VAD-driven barge-in, jitter, pacing).
+- **The brain is green-zone** (per ADR-0008's explicit classification).
+- **The tap is core-authoritative** (slice-2 §4.1 — the playout buffer
+  structurally prevents the brain from gating playout).
+
+The only doctrine-consistent posture is: **OpenAI Realtime's server-side
+turn detection is disabled (`session.update` with `turn_detection: null`),
+and the FOB owns turn-taking.** The `speech_started` /
+`speech_stopped` events are forwarded as advisory signals so step 4 can
+use them as one input to the FOB reflex loop — but OpenAI does not gate
+playout, ever. The core-authoritative playout buffer (slice-2 §4.1) is
+the only thing that gates playout in slice-3 (and continues to be in
+step 4).
+
+**Why this is the right call now (rather than deferring to step 4):**
+leaving OpenAI's turn detection on would mean slice-3's demo
+"works by accident" — the slice-4 barge-in integration would then have to
+disable it mid-flight, with unpredictable behavioral changes. By nailing
+it down at slice-3's `session.update`, the brain process never wires up
+the wrong posture and step 4's barge-in work composes cleanly on top.
+
+### 4.4 Failure mode + reconnect
+
+- **Tap-side WS reconnect (brain unreachable):** slice-2's
+  `TapEngine::spawn_tap_engine` already does bounded-backoff reconnect
+  (`250 ms → 500 ms → 1 s → 2 s → cap 5 s`, infinite retries,
+  re-`hello`s with the same `session_id`). The brain process being
+  unreachable looks identical to slice-2's echo brain being unreachable
+  — the slice-2 reconnect path is unchanged. The seam test (slice-2
+  §8.5 #6) holds: `tap_engine.rs` is byte-identical.
+- **OpenAI-side WS failure:** the brain process's own concern — it enters
+  its own bounded-backoff reconnect to OpenAI, and forwards a tap `error`
+  event (`{ "code": "brain_upstream_disconnect", "message": "..." }`)
+  to the core as an FYI. The core logs + counts; the call stays up; the
+  tap-side playout goes silent (slice-2's underflow path) until OpenAI
+  reconnects. The slice-3 brain's OpenAI-side reconnect is independent
+  of the slice-2 tap-side reconnect — two distinct failure surfaces.
+- **API-key invalid (OpenAI returns 401):** the brain process emits a tap
+  `error` event (`{ "code": "brain_auth_failed", "message": "..." }`)
+  and exits the OpenAI leg. The core's response is unchanged from the
+  upstream-disconnect case (logs + silence + reconnect path). Operator
+  intervention; the call stays up.
+- **Tool-registry dispatch failure:** a tool-registry error (e.g.,
+  `hangup` fires but the channel state machine rejects the transition
+  because the call is already `Closing`) returns a `function_call_output`
+  with `status: "error"` and a result body explaining. The brain forwards
+  the error to OpenAI; OpenAI may or may not retry depending on its own
+  logic. No core-side retry; the model gets one chance.
+
+---
+
+## 5. Lifecycle & integration
+
+### 5.1 Session lifecycle (slice-3 delta on slice-2)
+
+1. `POST /v1/sessions` — body still *optionally* carries `tap_url`. If
+   absent, falls back to `RUTSTER_TAP_URL` env. Default now documented
+   as either `ws://127.0.0.1:8081/echo` (slice-2 Rust echo brain) or
+   `ws://127.0.0.1:8082/realtime` (slice-3 OpenAI Realtime brain); the
+   operator picks which brain to run.
+2. `POST /v1/sessions/:id/offer` — unchanged from slice-1/2; SDP answer.
+3. On `Connected`: the binary's poll task observes the state transition
+   + spawns the `TapEngine` task (unchanged from slice-2). The
+   `TapEngine` connects to the brain process's WS server.
+4. Brain process startup: connects to `wss://api.openai.com/v1/realtime`,
+   sends `session.update` with `turn_detection: null`, waits for OpenAI
+   `session.created` ack.
+5. Brain process WS server accepts the core's tap WS; `hello` exchange;
+   brain process sends `tools.update` with the catalog (currently:
+   `hangup` only, plus any tool schemas the brain process's startup
+   config declares — see §6).
+6. Audio flows: core decodes Opus → PCM → `audio_in` to brain → translator
+   formats as `input_audio_buffer.append` → OpenAI processes →
+   `response.audio.delta` → translator formats as `audio_out` → core's
+   playout ring → str0m encode.
+7. Interruption signals: OpenAI `input_audio_buffer.speech_started` /
+   `.speech_stopped` → translator → tap `speech_started` / `speech_stopped`
+   → core logs + counts (advisory). Step 4 will wire these into the FOB
+   reflex loop.
+8. Tool calls: OpenAI `response.function_call_arguments.done` → translator
+   → tap `function_call` → core's `tool_registry` dispatches (via the
+   TapClient's new side-channel mpsc) → `function_call_output` reply →
+   translator → OpenAI `conversation.item.create`.
+9. `DELETE /v1/sessions/:id` or peer-close → `Closing`: slice-2's
+   unmodified teardown sequence fires — the TapEngine's `session_end` →
+   brain process's `session.delete` to OpenAI → WS close on both legs.
+
+### 5.2 The tool-call side-channel (the only brown-binary wiring)
+
+Slice-2's `TapClient` runs the WSS pump loop with two mpsc channels
+(`tx_pcm_in` for inbound audio to brain, `rx_audio_out` for outbound
+audio from brain). Slice-3 adds a third:
+
+- `tx_function_call: mpsc::Sender<FunctionCallEvent>` — the TapClient
+  emits a `FunctionCallEvent` whenever it observes a tap `function_call`
+  message on its inbound stream. The binary's poll task drains this in
+  the same cycle it drains the existing `flush_tx` side-channel
+  (slice-2 §5.3 step 4) — same pattern, one extra channel.
+- `rx_function_call_output: mpsc::Receiver<FunctionCallOutputEvent>` —
+  the binary writes `function_call_output` events through this channel;
+  the TapClient picks them up and sends them as tap WS frames.
+
+The shape keeps the seam test (§7 #5 of done criteria below): all this
+lives in the binary's `tap_engine.rs` callers + the new `tool_registry.rs`
+module. `loop_driver.rs` and `rtc_session.rs` are untouched.
+
+### 5.3 Brain process startup configuration
+
+| Env var | Purpose | Default |
+|---|---|---|
+| `RUTSTER_TAP_BIND` | WS server bind addr for the brain process | `127.0.0.1:8082` |
+| `OPENAI_API_KEY` | OpenAI Realtime API key (mutually exclusive with `_FILE`) | required unless `--features=mock` |
+| `OPENAI_API_KEY_FILE` | Path to a file containing the API key | optional; overrides `_KEY` |
+| `OPENAI_REALTIME_MODEL` | OpenAI Realtime model id | `gpt-4o-realtime` (or current equivalent) |
+| `OPENAI_REALTIME_VOICE` | Voice for TTS output | `alloy` (OpenAI's default; documented) |
+| `RUTSTER_BRAIN_TOOLS` | Comma-separated tool names the brain should advertise in `tools.update` | `hangup` (only tool the core wires; others cause `not_implemented` replies) |
+
+The dev mode (`--features=mock`) doesn't read `OPENAI_API_KEY` and
+instruments an in-process mock OpenAI Realtime WS server that generates
+canned `response.audio.delta` events on `input_audio_buffer.append` and
+asserts on the `session.update` having `turn_detection: null`. Used by:
+the slice-3 integration test (no real OpenAI credentials, no network
+calls), and the offline dev loop.
+
+---
+
+## 6. The tool registry (`crates/rutster/src/tool_registry.rs`)
+
+A FOB-internal dispatch over the brain's proposed tool calls. The brain
+proposes (via `function_call`); the FOB disposes (via `function_call_output`).
+Per ADR-0007's spend-gate posture: the brain never holds the wire,
+structurally.
+
+### 6.1 Trait + registry shape
+
+```rust
+#[async_trait]
+pub trait Tool: Send + Sync {
+    /// Tool name as the brain will reference it in function_call events.
+    fn name(&self) -> &str;
+    /// JSON-schema description the registry sends to the brain on tools.update.
+    fn schema(&self) -> serde_json::Value;
+    /// Execute the tool; return a result that the registry will serialize
+    /// into the function_call_output payload.
+    async fn call(&self, args: serde_json::Value) -> ToolResult;
+}
+
+pub enum ToolResult {
+    Ok(serde_json::Value),
+    Error(String),
+    NotImplemented,
+}
+
+pub struct ToolRegistry {
+    tools: Vec<Box<dyn Tool>>,
+}
+
+impl ToolRegistry {
+    pub fn new() -> Self;
+    pub fn register(&mut self, tool: Box<dyn Tool>);
+    /// Dispatch by tool name; returns ToolResult::NotImplemented if unknown.
+    pub async fn dispatch(&self, name: &str, args: serde_json::Value) -> ToolResult;
+    /// Used on startup + tools.update: enumerate the registered catalog.
+    pub fn catalog(&self) -> Vec<serde_json::Value>;
+}
+```
+
+### 6.2 The `hangup` tool (the only wired impl in slice-3)
+
+A `Tool` impl that on `call`:
+1. Looks up the `ChannelId` (passed via the dispatch context — the
+   `tool_registry` is keyed by `ChannelId`, one registry per active
+   channel).
+2. Fires `AppState::close(channel_id).await` — the existing slice-2
+   teardown path (sets `Closing`, sends `session_end`, aborts the engine).
+3. Returns `ToolResult::Ok({"channel_state": "Closing"})` so the brain
+   gets confirmation.
+
+Other tool names (anything in `RUTSTER_BRAIN_TOOLS` the brain advertises
+that isn't `hangup`) → the registry returns `ToolResult::NotImplemented`:
+the brain process forwards the reply as an OpenAI
+`conversation.item.create` with the not_implemented status, and the model
+is free to continue.
+
+### 6.3 What slice-3 does NOT wire
+
+- No `transfer` tool (escalation rung 2 — separate slice).
+- No `lookup` / CRM-integration tools (business-logic integrations are
+  green-zone + future-rung).
+- No spend/abuse-triggered tools (step 6).
+- No dynamic tool registration at runtime (the catalog is fixed at
+  startup from `RUTSTER_BRAIN_TOOLS`).
+
+If a future brain proposes a tool not in the registry: `not_implemented`
+reply; the model is free to retry with different arguments or give up.
+The registry's job is to be the auditable dispatch boundary, not to be
+extensible mid-call.
+
+---
+
+## 7. CI, dev loop, testing
+
+### 7.1 New `[workspace.dependencies]` (Cargo.toml)
+
+- `async-trait = "0.1"` (for the `Tool` trait's async fn).
+- `tokio-tungstenite`, `serde_json`, `tracing`, `url` — already pulled
+  by slice-2; reused.
+
+Member crate `rutster-brain-realtime` references these with
+`dep.workspace = true`.
+
+### 7.2 CI (`.github/workflows/ci.yml`)
+
+Unchanged structure: `cargo fmt --check`, `cargo clippy -D warnings`,
+`cargo test --all`, `cargo deny check`. The new `rutster-brain-realtime`
+crate joins `--all`. **No Python in CI** — the OpenAI Realtime Python
+reference brain is README-documented only. The Rust
+`rutster-brain-realtime`'s in-process `MockRealtimeBrain` powers the
+integration test; no real OpenAI credentials needed.
+
+### 7.3 Dev loop
+
+- `cargo run -p rutster-brain-realtime --features=mock` → starts the
+  brain process with an in-process mock OpenAI Realtime (on `:8082`):
+  no API key required, no network calls to OpenAI.
+- `cargo run -p rutster-brain-realtime` (with `OPENAI_API_KEY` set) →
+  starts the brain process with the real OpenAI Realtime.
+- `cargo run` (or `cargo run -p rutster`) → starts the axum signaling
+  server on `0.0.0.0:8080`, dials out to `$RUTSTER_TAP_URL` on each
+  session.
+- Browser → `http://localhost:8080/` → click "Start call" → grant mic →
+  speak → hear AI reply through the brain process.
+- `RUST_LOG=rutster=debug cargo run` for verbose tracing including
+  tap + tool-call events.
+- `--features=echo` on the binary (slice-2 inherited) → bypasses the
+  brain entirely, routes audio through `EchoAudioPipe`.
+
+### 7.4 Testing strategy
+
+- **Unit tests in `rutster-brain-realtime`:**
+  - Protocol-event translation round-trips (tap `audio_in` ↔ OpenAI
+    `input_audio_buffer.append`; tap `function_call` ↔ OpenAI
+    `response.function_call_arguments.done`; tap `function_call_output`
+    ↔ OpenAI `conversation.item.create`).
+  - `session.update` body asserts `turn_detection: null` (S4 decision
+    encoded as a test).
+  - `tools.update` serialization round-trips.
+  - Error path: OpenAI-side WS error → tap `error` event forwarding.
+- **Unit tests in `rutster-tap`:**
+  - New event types (de)serialization round-trips with golden JSON
+    fixtures in `tests/fixtures/`.
+  - Slice-2's existing echo-brain tests stay green (the new event types
+    are ignored by the echo brain — forwards-compat).
+- **Unit tests in `rutster` `tool_registry.rs`:**
+  - `hangup` tool fires `AppState::close` correctly.
+  - Unknown tool name → `ToolResult::NotImplemented`.
+  - Registry catalog serialization.
+- **Integration test in `rutster` binary crate:** spin up the axum server
+  (ephemeral port) + the in-process `MockRealtimeBrain` (ephemeral port) +
+  set `RUTSTER_TAP_URL`. Drive a synthetic WebRTC peer (extending
+  slice-2's `tap_integration.rs` harness): push PCM into the core via
+  the WebRTC peer → assert `audio_out` flows back through the tap (the
+  mock brain generates canned `response.audio.delta` on
+  `input_audio_buffer.append`) → assert they're re-encoded + pushed to
+  str0m. Plus: function-call round-trip (mock brain emits a
+  `function_call` for `hangup` → core's `tool_registry.dispatch("hangup")`
+  fires → channel state `Closing` → `session_end` over the tap →
+  mock brain's `session.delete` recorded).
+- **Manual e2e test plan (README):**
+  1. `cargo run -p rutster-brain-realtime --features=mock` (mock brain on `:8082`).
+  2. `cargo run` (core on `:8080`).
+  3. Browser → speak → hear mock-brain reply within ~250 ms (slice-2's
+     round-trip; no real OpenAI).
+  4. Repeat with the real brain (`cargo run -p rutster-brain-realtime`
+     with `OPENAI_API_KEY` set) → end-to-end speech-to-speech with real
+     OpenAI Realtime within ~700 ms.
+  5. Repeat with the Python brain (`python
+     examples/openai_realtime_brain/openai_realtime_brain.py`) → should
+     also work (proves the protocol is language-agnostic, same as
+     slice-2's Python echo brain).
+  6. Function-call: trigger a model behavior that emits a tool call
+     (e.g. a test prompt that says "please hang up") → assert the call
+     closes + `session_end` over the tap + `session.delete` to OpenAI.
+  7. `cargo test --all` green; `cargo fmt --check` /
+     `cargo clippy -D warnings` / `cargo deny check` green.
+
+### 7.5 Slice 3 "done" criteria
+
+The slice is complete when, on a clean checkout (+ open pivot / slice-1
+review fixes / slice-2 plan-rebaseline PRs merged):
+
+1. `cargo test --all` passes (unit + integration). The new
+   `rutster-brain-realtime` crate tests green alongside slice-1/2's suite.
+2. `cargo fmt --check`, `cargo clippy -D warnings`, `cargo deny check`
+   all pass.
+3. `cargo run -p rutster-brain-realtime --features=mock` + `cargo run` →
+   browser, speak, hear mock-brain reply within ~250 ms.
+4. With `OPENAI_API_KEY` set: real OpenAI Realtime end-to-end within
+   ~700 ms.
+5. **Both** `rutster-brain-realtime` (Rust, with `--features=mock`) **and**
+   `examples/openai_realtime_brain/openai_realtime_brain.py` (Python)
+   successfully interop against the core — proves the extended protocol
+   is language-agnostic.
+6. **The seam test (load-bearing):** `rutster-media`'s `loop_driver.rs`
+   and `rtc_session.rs` keep their media-loop trait-method call sites
+   **byte-identical** to slice-2 (post-review-fix) baseline. The only
+   brown-binary additions are: `crates/rutster/src/tool_registry.rs`
+   (new module) + a new side-channel drain in `session_map.rs`. The
+   `tap_engine.rs` and `TapClient` get only additive new mpsc channels
+   (one new sender + one new receiver); the WSS pump loop's structure
+   is unchanged. A `git diff v<slice-2-post-review-fix-tag> --
+   crates/rutster-media/src/loop_driver.rs
+   crates/rutster-media/src/rtc_session.rs` shows no behavior-changing
+   hunks (doc-comment or import changes permitted).
+7. **S4 turn-ownership test:** the integration test asserts that the
+   `session.update` sent to OpenAI Realtime (or its mock equivalent)
+   contains `turn_detection: null`. The brain never auto-barges; the
+   core-authoritative playout buffer is the only playout gate.
+8. `LEARNING.md` grows ≥3 new pointers: `async-trait` patterns →
+   `crates/rutster/src/tool_registry.rs`; OpenAI Realtime adapter →
+   `crates/rutster-brain-realtime/src/translator.rs`; tap protocol
+   extension + forwards-compat → `crates/rutster-tap/src/protocol.rs`.
+
+---
+
+## 8. Open decisions (tracked)
+
+- **`response.audio.delta` batching.** OpenAI sends many small delta
+  events per response; the translator MAY batch into the slice-2
+  `audio_out` 20 ms frame, or pass each delta through immediately as
+  its own `audio_out`. Batching reduces WS round-trips but adds latency.
+  Track + revisit after latency measurement with the real brain.
+- **Tool-registry extensibility.** Slice-3 fixes the catalog at startup
+  via `RUTSTER_BRAIN_TOOLS`. Runtime-extensible registration (new tools
+  added mid-call) is a future-rung concern — would need a tool-register
+  API event. Track.
+- **API-key rotation.** Slice-3 reads the key once at startup; a key
+  rotation requires a brain-process restart. KMS / Vault integration
+  lands with the real trust boundary (step 6). Track.
+- **Voice + persona selection.** `OPENAI_REALTIME_VOICE` is the only
+  persona knob now; future brain processes may carry full prompt /
+  system-message customization (rung 2+). Track.
+- **Multi-brain routing.** Slice-3 ships one brain process: OpenAI
+  Realtime. A Deepgram+LLM+TTS composite adapter or a self-hosted
+  open-weights brain would be its own crate
+  (`rutster-brain-deepgram`, `rutster-brain-local`) — the tap protocol
+  is brain-agnostic by design. Track + revisit when a second brain
+  lands.
+
+---
+
+## 9. Out-of-scope re-check (against AGENTS.md + ADR-0007/0008)
+
+| Item | Status |
+|---|---|
+| Dedicated timing thread for media loop | Still step 4. |
+| TLS on the HTTP signaling surface | Still step 5. |
+| Authn / authz / multi-tenancy on `/v1/sessions` | Still step 6. |
+| Trickle ICE | Unchanged. |
+| Barge-in / VAD-driven playout kill | **Step 4.** Slice-3 pre-paves the `speech_started` / `speech_stopped` advisory event seam so step 4 lands cleanly. |
+| PSTN trunk / rented transport | Still step 5. |
+| Spend cap / abuse gate | Still step 6. |
+| CDR / event bus / OTel beyond per-Channel tracing | Still step 5. |
+| Browser automation / Playwright | Still post-slice-1. |
+| Docker / compose | Still later-rung. |
+| Transfer / park / pickup / barge (call features) | Still escalation rung 2. |
+
+If an agent proposes adding any of these in slice 3, the right answer is
+"no, see the slice-3 spec §1.2."
+
+---
+
+## 10. Key design decisions (summary of the brainstorming session)
+
+| Decision | Choice | Rejected alternatives | Why |
+|---|---|---|---|
+| Brain target | **OpenAI Realtime NOW; design for multi-brain later.** Adapter trait shape is the tap protocol itself, not a Rust trait. | Trait crate `rutster-brain` + impl crate; Deepgram+LLM+TTS now | Slice-2's protocol-as-contract framing already delivers "multi-brain later" without a speculative trait; YAGNI on abstraction. A Deepgram+LLM+TTS composite brain's friction will be in its own internal topology (composing three async WS/HTTP streams), not in `trait BrainAdapter` ergonomics — a trait wouldn't ease that anyway. |
+| Cool scope | **Round-trip speech + interruption signals (advisory) + function-calling plumbing** | Speech only; speech + interruption; speech + interruption + function calling | Function calling is the gateway to escalation (rung 2 — `hangup` is the only wired tool; `transfer` lands later). Plumbing the FOB dispatch contract in slice-3 means step-4 barge-in composes on a clean foundation; deferring it would force step-4 to retrofit the seam. |
+| Function-call handler | **Built-in tool registry in the core (FOB)** | Outbound tool-server URL (second egress); adapter handles function calls itself; plumbing only | Per ADR-0007: "rutster mediates both the provider call-control API and the brain tap, so the brain never holds the wire." The FOB disposes; the brain proposes. `hangup` is the only wired tool; others reply `not_implemented` honestly. The registry's job is the auditable dispatch boundary, not extensibility. |
+| Turn-detection ownership (S4) | **Core-authoritative, brain VAD off** (`session.update` with `turn_detection: null`) | Accept split-brain (OpenAI VAD on; core-authoritative too → disagreement); defer the decision to step 4 | ADR-0008 makes the FOB/green-zone split mechanical: reflex loop is FOB, brain is green-zone, playout is core-authoritative. The only doctrine-consistent choice is to disable OpenAI's server-side VAD and let the FOB own turn-taking. Nailing it down at slice-3's `session.update` means step-4 barge-in composes cleanly. |
+| API-key posture | **Env var + file-path override.** `OPENAI_API_KEY` env default; `OPENAI_API_KEY_FILE` overrides. | Env-only; KMS-required-now | Matches slice-2's "no auth yet" stance cleanly. KMS / Vault lands with the real trust boundary (step 6). The file-path override makes secret-manager injection (k8s secrets, Vault agent) trivial when that layer exists. |
+| Dev mode | **`--features=mock` in-process mocked OpenAI Realtime** | No dev mock (always require real OpenAI); external mock server | The slice-3 dev loop must be zero-OpenAI-credentials + zero-network-calls-to-OpenAI. The integration test uses the same mock. Same pattern as slice-2's `EchoServer`. |
+| Workspace shape | **One new crate `rutster-brain-realtime` (library + binary), mirroring `rutster-tap-echo`** | Put the adapter in `examples/`; trait crate + impl crate | Slices-2's precedent. Real workspace member runs fmt/clippy/test/deny like everyone else. Reuses `rutster-tap`'s protocol types — the contract test that the wire types are reusable from outside the core. A future second brain is its own crate, same shape. |
+| Brain process port + protocol | **`ws://127.0.0.1:8082/realtime` (slice-2's echo brain defaults to `:8081/echo`)** | Single brain port with feature toggle | Two brain processes coexist (operator picks which to run via `RUTSTER_TAP_URL`). The default URL stays `ws://127.0.0.1:8081/echo` (slice-2 documented); slice-3 adds `ws://127.0.0.1:8082/realtime` as the documented OpenAI-brain default. |
+| Tap protocol extension posture | **Additive v1 events** (not v2) | v2 protocol bump; OpenAI Realtime event-schema adoption | Slice-2 §3.4's "unknown type → log + count + drop" rule makes additive events free — old echo brains ignore them. No wire-format break, no version negotiation needed. A v2 binary mode (raw LE i16 over WS binary frames) remains a future-rung optimization per slice-2 §9. |
+
+---
+
+## 11. References
+
+- [README.md](../../../README.md) — north star, capability ladder
+- [ARCHITECTURE.md §"Agent tap"](../../ARCHITECTURE.md) — the presumptive
+  tap shape this slice hardens against a real brain
+- [PORT_PLAN.md](../../PORT_PLAN.md) — capability checklist + thin-slice
+  phasing; §10 "WASM demoted, agent tap is the extension point"
+- [ADR-0002](../../adr/0002-north-star-and-fused-core.md) — fused vertical
+- [ADR-0007](../../adr/0007-trunk-rented-transport.md) — rent the trunk;
+  the brain is unaffected (PSTN reaches the reflex loop via media-leg
+  ingress, not via the tap)
+- [ADR-0008](../../adr/0008-fob-and-green-zone.md) — FOB / green-zone
+  doctrine; the brain is green-zone, the reflex loop is FOB, the tap is
+  core-authoritative (the S4 turn-ownership decision follows)
+- [Slice 1 — WebRTC media loopback](2026-06-28-slice-1-webrtc-loopback-design.md)
+  — this slice's foundation
+- [Slice 2 — the agent tap](2026-06-28-slice-2-agent-tap-design.md) — the
+  tap interface + the `TapAudioPipe` seam this slice composes on
+- [Vision-revision spec](2026-06-26-vision-revision-design.md) — the
+  pressure-test that produced the architecture
+- [Default UI design](2026-06-29-default-ui-design.md) — the operator
+  console design record (later-rung, not slice-3's concern)
+- [Vision sanity-check review S4](../reviews/2026-06-28-vision-sanity-check.md)
+  — the original finding that named the S4 turn-ownership decision;
+  slice-3 §4.3 resolves it under ADR-0008