diff --git a/docs/superpowers/specs/2026-06-30-slice-3-realtime-brain-design.md b/docs/superpowers/specs/2026-06-30-slice-3-realtime-brain-design.md new file mode 100644 index 0000000..7e42819 --- /dev/null +++ b/docs/superpowers/specs/2026-06-30-slice-3-realtime-brain-design.md @@ -0,0 +1,767 @@ +# Rutster slice 3 — Swap echo for OpenAI Realtime: the brain lands + +- **Status:** Draft (pending review) +- **Date:** 2026-06-30 +- **Spearhead step:** 3 of 6 (vision-revision §10 / PORT_PLAN "Phasing") +- **Origin:** brainstorming session 2026-06-30, resumed after the 2026-06-29 + strategic pivot (ADR-0007 + ADR-0008). +- **Depends on:** [slice 2 — the agent tap](2026-06-28-slice-2-agent-tap-design.md) + (must be implemented + green; the tap protocol and `TapAudioPipe` seam are + the foundation this slice swaps content into). +- **Related:** + [ADR-0002](../../adr/0002-north-star-and-fused-core.md) (fused vertical), + [ADR-0004](../../adr/0004-license.md) (GPL-3.0-or-later), + [ADR-0008](../../adr/0008-fob-and-green-zone.md) (the brain is **green-zone**; + the reflex loop is **FOB** — load-bearing for the S4 turn-ownership + decision below), + [ARCHITECTURE.md §"Agent tap"](../../ARCHITECTURE.md). + +--- + +## TL;DR + +Stand up spearhead step 3: swap slice-2's echo brain for a real +**speech-to-speech brain — OpenAI Realtime** — reached through slice-2's +existing tap. The core dials out (core-as-client; brain-as-server; still +**no inbound tap port on the core**), slice-2's tap protocol carries +audio + new event types (additive, v1, forwards-compatible), and the +brain process translates between our protocol and OpenAI Realtime's event +taxonomy. + +Slice 3 proves **agent integration**: the same WSS plumbing that today +echoes will, in step 4, carry VAD/barge-in signals back from the brain to +the core's reflex loop (the FOB). It deliberately defers real barge-in / +VAD-driven playout kill (step 4), the PSTN trunk (step 5), and spend +control (step 6) — but it *pre-paves* the `speech_started` / +`speech_stopped` event seam so step 4 lands cleanly. + +The seam slice-2 pre-paved (`AudioSource` / `AudioSink` traits + the +`TapAudioPipe` shape + the `TapEngine` task) is the **test of this +slice**: `RtcSession`'s media-loop path changes by zero lines; the +TapEngine's spawn / reconnect / teardown logic is untouched; only the +*tap protocol module* (`rutster-tap/src/protocol.rs`) grows new event +types and a new tool-registry module (`crates/rutster/src/tool_registry.rs`) +lands in the binary. + +--- + +## 1. Scope + +### 1.1 In scope + +- Implementation of spearhead step 3: WebRTC peer → core terminates + DTLS-SRTP, decodes Opus → canonical PCM @ 24 kHz mono, ships PCM + **over WSS** to an external OpenAI Realtime brain process, receives PCM + back, encodes + plays out via str0m. The user speaks and hears the AI + reply through an end-to-end speech-to-speech loop within ~700 ms + (slice-1's 200 ms + tap round-trip + OpenAI latency + 100 ms playout + buffer headroom). +- **`crates/rutster-brain-realtime`** — a new workspace member; library + + binary. Dual purpose like slice-2's `rutster-tap-echo`: a standalone + dev-loop binary (`cargo run -p rutster-brain-realtime`) and an in-process + `MockRealtimeBrain` for integration tests (no network calls to OpenAI). + Default port `ws://127.0.0.1:8082/realtime` (the slice-2 echo brain + defaults to `:8081/echo`; the two coexist). +- **Tap protocol extension (additive, v1, forwards-compatible):** + - `speech_started`, `speech_stopped` (brain → core, advisory). + - `function_call` (brain → core: tool name + args). + - `function_call_output` (core → brain: status + result). + - `tools.update` (brain → core: tool catalog so the core can validate). + Old echo brains ignore the new types per slice-2's "unknown type → log + + count + drop" rule (§3.4 of the slice-2 spec). +- **In-boundary tool registry** in the brown binary + (`crates/rutster/src/tool_registry.rs`). `hangup` is the only wired tool + — fires the existing `Channel: Connected → Closing` path. Other tool + names reply `status: "not_implemented"`. The brain's `tools.update` + event declares the catalog; the registry validates function_call events + against it before dispatch. +- **OpenAI Realtime translation layer** in `rutster-brain-realtime`: tap + events ↔ OpenAI Realtime events. Audio is 24 kHz mono PCM inside base64 + LE i16 — **matching slice-1's canonical tap format exactly**, no resample. +- **S4 turn-ownership decision (load-bearing per ADR-0008):** OpenAI + Realtime's `session.update` is sent with `turn_detection: null` + (disabled). The core drives turn-taking through its own (FOB) reflex + loop in step 4; OpenAI is treated as a speech-to-speech transducer. + The `speech_started` / `speech_stopped` events are caught and forwarded + through the tap protocol as advisory signals so step 4 can use them, + but OpenAI does **not** auto-bar the brain's `audio_out` — the + core-authoritative playout buffer (slice 2 §4.1) is the only thing + that gates playout. +- **Two-source config** for the brain process: `OPENAI_API_KEY` env + default + `OPENAI_API_KEY_FILE` path override. `OPENAI_REALTIME_MODEL` + env (default: `gpt-4o-realtime` or current equivalent; documented). +- **`--features=mock` dev mode** on the brain binary: runs the brain + process with an in-process mocked Realtime (no API key, no network + calls to OpenAI) — for offline dev loop + integration tests. +- New workspace deps: `tokio-tungstenite` (already pulled by slice-2), + `reqwest` or `serde_json` (already pulled), and **no new** workspace + member-deps beyond what slice-2 already pinned. Reuse slice-2's + protocol types from `rutster-tap`. + +### 1.2 Out of scope (with scheduled return) + +| Deferred item | Returns in | Why deferred | +|---|---|---| +| Real barge-in / VAD-driven playout kill | Step 4 | Slice-3 pre-paves the `speech_started` / `speech_stopped` advisory event seam; step 4 wires the FOB reflex loop to act on them. The core-authoritative playout buffer from slice-2 §4.1 is already in place. | +| PSTN trunk / rented-transport integration | Step 5 | ADR-0007's rented CPaaS raw-media fork lands the phone number; the brain is unaffected. | +| Spend cap / abuse gate | Step 6 | The brain has no spend surface yet — OpenAI bills the operator directly. In-boundary spend pacing lands with `rutster-spend` in step 6. | +| Multi-brain routing / per-tenant brain selection | Step 6 | One tap URL per call in slice-3 (env + per-call override). The slice-3 brain process is OpenAI-Realtime-only. A Deepgram+LLM+TTS composite adapter or a self-hosted open-weights brain is a future-rung concern; the tap protocol is brain-agnostic by design. | +| API-key rotation / KMS integration | Step 6 + later | `OPENAI_API_KEY` (env) + `OPENAI_API_KEY_FILE` (path) is the dev posture. KMS / Vault integration lands with the real trust boundary (step 6). | +| TLS on the tap and on the OpenAI leg | Step 6 | Slice-2 rejects `wss://` URLs at session-create (deferred to step 6); the OpenAI Realtime leg is `wss://` and uses rustls-native roots (no cert pinning in slice-3 — defer to step 6). | +| Authentication / authorization on the `tap_url` override | Step 6 | Inherits slice-2's "no auth yet" posture. | +| Re-INVITE / session migration / resumability | Later | Refresh the page → new session, same as slice-1/2. OpenAI Realtime session is per-call; no in-flight call preservation across server restart. | +| CDR / event bus / OTel beyond per-Channel `tracing` spans | Step 5 | Single peer + single brain; no fanout yet. Tool-call events go to logs + counters only. | +| Quality dashboard (containment, escalation reasons) | Capability ladder rung 3 | The current build target proves agent integration, not analytics. | +| Audio resampling for brains using 16 kHz | Future | OpenAI Realtime uses 24 kHz mono, matching slice-1's canonical tap format. Will be needed if a future brain uses 16 kHz; the translator resamples at that time. | +| `response.audio.delta` batching optimization | Future | OpenAI sends many small delta events; the translator MAY batch into the slice-2 `audio_out` 20 ms frame. Optimization optional for v1; tracked. | + +### 1.3 What this slice does NOT prove + +It does **not** prove: a real barge-in reflex (only the event seam), +latency determinism under reflex timing, PSTN trunking, spending controls, +multi-tenancy on the tap URL, API-key rotation, or wss:// TLS posture in +production. It proves **only** agent integration: the OpenAI Realtime +adapter as green-zone (per ADR-0008), the tap protocol extension carrying +audio + interruption signals + function-call plumbing, the in-boundary +tool registry dispatching `hangup` cleanly, and the seam test (slice-2's +`RtcSession` accepts the new brain with zero internal change). + +--- + +## 2. Workspace layout (delta on slice-2) + +One new workspace member; one new module in the binary; additive +extension to `rutster-tap`'s protocol module. + +``` +rutster/ +├── Cargo.toml # +rutster-brain-realtime in members +├── crates/ +│ ├── rutster/ # binary: +tool_registry module +│ │ ├── src/main.rs +│ │ ├── src/session_map.rs # +tool-call side-channel drain +│ │ ├── src/routes.rs # unchanged shape +│ │ ├── src/tap_engine.rs # UNCHANGED — the seam test +│ │ ├── src/tool_registry.rs # NEW: Tool trait, hangup tool +│ │ └── static/index.html # minor: surface brain connection status +│ ├── rutster-media/ # UNCHANGED — the seam test +│ │ └── src/loop_driver.rs # UNCHANGED +│ ├── rutster-call-model/ # UNCHANGED +│ ├── rutster-tap/ # +additive protocol event types +│ │ └── src/protocol.rs # +speech_started/stopped, function_call/output, tools.update +│ ├── rutster-tap-echo/ # UNCHANGED (still works against extended protocol) +│ ├── rutster-brain-realtime/ # NEW crate +│ │ ├── Cargo.toml # deps: rutster-tap, tokio, tokio-tungstenite, serde_json, tracing, url +│ │ ├── src/lib.rs # lib + MockRealtimeBrain for tests +│ │ ├── src/main.rs # standalone binary: ws://127.0.0.1:8082/realtime + OpenAI WS client +│ │ ├── src/translator.rs # tap ⇄ OpenAI event translation +│ │ └── src/openai_client.rs # wss://api.openai.com/v1/realtime client +│ ├── rutster-trunk/ # STUB (unchanged) +│ └── rutster-spend/ # STUB (unchanged) +└── examples/ + └── echo_brain/ # unchanged (Python reference; ignores new events) +``` + +### 2.1 Dependency direction + +- `rutster-brain-realtime` → `rutster-tap` (for protocol types — same + re-export pattern slice-2 established for `rutster-tap-echo`). +- `rutster-brain-realtime` is its own workspace member that's both a + binary (dev loop) and a library (test fixture). The library re-exports + a `MockRealtimeBrain` for use by integration tests in the binary crate. +- `rutster` (binary) gains `tool_registry.rs` as a sibling of + `tap_engine.rs` — the new module dispatches tool-call events the + TapClient observes via the new side-channel mpsc. +- `rutster-tap`'s protocol module is the contract both the brown core + and the new brain process share; new event types added here are re-used + by `rutster-brain-realtime` (the contract test that the wire types are + reusable from outside the core, exactly as `rutster-tap-echo` did). +- `rutster-media` and `rutster-call-model` are untouched. `loop_driver.rs` + and `rtc_session.rs` are byte-identical to slice-2 (post-review-fix) + baseline. + +### 2.2 Why one new crate for the OpenAI Realtime brain (not an `examples/` file) + +Mirrors slice-2's dual-purpose pattern for `rutster-tap-echo`: a real +workspace member that runs `cargo fmt`, `cargo clippy -D warnings`, +`cargo test`, and `cargo deny check` like every other crate. It reuses +`rutster-tap`'s protocol types — the contract test that the wire types +are reusable from **outside the core**, which is what makes the tap a +real extension point and not a closed system. + +A Python reference brain (`examples/openai_realtime_brain/openai_realtime_brain.py`) +is the canonical foreign-language brain demo for **OpenAI Realtime +specifically** (slice-2 already shipped the echo brain in Python). This +file is **not in CI** (Python would violate the zero-non-Rust-dev-deps +dev loop). README-documented runnable: `pip install websockets openai` +and run. + +### 2.3 Why the brain is green-zone (per ADR-0008) + +ADR-0008 classifies "the agent brain" explicitly under green-zone: it's +not hot-path inside the boundary (its round-trip is hundreds of ms; the +timing-critical work happens in the FOB reflex loop), not +security-constitutive (the core-authoritative playout buffer structurally +prevents the brain from flooding the wire), and not differentiating (the +tap-as-open-protocol is the differentiator; any brain speaking it works). +So the OpenAI Realtime adapter lives **outside** the FOB trust boundary — +its own process, its own API key, its own failure domain. The brain is +**reached by** the FOB; it does not live in it. The only FOB-side +additions in slice-3 are: + +- `rutster-tap/src/protocol.rs` — additive event types (protocol + extension, not new behavior — the existing TapClient + TapAudioPipe + surface is unchanged). +- `crates/rutster/src/tool_registry.rs` — in-boundary tool dispatch (a + security-constitutive capability: the brain proposes tool calls; the + FOB disposes. Per ADR-0007's spend-gate posture: "rutster mediates both + the provider call-control API and the brain tap, so the brain never + holds the wire"). + +--- + +## 3. Tap wire protocol extension (`rutster-tap/src/protocol.rs`) + +The slice-2 v1 protocol (envelope: `{v, type, seq, ts}`). The new event +types are additive — slice-2's §3.4 "unknown type → log + count + drop" +rule means an old brain (slice-2's `rutster-tap-echo`) ignores them. +Slice-2's envelope invariants (per-direction `seq`, advisory `ts`, +`samples: 480` for audio frames, explicit LE byte order) carry over +unchanged. + +### 3.1 Envelope (slice-2's, unchanged) + +```jsonc +{ + "v": 1, + "type": "", + "seq": , + "ts": +} +``` + +### 3.2 New events — brain → core (advisory) + +| `type` | payload fields | when | +|---|---|---| +| `speech_started` | `{}` | brain detected user speech started (translated from OpenAI `input_audio_buffer.speech_started`; advisory — the core’s FOB reflex loop, when wired in step 4, may use it for barge-in) | +| `speech_stopped` | `{}` | brain detected user speech stopped (translated from OpenAI `input_audio_buffer.speech_stopped`; advisory) | +| `tools.update` | `{ "tools": [{ "name": "", "description": "...", "parameters": {...} }] }` | brain declares its tool catalog (sent on `hello` ack, re-sent on tool catalog changes). The core's tool registry uses this to validate function_call events. | +| `function_call` | `{ "id": "", "name": "", "args": { ... } }` | brain wants the core to execute a tool (translated from OpenAI `response.function_call_arguments.done`). The core dispatches via the tool registry and replies via `function_call_output`. | + +### 3.3 New events — core → brain + +| `type` | payload fields | when | +|---|---|---| +| `function_call_output` | `{ "id": "", "status": "ok"\|"error"\|"not_implemented", "result": { ... } }` | core's tool-registry reply. Translated by the brain process into an OpenAI `conversation.item.create` with `type: function_call_output` so OpenAI continues the conversation appropriately. | + +### 3.4 Invariants + +- **Tool-call id:** `function_call` and `function_call_output` carry the + same `id` (a UUID minted by the brain; OpenAI calls it `call_id`, we + translate verbatim). Mismatches are logged + counted; the tool-registry + dispatch keys off the id. +- **Tool validation:** the core's tool registry validates `function_call` + events against the most recent `tools.update` catalog. An unknown tool + name → `function_call_output` with `status: "not_implemented"` (not + `error`; distinguishing "the FOB doesn't know this tool" from "the tool + failed"). The catalog is brain-authoritative — the brain declares it; + the core merely checks dispatch. +- **Advisory interruption events:** `speech_started` / `speech_stopped` + are **advisory only** in slice-3. The core logs + counts them; the FOB + reflex loop lands in step 4 and will use them. No slice-3 code path + acts on them in the hot media loop — `loop_driver.rs` is byte-identical + to slice-2 (post-review-fix) baseline. +- **Forward-compat:** unknown `type` values continue to be logged + + counted + dropped (slice-2 §3.4). The echo brain from slice-2 ignores + every new type this slice introduces; it echoes audio unchanged. + +### 3.5 Byte-order + audio format (unchanged from slice-2) + +PCM inside the base64 payload is explicit little-endian `i16` bytes (v1 +wire contract, slice-2 §3.4). OpenAI Realtime's audio events also use +24 kHz mono PCM inside base64 LE i16 — **no transcoding, no resample, +no endianness swap.** The translator passes the audio payload through +to the WS frame verbatim. This is a happy convergence that simplifies +the translator; it is not a coincidence (slice-1's choice of 24 kHz mono +was made with this ecosystem in mind). + +--- + +## 4. The OpenAI Realtime translation layer (`rutster-brain-realtime/src/translator.rs`) + +Pure (no schema of its own beyond the new tap event types); maps event +names + payload shapes between the two wire protocols. Owns no call +state beyond the OpenAI Realtime session/connection state. + +### 4.1 The two-leg brain process + +``` + WS server side WS client side + (core-as-client dials here) (brain-as-client dials OpenAI) + ┌──────────────────┐ ┌──────────┐ + tap protocol │ rutster-brain- │ translation layer │ OpenAI │ + core ◄────────┤ realtime ├────────────────────────────► │ Realtime │ + (audio_in/out, │ │ tap event ⇄ OpenAI event │ API WS │ + speech_started, │ │ │ (wss://api.openai.com)│ + func_call, ...) └──────────────────┘ └──────────┘ + │ + │ brain process also owns: + │ • OpenAI session.update (turn_detection: null — S4) + │ • interruption-signal forwarding (OpenAI → tap) + │ • function_call forwarding (bidirectional) + │ • tools.update on startup (catalog to core) +``` + +The translation layer is pure (no schema of its own beyond the new tap +event types); it maps event names + payload shapes between the two wire +protocols. It owns no call state beyond OpenAI's session/connection state. + +### 4.2 Event mapping table + +| Tap protocol | OpenAI Realtime | Notes | +|---|---|---| +| `hello` | `session.update` (modalities: ["audio", "text"], audio config, **turn_detection: null**) | Sent on handshake + reconnect; the S4 decision is encoded here | +| `audio_in` (base64 PCM LE i16) | `input_audio_buffer.append` (base64, same wire shape) | Pass-through; no transcoding | +| `audio_out` (from brain → core, advisory) | `response.audio.delta` events | The brain receives, formats, sends to core via tap_audio_out mpsc | +| `speech_started` (brain → core advisory) | `input_audio_buffer.speech_started` | Caught and forwarded as advisory | +| `speech_stopped` (brain → core advisory) | `input_audio_buffer.speech_stopped` | Caught and forwarded as advisory | +| `function_call` (brain → core) | `response.function_call_arguments.done` (OpenAI emits) → translator formats as tap function_call | Brain's translator extracts `call_id`, `name`, `arguments`; the core dispatches via tool registry | +| `function_call_output` (core → brain) | `conversation.item.create` with `item.type: function_call_output`, `call_id: `, `output: ` | Closes the tool-call loop | +| `bye` / `session_end` | `session.delete` + WS close | Graceful teardown | + +### 4.3 The S4 turn-ownership decision (load-bearing per ADR-0008) + +> ARCHITECTURE.md is right that the core should be authoritative on +> VAD/barge-in ("the tap carries the *results* of reflexes, not the +> responsibility"; "`AudioOut` advisory / core-authoritative"). But the +> most likely first brain — OpenAI Realtime — does its own server-side +> VAD and turn detection by default. Integrating step 3–4 means either: +> disable the brain's turn detection and feed it clean, locally-detected +> turns (preserves the reflex-authoritative principle, but is more +> integration work and fights the API's grain), or +> accept split-brain turn-taking where local VAD and the brain's VAD can +> disagree (double-triggers, dropped barge-ins). +> +> **Decision implied:** make "who owns turn detection" an explicit decision +> in the step-3 (brain) design doc, defaulting to core-authoritative, +> brain VAD off, with the integration cost budgeted. Don't let it be +> discovered at wiring time. +> — Claude, vision-sanity-check S4 + +ADR-0008's FOB/green-zone split makes this decidable without ambiguity: + +- **The reflex loop is FOB** (hot-path + differentiating — turn-taking, + VAD-driven barge-in, jitter, pacing). +- **The brain is green-zone** (per ADR-0008's explicit classification). +- **The tap is core-authoritative** (slice-2 §4.1 — the playout buffer + structurally prevents the brain from gating playout). + +The only doctrine-consistent posture is: **OpenAI Realtime's server-side +turn detection is disabled (`session.update` with `turn_detection: null`), +and the FOB owns turn-taking.** The `speech_started` / +`speech_stopped` events are forwarded as advisory signals so step 4 can +use them as one input to the FOB reflex loop — but OpenAI does not gate +playout, ever. The core-authoritative playout buffer (slice-2 §4.1) is +the only thing that gates playout in slice-3 (and continues to be in +step 4). + +**Why this is the right call now (rather than deferring to step 4):** +leaving OpenAI's turn detection on would mean slice-3's demo +"works by accident" — the slice-4 barge-in integration would then have to +disable it mid-flight, with unpredictable behavioral changes. By nailing +it down at slice-3's `session.update`, the brain process never wires up +the wrong posture and step 4's barge-in work composes cleanly on top. + +### 4.4 Failure mode + reconnect + +- **Tap-side WS reconnect (brain unreachable):** slice-2's + `TapEngine::spawn_tap_engine` already does bounded-backoff reconnect + (`250 ms → 500 ms → 1 s → 2 s → cap 5 s`, infinite retries, + re-`hello`s with the same `session_id`). The brain process being + unreachable looks identical to slice-2's echo brain being unreachable + — the slice-2 reconnect path is unchanged. The seam test (slice-2 + §8.5 #6) holds: `tap_engine.rs` is byte-identical. +- **OpenAI-side WS failure:** the brain process's own concern — it enters + its own bounded-backoff reconnect to OpenAI, and forwards a tap `error` + event (`{ "code": "brain_upstream_disconnect", "message": "..." }`) + to the core as an FYI. The core logs + counts; the call stays up; the + tap-side playout goes silent (slice-2's underflow path) until OpenAI + reconnects. The slice-3 brain's OpenAI-side reconnect is independent + of the slice-2 tap-side reconnect — two distinct failure surfaces. +- **API-key invalid (OpenAI returns 401):** the brain process emits a tap + `error` event (`{ "code": "brain_auth_failed", "message": "..." }`) + and exits the OpenAI leg. The core's response is unchanged from the + upstream-disconnect case (logs + silence + reconnect path). Operator + intervention; the call stays up. +- **Tool-registry dispatch failure:** a tool-registry error (e.g., + `hangup` fires but the channel state machine rejects the transition + because the call is already `Closing`) returns a `function_call_output` + with `status: "error"` and a result body explaining. The brain forwards + the error to OpenAI; OpenAI may or may not retry depending on its own + logic. No core-side retry; the model gets one chance. + +--- + +## 5. Lifecycle & integration + +### 5.1 Session lifecycle (slice-3 delta on slice-2) + +1. `POST /v1/sessions` — body still *optionally* carries `tap_url`. If + absent, falls back to `RUTSTER_TAP_URL` env. Default now documented + as either `ws://127.0.0.1:8081/echo` (slice-2 Rust echo brain) or + `ws://127.0.0.1:8082/realtime` (slice-3 OpenAI Realtime brain); the + operator picks which brain to run. +2. `POST /v1/sessions/:id/offer` — unchanged from slice-1/2; SDP answer. +3. On `Connected`: the binary's poll task observes the state transition + + spawns the `TapEngine` task (unchanged from slice-2). The + `TapEngine` connects to the brain process's WS server. +4. Brain process startup: connects to `wss://api.openai.com/v1/realtime`, + sends `session.update` with `turn_detection: null`, waits for OpenAI + `session.created` ack. +5. Brain process WS server accepts the core's tap WS; `hello` exchange; + brain process sends `tools.update` with the catalog (currently: + `hangup` only, plus any tool schemas the brain process's startup + config declares — see §6). +6. Audio flows: core decodes Opus → PCM → `audio_in` to brain → translator + formats as `input_audio_buffer.append` → OpenAI processes → + `response.audio.delta` → translator formats as `audio_out` → core's + playout ring → str0m encode. +7. Interruption signals: OpenAI `input_audio_buffer.speech_started` / + `.speech_stopped` → translator → tap `speech_started` / `speech_stopped` + → core logs + counts (advisory). Step 4 will wire these into the FOB + reflex loop. +8. Tool calls: OpenAI `response.function_call_arguments.done` → translator + → tap `function_call` → core's `tool_registry` dispatches (via the + TapClient's new side-channel mpsc) → `function_call_output` reply → + translator → OpenAI `conversation.item.create`. +9. `DELETE /v1/sessions/:id` or peer-close → `Closing`: slice-2's + unmodified teardown sequence fires — the TapEngine's `session_end` → + brain process's `session.delete` to OpenAI → WS close on both legs. + +### 5.2 The tool-call side-channel (the only brown-binary wiring) + +Slice-2's `TapClient` runs the WSS pump loop with two mpsc channels +(`tx_pcm_in` for inbound audio to brain, `rx_audio_out` for outbound +audio from brain). Slice-3 adds a third: + +- `tx_function_call: mpsc::Sender` — the TapClient + emits a `FunctionCallEvent` whenever it observes a tap `function_call` + message on its inbound stream. The binary's poll task drains this in + the same cycle it drains the existing `flush_tx` side-channel + (slice-2 §5.3 step 4) — same pattern, one extra channel. +- `rx_function_call_output: mpsc::Receiver` — + the binary writes `function_call_output` events through this channel; + the TapClient picks them up and sends them as tap WS frames. + +The shape keeps the seam test (§7 #5 of done criteria below): all this +lives in the binary's `tap_engine.rs` callers + the new `tool_registry.rs` +module. `loop_driver.rs` and `rtc_session.rs` are untouched. + +### 5.3 Brain process startup configuration + +| Env var | Purpose | Default | +|---|---|---| +| `RUTSTER_TAP_BIND` | WS server bind addr for the brain process | `127.0.0.1:8082` | +| `OPENAI_API_KEY` | OpenAI Realtime API key (mutually exclusive with `_FILE`) | required unless `--features=mock` | +| `OPENAI_API_KEY_FILE` | Path to a file containing the API key | optional; overrides `_KEY` | +| `OPENAI_REALTIME_MODEL` | OpenAI Realtime model id | `gpt-4o-realtime` (or current equivalent) | +| `OPENAI_REALTIME_VOICE` | Voice for TTS output | `alloy` (OpenAI's default; documented) | +| `RUTSTER_BRAIN_TOOLS` | Comma-separated tool names the brain should advertise in `tools.update` | `hangup` (only tool the core wires; others cause `not_implemented` replies) | + +The dev mode (`--features=mock`) doesn't read `OPENAI_API_KEY` and +instruments an in-process mock OpenAI Realtime WS server that generates +canned `response.audio.delta` events on `input_audio_buffer.append` and +asserts on the `session.update` having `turn_detection: null`. Used by: +the slice-3 integration test (no real OpenAI credentials, no network +calls), and the offline dev loop. + +--- + +## 6. The tool registry (`crates/rutster/src/tool_registry.rs`) + +A FOB-internal dispatch over the brain's proposed tool calls. The brain +proposes (via `function_call`); the FOB disposes (via `function_call_output`). +Per ADR-0007's spend-gate posture: the brain never holds the wire, +structurally. + +### 6.1 Trait + registry shape + +```rust +#[async_trait] +pub trait Tool: Send + Sync { + /// Tool name as the brain will reference it in function_call events. + fn name(&self) -> &str; + /// JSON-schema description the registry sends to the brain on tools.update. + fn schema(&self) -> serde_json::Value; + /// Execute the tool; return a result that the registry will serialize + /// into the function_call_output payload. + async fn call(&self, args: serde_json::Value) -> ToolResult; +} + +pub enum ToolResult { + Ok(serde_json::Value), + Error(String), + NotImplemented, +} + +pub struct ToolRegistry { + tools: Vec>, +} + +impl ToolRegistry { + pub fn new() -> Self; + pub fn register(&mut self, tool: Box); + /// Dispatch by tool name; returns ToolResult::NotImplemented if unknown. + pub async fn dispatch(&self, name: &str, args: serde_json::Value) -> ToolResult; + /// Used on startup + tools.update: enumerate the registered catalog. + pub fn catalog(&self) -> Vec; +} +``` + +### 6.2 The `hangup` tool (the only wired impl in slice-3) + +A `Tool` impl that on `call`: +1. Looks up the `ChannelId` (passed via the dispatch context — the + `tool_registry` is keyed by `ChannelId`, one registry per active + channel). +2. Fires `AppState::close(channel_id).await` — the existing slice-2 + teardown path (sets `Closing`, sends `session_end`, aborts the engine). +3. Returns `ToolResult::Ok({"channel_state": "Closing"})` so the brain + gets confirmation. + +Other tool names (anything in `RUTSTER_BRAIN_TOOLS` the brain advertises +that isn't `hangup`) → the registry returns `ToolResult::NotImplemented`: +the brain process forwards the reply as an OpenAI +`conversation.item.create` with the not_implemented status, and the model +is free to continue. + +### 6.3 What slice-3 does NOT wire + +- No `transfer` tool (escalation rung 2 — separate slice). +- No `lookup` / CRM-integration tools (business-logic integrations are + green-zone + future-rung). +- No spend/abuse-triggered tools (step 6). +- No dynamic tool registration at runtime (the catalog is fixed at + startup from `RUTSTER_BRAIN_TOOLS`). + +If a future brain proposes a tool not in the registry: `not_implemented` +reply; the model is free to retry with different arguments or give up. +The registry's job is to be the auditable dispatch boundary, not to be +extensible mid-call. + +--- + +## 7. CI, dev loop, testing + +### 7.1 New `[workspace.dependencies]` (Cargo.toml) + +- `async-trait = "0.1"` (for the `Tool` trait's async fn). +- `tokio-tungstenite`, `serde_json`, `tracing`, `url` — already pulled + by slice-2; reused. + +Member crate `rutster-brain-realtime` references these with +`dep.workspace = true`. + +### 7.2 CI (`.github/workflows/ci.yml`) + +Unchanged structure: `cargo fmt --check`, `cargo clippy -D warnings`, +`cargo test --all`, `cargo deny check`. The new `rutster-brain-realtime` +crate joins `--all`. **No Python in CI** — the OpenAI Realtime Python +reference brain is README-documented only. The Rust +`rutster-brain-realtime`'s in-process `MockRealtimeBrain` powers the +integration test; no real OpenAI credentials needed. + +### 7.3 Dev loop + +- `cargo run -p rutster-brain-realtime --features=mock` → starts the + brain process with an in-process mock OpenAI Realtime (on `:8082`): + no API key required, no network calls to OpenAI. +- `cargo run -p rutster-brain-realtime` (with `OPENAI_API_KEY` set) → + starts the brain process with the real OpenAI Realtime. +- `cargo run` (or `cargo run -p rutster`) → starts the axum signaling + server on `0.0.0.0:8080`, dials out to `$RUTSTER_TAP_URL` on each + session. +- Browser → `http://localhost:8080/` → click "Start call" → grant mic → + speak → hear AI reply through the brain process. +- `RUST_LOG=rutster=debug cargo run` for verbose tracing including + tap + tool-call events. +- `--features=echo` on the binary (slice-2 inherited) → bypasses the + brain entirely, routes audio through `EchoAudioPipe`. + +### 7.4 Testing strategy + +- **Unit tests in `rutster-brain-realtime`:** + - Protocol-event translation round-trips (tap `audio_in` ↔ OpenAI + `input_audio_buffer.append`; tap `function_call` ↔ OpenAI + `response.function_call_arguments.done`; tap `function_call_output` + ↔ OpenAI `conversation.item.create`). + - `session.update` body asserts `turn_detection: null` (S4 decision + encoded as a test). + - `tools.update` serialization round-trips. + - Error path: OpenAI-side WS error → tap `error` event forwarding. +- **Unit tests in `rutster-tap`:** + - New event types (de)serialization round-trips with golden JSON + fixtures in `tests/fixtures/`. + - Slice-2's existing echo-brain tests stay green (the new event types + are ignored by the echo brain — forwards-compat). +- **Unit tests in `rutster` `tool_registry.rs`:** + - `hangup` tool fires `AppState::close` correctly. + - Unknown tool name → `ToolResult::NotImplemented`. + - Registry catalog serialization. +- **Integration test in `rutster` binary crate:** spin up the axum server + (ephemeral port) + the in-process `MockRealtimeBrain` (ephemeral port) + + set `RUTSTER_TAP_URL`. Drive a synthetic WebRTC peer (extending + slice-2's `tap_integration.rs` harness): push PCM into the core via + the WebRTC peer → assert `audio_out` flows back through the tap (the + mock brain generates canned `response.audio.delta` on + `input_audio_buffer.append`) → assert they're re-encoded + pushed to + str0m. Plus: function-call round-trip (mock brain emits a + `function_call` for `hangup` → core's `tool_registry.dispatch("hangup")` + fires → channel state `Closing` → `session_end` over the tap → + mock brain's `session.delete` recorded). +- **Manual e2e test plan (README):** + 1. `cargo run -p rutster-brain-realtime --features=mock` (mock brain on `:8082`). + 2. `cargo run` (core on `:8080`). + 3. Browser → speak → hear mock-brain reply within ~250 ms (slice-2's + round-trip; no real OpenAI). + 4. Repeat with the real brain (`cargo run -p rutster-brain-realtime` + with `OPENAI_API_KEY` set) → end-to-end speech-to-speech with real + OpenAI Realtime within ~700 ms. + 5. Repeat with the Python brain (`python + examples/openai_realtime_brain/openai_realtime_brain.py`) → should + also work (proves the protocol is language-agnostic, same as + slice-2's Python echo brain). + 6. Function-call: trigger a model behavior that emits a tool call + (e.g. a test prompt that says "please hang up") → assert the call + closes + `session_end` over the tap + `session.delete` to OpenAI. + 7. `cargo test --all` green; `cargo fmt --check` / + `cargo clippy -D warnings` / `cargo deny check` green. + +### 7.5 Slice 3 "done" criteria + +The slice is complete when, on a clean checkout (+ open pivot / slice-1 +review fixes / slice-2 plan-rebaseline PRs merged): + +1. `cargo test --all` passes (unit + integration). The new + `rutster-brain-realtime` crate tests green alongside slice-1/2's suite. +2. `cargo fmt --check`, `cargo clippy -D warnings`, `cargo deny check` + all pass. +3. `cargo run -p rutster-brain-realtime --features=mock` + `cargo run` → + browser, speak, hear mock-brain reply within ~250 ms. +4. With `OPENAI_API_KEY` set: real OpenAI Realtime end-to-end within + ~700 ms. +5. **Both** `rutster-brain-realtime` (Rust, with `--features=mock`) **and** + `examples/openai_realtime_brain/openai_realtime_brain.py` (Python) + successfully interop against the core — proves the extended protocol + is language-agnostic. +6. **The seam test (load-bearing):** `rutster-media`'s `loop_driver.rs` + and `rtc_session.rs` keep their media-loop trait-method call sites + **byte-identical** to slice-2 (post-review-fix) baseline. The only + brown-binary additions are: `crates/rutster/src/tool_registry.rs` + (new module) + a new side-channel drain in `session_map.rs`. The + `tap_engine.rs` and `TapClient` get only additive new mpsc channels + (one new sender + one new receiver); the WSS pump loop's structure + is unchanged. A `git diff v -- + crates/rutster-media/src/loop_driver.rs + crates/rutster-media/src/rtc_session.rs` shows no behavior-changing + hunks (doc-comment or import changes permitted). +7. **S4 turn-ownership test:** the integration test asserts that the + `session.update` sent to OpenAI Realtime (or its mock equivalent) + contains `turn_detection: null`. The brain never auto-barges; the + core-authoritative playout buffer is the only playout gate. +8. `LEARNING.md` grows ≥3 new pointers: `async-trait` patterns → + `crates/rutster/src/tool_registry.rs`; OpenAI Realtime adapter → + `crates/rutster-brain-realtime/src/translator.rs`; tap protocol + extension + forwards-compat → `crates/rutster-tap/src/protocol.rs`. + +--- + +## 8. Open decisions (tracked) + +- **`response.audio.delta` batching.** OpenAI sends many small delta + events per response; the translator MAY batch into the slice-2 + `audio_out` 20 ms frame, or pass each delta through immediately as + its own `audio_out`. Batching reduces WS round-trips but adds latency. + Track + revisit after latency measurement with the real brain. +- **Tool-registry extensibility.** Slice-3 fixes the catalog at startup + via `RUTSTER_BRAIN_TOOLS`. Runtime-extensible registration (new tools + added mid-call) is a future-rung concern — would need a tool-register + API event. Track. +- **API-key rotation.** Slice-3 reads the key once at startup; a key + rotation requires a brain-process restart. KMS / Vault integration + lands with the real trust boundary (step 6). Track. +- **Voice + persona selection.** `OPENAI_REALTIME_VOICE` is the only + persona knob now; future brain processes may carry full prompt / + system-message customization (rung 2+). Track. +- **Multi-brain routing.** Slice-3 ships one brain process: OpenAI + Realtime. A Deepgram+LLM+TTS composite adapter or a self-hosted + open-weights brain would be its own crate + (`rutster-brain-deepgram`, `rutster-brain-local`) — the tap protocol + is brain-agnostic by design. Track + revisit when a second brain + lands. + +--- + +## 9. Out-of-scope re-check (against AGENTS.md + ADR-0007/0008) + +| Item | Status | +|---|---| +| Dedicated timing thread for media loop | Still step 4. | +| TLS on the HTTP signaling surface | Still step 5. | +| Authn / authz / multi-tenancy on `/v1/sessions` | Still step 6. | +| Trickle ICE | Unchanged. | +| Barge-in / VAD-driven playout kill | **Step 4.** Slice-3 pre-paves the `speech_started` / `speech_stopped` advisory event seam so step 4 lands cleanly. | +| PSTN trunk / rented transport | Still step 5. | +| Spend cap / abuse gate | Still step 6. | +| CDR / event bus / OTel beyond per-Channel tracing | Still step 5. | +| Browser automation / Playwright | Still post-slice-1. | +| Docker / compose | Still later-rung. | +| Transfer / park / pickup / barge (call features) | Still escalation rung 2. | + +If an agent proposes adding any of these in slice 3, the right answer is +"no, see the slice-3 spec §1.2." + +--- + +## 10. Key design decisions (summary of the brainstorming session) + +| Decision | Choice | Rejected alternatives | Why | +|---|---|---|---| +| Brain target | **OpenAI Realtime NOW; design for multi-brain later.** Adapter trait shape is the tap protocol itself, not a Rust trait. | Trait crate `rutster-brain` + impl crate; Deepgram+LLM+TTS now | Slice-2's protocol-as-contract framing already delivers "multi-brain later" without a speculative trait; YAGNI on abstraction. A Deepgram+LLM+TTS composite brain's friction will be in its own internal topology (composing three async WS/HTTP streams), not in `trait BrainAdapter` ergonomics — a trait wouldn't ease that anyway. | +| Cool scope | **Round-trip speech + interruption signals (advisory) + function-calling plumbing** | Speech only; speech + interruption; speech + interruption + function calling | Function calling is the gateway to escalation (rung 2 — `hangup` is the only wired tool; `transfer` lands later). Plumbing the FOB dispatch contract in slice-3 means step-4 barge-in composes on a clean foundation; deferring it would force step-4 to retrofit the seam. | +| Function-call handler | **Built-in tool registry in the core (FOB)** | Outbound tool-server URL (second egress); adapter handles function calls itself; plumbing only | Per ADR-0007: "rutster mediates both the provider call-control API and the brain tap, so the brain never holds the wire." The FOB disposes; the brain proposes. `hangup` is the only wired tool; others reply `not_implemented` honestly. The registry's job is the auditable dispatch boundary, not extensibility. | +| Turn-detection ownership (S4) | **Core-authoritative, brain VAD off** (`session.update` with `turn_detection: null`) | Accept split-brain (OpenAI VAD on; core-authoritative too → disagreement); defer the decision to step 4 | ADR-0008 makes the FOB/green-zone split mechanical: reflex loop is FOB, brain is green-zone, playout is core-authoritative. The only doctrine-consistent choice is to disable OpenAI's server-side VAD and let the FOB own turn-taking. Nailing it down at slice-3's `session.update` means step-4 barge-in composes cleanly. | +| API-key posture | **Env var + file-path override.** `OPENAI_API_KEY` env default; `OPENAI_API_KEY_FILE` overrides. | Env-only; KMS-required-now | Matches slice-2's "no auth yet" stance cleanly. KMS / Vault lands with the real trust boundary (step 6). The file-path override makes secret-manager injection (k8s secrets, Vault agent) trivial when that layer exists. | +| Dev mode | **`--features=mock` in-process mocked OpenAI Realtime** | No dev mock (always require real OpenAI); external mock server | The slice-3 dev loop must be zero-OpenAI-credentials + zero-network-calls-to-OpenAI. The integration test uses the same mock. Same pattern as slice-2's `EchoServer`. | +| Workspace shape | **One new crate `rutster-brain-realtime` (library + binary), mirroring `rutster-tap-echo`** | Put the adapter in `examples/`; trait crate + impl crate | Slices-2's precedent. Real workspace member runs fmt/clippy/test/deny like everyone else. Reuses `rutster-tap`'s protocol types — the contract test that the wire types are reusable from outside the core. A future second brain is its own crate, same shape. | +| Brain process port + protocol | **`ws://127.0.0.1:8082/realtime` (slice-2's echo brain defaults to `:8081/echo`)** | Single brain port with feature toggle | Two brain processes coexist (operator picks which to run via `RUTSTER_TAP_URL`). The default URL stays `ws://127.0.0.1:8081/echo` (slice-2 documented); slice-3 adds `ws://127.0.0.1:8082/realtime` as the documented OpenAI-brain default. | +| Tap protocol extension posture | **Additive v1 events** (not v2) | v2 protocol bump; OpenAI Realtime event-schema adoption | Slice-2 §3.4's "unknown type → log + count + drop" rule makes additive events free — old echo brains ignore them. No wire-format break, no version negotiation needed. A v2 binary mode (raw LE i16 over WS binary frames) remains a future-rung optimization per slice-2 §9. | + +--- + +## 11. References + +- [README.md](../../../README.md) — north star, capability ladder +- [ARCHITECTURE.md §"Agent tap"](../../ARCHITECTURE.md) — the presumptive + tap shape this slice hardens against a real brain +- [PORT_PLAN.md](../../PORT_PLAN.md) — capability checklist + thin-slice + phasing; §10 "WASM demoted, agent tap is the extension point" +- [ADR-0002](../../adr/0002-north-star-and-fused-core.md) — fused vertical +- [ADR-0007](../../adr/0007-trunk-rented-transport.md) — rent the trunk; + the brain is unaffected (PSTN reaches the reflex loop via media-leg + ingress, not via the tap) +- [ADR-0008](../../adr/0008-fob-and-green-zone.md) — FOB / green-zone + doctrine; the brain is green-zone, the reflex loop is FOB, the tap is + core-authoritative (the S4 turn-ownership decision follows) +- [Slice 1 — WebRTC media loopback](2026-06-28-slice-1-webrtc-loopback-design.md) + — this slice's foundation +- [Slice 2 — the agent tap](2026-06-28-slice-2-agent-tap-design.md) — the + tap interface + the `TapAudioPipe` seam this slice composes on +- [Vision-revision spec](2026-06-26-vision-revision-design.md) — the + pressure-test that produced the architecture +- [Default UI design](2026-06-29-default-ui-design.md) — the operator + console design record (later-rung, not slice-3's concern) +- [Vision sanity-check review S4](../reviews/2026-06-28-vision-sanity-check.md) + — the original finding that named the S4 turn-ownership decision; + slice-3 §4.3 resolves it under ADR-0008