rutster/docs/superpowers/specs/2026-06-30-slice-3-realtime-brain-design.md

# Rutster slice 3 — Swap echo for OpenAI Realtime: the brain lands

- **Status:** Draft (pending review)
- **Date:** 2026-06-30
- **Spearhead step:** 3 of 6 (vision-revision §10 / PORT_PLAN "Phasing")
- **Origin:** brainstorming session 2026-06-30, resumed after the 2026-06-29
  strategic pivot (ADR-0007 + ADR-0008).
- **Depends on:** [slice 2 — the agent tap](2026-06-28-slice-2-agent-tap-design.md)
  (must be implemented + green; the tap protocol and `TapAudioPipe` seam are
  the foundation this slice swaps content into).
- **Related:**
  [ADR-0002](../../adr/0002-north-star-and-fused-core.md) (fused vertical),
  [ADR-0004](../../adr/0004-license.md) (GPL-3.0-or-later),
  [ADR-0008](../../adr/0008-fob-and-green-zone.md) (the brain is **green-zone**;
  the reflex loop is **FOB** — load-bearing for the S4 turn-ownership
  decision below),
  [ARCHITECTURE.md §"Agent tap"](../../ARCHITECTURE.md).

---

## TL;DR

Stand up spearhead step 3: swap slice-2's echo brain for a real
**speech-to-speech brain — OpenAI Realtime** — reached through slice-2's
existing tap. The core dials out (core-as-client; brain-as-server; still
**no inbound tap port on the core**), slice-2's tap protocol carries
audio + new event types (additive, v1, forwards-compatible), and the
brain process translates between our protocol and OpenAI Realtime's event
taxonomy.

Slice 3 proves **agent integration**: the same WSS plumbing that today
echoes will, in step 4, carry VAD/barge-in signals back from the brain to
the core's reflex loop (the FOB). It deliberately defers real barge-in /
VAD-driven playout kill (step 4), the PSTN trunk (step 5), and spend
control (step 6) — but it *pre-paves* the `speech_started` /
`speech_stopped` event seam so step 4 lands cleanly.

The seam slice-2 pre-paved (`AudioSource` / `AudioSink` traits + the
`TapAudioPipe` shape + the `TapEngine` task) is the **test of this
slice**: `RtcSession`'s media-loop path changes by zero lines; the
TapEngine's spawn / reconnect / teardown logic is untouched; only the
*tap protocol module* (`rutster-tap/src/protocol.rs`) grows new event
types and a new tool-registry module (`crates/rutster/src/tool_registry.rs`)
lands in the binary.

---

## 1. Scope

### 1.1 In scope

- Implementation of spearhead step 3: WebRTC peer → core terminates
  DTLS-SRTP, decodes Opus → canonical PCM @ 24 kHz mono, ships PCM
  **over WSS** to an external OpenAI Realtime brain process, receives PCM
  back, encodes + plays out via str0m. The user speaks and hears the AI
  reply through an end-to-end speech-to-speech loop within ~700 ms
  (slice-1's 200 ms + tap round-trip + OpenAI latency + 100 ms playout
  buffer headroom).
- **`crates/rutster-brain-realtime`** — a new workspace member; library +
  binary. Dual purpose like slice-2's `rutster-tap-echo`: a standalone
  dev-loop binary (`cargo run -p rutster-brain-realtime`) and an in-process
  `MockRealtimeBrain` for integration tests (no network calls to OpenAI).
  Default port `ws://127.0.0.1:8082/realtime` (the slice-2 echo brain
  defaults to `:8081/echo`; the two coexist).
- **Tap protocol extension (additive, v1, forwards-compatible):**
  - `speech_started`, `speech_stopped` (brain → core, advisory).
  - `function_call` (brain → core: tool name + args).
  - `function_call_output` (core → brain: status + result).
  - `tools.update` (brain → core: tool catalog so the core can validate).
  Old echo brains ignore the new types per slice-2's "unknown type → log +
  count + drop" rule (§3.4 of the slice-2 spec).
- **In-boundary tool registry** in the brown binary
  (`crates/rutster/src/tool_registry.rs`). `hangup` is the only wired tool
  — fires the existing `Channel: Connected → Closing` path. Other tool
  names reply `status: "not_implemented"`. The brain's `tools.update`
  event declares the catalog; the registry validates function_call events
  against it before dispatch.
- **OpenAI Realtime translation layer** in `rutster-brain-realtime`: tap
  events ↔ OpenAI Realtime events. Audio is 24 kHz mono PCM inside base64
  LE i16 — **matching slice-1's canonical tap format exactly**, no resample.
- **S4 turn-ownership decision (load-bearing per ADR-0008):** OpenAI
  Realtime's `session.update` is sent with `turn_detection: null`
  (disabled). The core drives turn-taking through its own (FOB) reflex
  loop in step 4; OpenAI is treated as a speech-to-speech transducer.
  The `speech_started` / `speech_stopped` events are caught and forwarded
  through the tap protocol as advisory signals so step 4 can use them,
  but OpenAI does **not** auto-bar the brain's `audio_out` — the
  core-authoritative playout buffer (slice 2 §4.1) is the only thing
  that gates playout.
- **Two-source config** for the brain process: `OPENAI_API_KEY` env
  default + `OPENAI_API_KEY_FILE` path override. `OPENAI_REALTIME_MODEL`
  env (default: `gpt-4o-realtime` or current equivalent; documented).
- **`--features=mock` dev mode** on the brain binary: runs the brain
  process with an in-process mocked Realtime (no API key, no network
  calls to OpenAI) — for offline dev loop + integration tests.
- New workspace deps: `tokio-tungstenite` (already pulled by slice-2),
  `reqwest` or `serde_json` (already pulled), and **no new** workspace
  member-deps beyond what slice-2 already pinned. Reuse slice-2's
  protocol types from `rutster-tap`.

### 1.2 Out of scope (with scheduled return)

| Deferred item | Returns in | Why deferred |
|---|---|---|
| Real barge-in / VAD-driven playout kill | Step 4 | Slice-3 pre-paves the `speech_started` / `speech_stopped` advisory event seam; step 4 wires the FOB reflex loop to act on them. The core-authoritative playout buffer from slice-2 §4.1 is already in place. |
| PSTN trunk / rented-transport integration | Step 5 | ADR-0007's rented CPaaS raw-media fork lands the phone number; the brain is unaffected. |
| Spend cap / abuse gate | Step 6 | The brain has no spend surface yet — OpenAI bills the operator directly. In-boundary spend pacing lands with `rutster-spend` in step 6. |
| Multi-brain routing / per-tenant brain selection | Step 6 | One tap URL per call in slice-3 (env + per-call override). The slice-3 brain process is OpenAI-Realtime-only. A Deepgram+LLM+TTS composite adapter or a self-hosted open-weights brain is a future-rung concern; the tap protocol is brain-agnostic by design. |
| API-key rotation / KMS integration | Step 6 + later | `OPENAI_API_KEY` (env) + `OPENAI_API_KEY_FILE` (path) is the dev posture. KMS / Vault integration lands with the real trust boundary (step 6). |
| TLS on the tap and on the OpenAI leg | Step 6 | Slice-2 rejects `wss://` URLs at session-create (deferred to step 6); the OpenAI Realtime leg is `wss://` and uses rustls-native roots (no cert pinning in slice-3 — defer to step 6). |
| Authentication / authorization on the `tap_url` override | Step 6 | Inherits slice-2's "no auth yet" posture. |
| Re-INVITE / session migration / resumability | Later | Refresh the page → new session, same as slice-1/2. OpenAI Realtime session is per-call; no in-flight call preservation across server restart. |
| CDR / event bus / OTel beyond per-Channel `tracing` spans | Step 5 | Single peer + single brain; no fanout yet. Tool-call events go to logs + counters only. |
| Quality dashboard (containment, escalation reasons) | Capability ladder rung 3 | The current build target proves agent integration, not analytics. |
| Audio resampling for brains using 16 kHz | Future | OpenAI Realtime uses 24 kHz mono, matching slice-1's canonical tap format. Will be needed if a future brain uses 16 kHz; the translator resamples at that time. |
| `response.audio.delta` batching optimization | Future | OpenAI sends many small delta events; the translator MAY batch into the slice-2 `audio_out` 20 ms frame. Optimization optional for v1; tracked. |

### 1.3 What this slice does NOT prove

It does **not** prove: a real barge-in reflex (only the event seam),
latency determinism under reflex timing, PSTN trunking, spending controls,
multi-tenancy on the tap URL, API-key rotation, or wss:// TLS posture in
production. It proves **only** agent integration: the OpenAI Realtime
adapter as green-zone (per ADR-0008), the tap protocol extension carrying
audio + interruption signals + function-call plumbing, the in-boundary
tool registry dispatching `hangup` cleanly, and the seam test (slice-2's
`RtcSession` accepts the new brain with zero internal change).

---

## 2. Workspace layout (delta on slice-2)

One new workspace member; one new module in the binary; additive
extension to `rutster-tap`'s protocol module.

```
rutster/
├── Cargo.toml                          # +rutster-brain-realtime in members
├── crates/
│   ├── rutster/                         # binary: +tool_registry module
│   │   ├── src/main.rs
│   │   ├── src/session_map.rs          # +tool-call side-channel drain
│   │   ├── src/routes.rs               # unchanged shape
│   │   ├── src/tap_engine.rs           # UNCHANGED — the seam test
│   │   ├── src/tool_registry.rs        # NEW: Tool trait, hangup tool
│   │   └── static/index.html           # minor: surface brain connection status
│   ├── rutster-media/                   # UNCHANGED — the seam test
│   │   └── src/loop_driver.rs          # UNCHANGED
│   ├── rutster-call-model/              # UNCHANGED
│   ├── rutster-tap/                     # +additive protocol event types
│   │   └── src/protocol.rs             # +speech_started/stopped, function_call/output, tools.update
│   ├── rutster-tap-echo/                # UNCHANGED (still works against extended protocol)
│   ├── rutster-brain-realtime/          # NEW crate
│   │   ├── Cargo.toml                  # deps: rutster-tap, tokio, tokio-tungstenite, serde_json, tracing, url
│   │   ├── src/lib.rs                  # lib + MockRealtimeBrain for tests
│   │   ├── src/main.rs                 # standalone binary: ws://127.0.0.1:8082/realtime + OpenAI WS client
│   │   ├── src/translator.rs           # tap ⇄ OpenAI event translation
│   │   └── src/openai_client.rs         # wss://api.openai.com/v1/realtime client
│   ├── rutster-trunk/                   # STUB (unchanged)
│   └── rutster-spend/                   # STUB (unchanged)
└── examples/
    └── echo_brain/                      # unchanged (Python reference; ignores new events)
```

### 2.1 Dependency direction

- `rutster-brain-realtime` → `rutster-tap` (for protocol types — same
  re-export pattern slice-2 established for `rutster-tap-echo`).
- `rutster-brain-realtime` is its own workspace member that's both a
  binary (dev loop) and a library (test fixture). The library re-exports
  a `MockRealtimeBrain` for use by integration tests in the binary crate.
- `rutster` (binary) gains `tool_registry.rs` as a sibling of
  `tap_engine.rs` — the new module dispatches tool-call events the
  TapClient observes via the new side-channel mpsc.
- `rutster-tap`'s protocol module is the contract both the brown core
  and the new brain process share; new event types added here are re-used
  by `rutster-brain-realtime` (the contract test that the wire types are
  reusable from outside the core, exactly as `rutster-tap-echo` did).
- `rutster-media` and `rutster-call-model` are untouched. `loop_driver.rs`
  and `rtc_session.rs` are byte-identical to slice-2 (post-review-fix)
  baseline.

### 2.2 Why one new crate for the OpenAI Realtime brain (not an `examples/` file)

Mirrors slice-2's dual-purpose pattern for `rutster-tap-echo`: a real
workspace member that runs `cargo fmt`, `cargo clippy -D warnings`,
`cargo test`, and `cargo deny check` like every other crate. It reuses
`rutster-tap`'s protocol types — the contract test that the wire types
are reusable from **outside the core**, which is what makes the tap a
real extension point and not a closed system.

A Python reference brain (`examples/openai_realtime_brain/openai_realtime_brain.py`)
is the canonical foreign-language brain demo for **OpenAI Realtime
specifically** (slice-2 already shipped the echo brain in Python). This
file is **not in CI** (Python would violate the zero-non-Rust-dev-deps
dev loop). README-documented runnable: `pip install websockets openai`
and run.

### 2.3 Why the brain is green-zone (per ADR-0008)

ADR-0008 classifies "the agent brain" explicitly under green-zone: it's
not hot-path inside the boundary (its round-trip is hundreds of ms; the
timing-critical work happens in the FOB reflex loop), not
security-constitutive (the core-authoritative playout buffer structurally
prevents the brain from flooding the wire), and not differentiating (the
tap-as-open-protocol is the differentiator; any brain speaking it works).
So the OpenAI Realtime adapter lives **outside** the FOB trust boundary —
its own process, its own API key, its own failure domain. The brain is
**reached by** the FOB; it does not live in it. The only FOB-side
additions in slice-3 are:

- `rutster-tap/src/protocol.rs` — additive event types (protocol
  extension, not new behavior — the existing TapClient + TapAudioPipe
  surface is unchanged).
- `crates/rutster/src/tool_registry.rs` — in-boundary tool dispatch (a
  security-constitutive capability: the brain proposes tool calls; the
  FOB disposes. Per ADR-0007's spend-gate posture: "rutster mediates both
  the provider call-control API and the brain tap, so the brain never
  holds the wire").

---

## 3. Tap wire protocol extension (`rutster-tap/src/protocol.rs`)

The slice-2 v1 protocol (envelope: `{v, type, seq, ts}`). The new event
types are additive — slice-2's §3.4 "unknown type → log + count + drop"
rule means an old brain (slice-2's `rutster-tap-echo`) ignores them.
Slice-2's envelope invariants (per-direction `seq`, advisory `ts`,
`samples: 480` for audio frames, explicit LE byte order) carry over
unchanged.

### 3.1 Envelope (slice-2's, unchanged)

```jsonc
{
  "v": 1,
  "type": "<event_name>",
  "seq": <uint>,
  "ts": <uint>
}
```

### 3.2 New events — brain → core (advisory)

| `type` | payload fields | when |
|---|---|---|
| `speech_started` | `{}` | brain detected user speech started (translated from OpenAI `input_audio_buffer.speech_started`; advisory — the core’s FOB reflex loop, when wired in step 4, may use it for barge-in) |
| `speech_stopped` | `{}` | brain detected user speech stopped (translated from OpenAI `input_audio_buffer.speech_stopped`; advisory) |
| `tools.update` | `{ "tools": [{ "name": "<slug>", "description": "...", "parameters": {...} }] }` | brain declares its tool catalog (sent on `hello` ack, re-sent on tool catalog changes). The core's tool registry uses this to validate function_call events. |
| `function_call` | `{ "id": "<uuid>", "name": "<slug>", "args": { ... } }` | brain wants the core to execute a tool (translated from OpenAI `response.function_call_arguments.done`). The core dispatches via the tool registry and replies via `function_call_output`. |

### 3.3 New events — core → brain

| `type` | payload fields | when |
|---|---|---|
| `function_call_output` | `{ "id": "<uuid>", "status": "ok"\|"error"\|"not_implemented", "result": { ... } }` | core's tool-registry reply. Translated by the brain process into an OpenAI `conversation.item.create` with `type: function_call_output` so OpenAI continues the conversation appropriately. |

### 3.4 Invariants

- **Tool-call id:** `function_call` and `function_call_output` carry the
  same `id` (a UUID minted by the brain; OpenAI calls it `call_id`, we
  translate verbatim). Mismatches are logged + counted; the tool-registry
  dispatch keys off the id.
- **Tool validation:** the core's tool registry validates `function_call`
  events against the most recent `tools.update` catalog. An unknown tool
  name → `function_call_output` with `status: "not_implemented"` (not
  `error`; distinguishing "the FOB doesn't know this tool" from "the tool
  failed"). The catalog is brain-authoritative — the brain declares it;
  the core merely checks dispatch.
- **Advisory interruption events:** `speech_started` / `speech_stopped`
  are **advisory only** in slice-3. The core logs + counts them; the FOB
  reflex loop lands in step 4 and will use them. No slice-3 code path
  acts on them in the hot media loop — `loop_driver.rs` is byte-identical
  to slice-2 (post-review-fix) baseline.
- **Forward-compat:** unknown `type` values continue to be logged +
  counted + dropped (slice-2 §3.4). The echo brain from slice-2 ignores
  every new type this slice introduces; it echoes audio unchanged.

### 3.5 Byte-order + audio format (unchanged from slice-2)

PCM inside the base64 payload is explicit little-endian `i16` bytes (v1
wire contract, slice-2 §3.4). OpenAI Realtime's audio events also use
24 kHz mono PCM inside base64 LE i16 — **no transcoding, no resample,
no endianness swap.** The translator passes the audio payload through
to the WS frame verbatim. This is a happy convergence that simplifies
the translator; it is not a coincidence (slice-1's choice of 24 kHz mono
was made with this ecosystem in mind).

---

## 4. The OpenAI Realtime translation layer (`rutster-brain-realtime/src/translator.rs`)

Pure (no schema of its own beyond the new tap event types); maps event
names + payload shapes between the two wire protocols. Owns no call
state beyond the OpenAI Realtime session/connection state.

### 4.1 The two-leg brain process

```
                WS server side                        WS client side
        (core-as-client dials here)         (brain-as-client dials OpenAI)
                  ┌──────────────────┐                              ┌──────────┐
   tap protocol   │  rutster-brain-  │     translation layer         │  OpenAI  │
   core ◄────────┤  realtime         ├────────────────────────────►  │ Realtime │
  (audio_in/out,  │                  │   tap event ⇄ OpenAI event   │  API WS  │
  speech_started, │                  │                              │ (wss://api.openai.com)│
  func_call, ...) └──────────────────┘                              └──────────┘
                        │
                        │ brain process also owns:
                        │  • OpenAI session.update (turn_detection: null — S4)
                        │  • interruption-signal forwarding (OpenAI → tap)
                        │  • function_call forwarding (bidirectional)
                        │  • tools.update on startup (catalog to core)
```

The translation layer is pure (no schema of its own beyond the new tap
event types); it maps event names + payload shapes between the two wire
protocols. It owns no call state beyond OpenAI's session/connection state.

### 4.2 Event mapping table

| Tap protocol | OpenAI Realtime | Notes |
|---|---|---|
| `hello` | `session.update` (modalities: ["audio", "text"], audio config, **turn_detection: null**) | Sent on handshake + reconnect; the S4 decision is encoded here |
| `audio_in` (base64 PCM LE i16) | `input_audio_buffer.append` (base64, same wire shape) | Pass-through; no transcoding |
| `audio_out` (from brain → core, advisory) | `response.audio.delta` events | The brain receives, formats, sends to core via tap_audio_out mpsc |
| `speech_started` (brain → core advisory) | `input_audio_buffer.speech_started` | Caught and forwarded as advisory |
| `speech_stopped` (brain → core advisory) | `input_audio_buffer.speech_stopped` | Caught and forwarded as advisory |
| `function_call` (brain → core) | `response.function_call_arguments.done` (OpenAI emits) → translator formats as tap function_call | Brain's translator extracts `call_id`, `name`, `arguments`; the core dispatches via tool registry |
| `function_call_output` (core → brain) | `conversation.item.create` with `item.type: function_call_output`, `call_id: <id>`, `output: <result-json>` | Closes the tool-call loop |
| `bye` / `session_end` | `session.delete` + WS close | Graceful teardown |

### 4.3 The S4 turn-ownership decision (load-bearing per ADR-0008)

> ARCHITECTURE.md is right that the core should be authoritative on
> VAD/barge-in ("the tap carries the *results* of reflexes, not the
> responsibility"; "`AudioOut` advisory / core-authoritative"). But the
> most likely first brain — OpenAI Realtime — does its own server-side
> VAD and turn detection by default. Integrating step 3–4 means either:
> disable the brain's turn detection and feed it clean, locally-detected
> turns (preserves the reflex-authoritative principle, but is more
> integration work and fights the API's grain), or
> accept split-brain turn-taking where local VAD and the brain's VAD can
> disagree (double-triggers, dropped barge-ins).
>
> **Decision implied:** make "who owns turn detection" an explicit decision
> in the step-3 (brain) design doc, defaulting to core-authoritative,
> brain VAD off, with the integration cost budgeted. Don't let it be
> discovered at wiring time.
> — Claude, vision-sanity-check S4

ADR-0008's FOB/green-zone split makes this decidable without ambiguity:

- **The reflex loop is FOB** (hot-path + differentiating — turn-taking,
  VAD-driven barge-in, jitter, pacing).
- **The brain is green-zone** (per ADR-0008's explicit classification).
- **The tap is core-authoritative** (slice-2 §4.1 — the playout buffer
  structurally prevents the brain from gating playout).

The only doctrine-consistent posture is: **OpenAI Realtime's server-side
turn detection is disabled (`session.update` with `turn_detection: null`),
and the FOB owns turn-taking.** The `speech_started` /
`speech_stopped` events are forwarded as advisory signals so step 4 can
use them as one input to the FOB reflex loop — but OpenAI does not gate
playout, ever. The core-authoritative playout buffer (slice-2 §4.1) is
the only thing that gates playout in slice-3 (and continues to be in
step 4).

**Why this is the right call now (rather than deferring to step 4):**
leaving OpenAI's turn detection on would mean slice-3's demo
"works by accident" — the slice-4 barge-in integration would then have to
disable it mid-flight, with unpredictable behavioral changes. By nailing
it down at slice-3's `session.update`, the brain process never wires up
the wrong posture and step 4's barge-in work composes cleanly on top.

### 4.4 Failure mode + reconnect

- **Tap-side WS reconnect (brain unreachable):** slice-2's
  `TapEngine::spawn_tap_engine` already does bounded-backoff reconnect
  (`250 ms → 500 ms → 1 s → 2 s → cap 5 s`, infinite retries,
  re-`hello`s with the same `session_id`). The brain process being
  unreachable looks identical to slice-2's echo brain being unreachable
  — the slice-2 reconnect path is unchanged. The seam test (slice-2
  §8.5 #6) holds: `tap_engine.rs` is byte-identical.
- **OpenAI-side WS failure:** the brain process's own concern — it enters
  its own bounded-backoff reconnect to OpenAI, and forwards a tap `error`
  event (`{ "code": "brain_upstream_disconnect", "message": "..." }`)
  to the core as an FYI. The core logs + counts; the call stays up; the
  tap-side playout goes silent (slice-2's underflow path) until OpenAI
  reconnects. The slice-3 brain's OpenAI-side reconnect is independent
  of the slice-2 tap-side reconnect — two distinct failure surfaces.
- **API-key invalid (OpenAI returns 401):** the brain process emits a tap
  `error` event (`{ "code": "brain_auth_failed", "message": "..." }`)
  and exits the OpenAI leg. The core's response is unchanged from the
  upstream-disconnect case (logs + silence + reconnect path). Operator
  intervention; the call stays up.
- **Tool-registry dispatch failure:** a tool-registry error (e.g.,
  `hangup` fires but the channel state machine rejects the transition
  because the call is already `Closing`) returns a `function_call_output`
  with `status: "error"` and a result body explaining. The brain forwards
  the error to OpenAI; OpenAI may or may not retry depending on its own
  logic. No core-side retry; the model gets one chance.

---

## 5. Lifecycle & integration

### 5.1 Session lifecycle (slice-3 delta on slice-2)

1. `POST /v1/sessions` — body still *optionally* carries `tap_url`. If
   absent, falls back to `RUTSTER_TAP_URL` env. Default now documented
   as either `ws://127.0.0.1:8081/echo` (slice-2 Rust echo brain) or
   `ws://127.0.0.1:8082/realtime` (slice-3 OpenAI Realtime brain); the
   operator picks which brain to run.
2. `POST /v1/sessions/:id/offer` — unchanged from slice-1/2; SDP answer.
3. On `Connected`: the binary's poll task observes the state transition
   + spawns the `TapEngine` task (unchanged from slice-2). The
   `TapEngine` connects to the brain process's WS server.
4. Brain process startup: connects to `wss://api.openai.com/v1/realtime`,
   sends `session.update` with `turn_detection: null`, waits for OpenAI
   `session.created` ack.
5. Brain process WS server accepts the core's tap WS; `hello` exchange;
   brain process sends `tools.update` with the catalog (currently:
   `hangup` only, plus any tool schemas the brain process's startup
   config declares — see §6).
6. Audio flows: core decodes Opus → PCM → `audio_in` to brain → translator
   formats as `input_audio_buffer.append` → OpenAI processes →
   `response.audio.delta` → translator formats as `audio_out` → core's
   playout ring → str0m encode.
7. Interruption signals: OpenAI `input_audio_buffer.speech_started` /
   `.speech_stopped` → translator → tap `speech_started` / `speech_stopped`
   → core logs + counts (advisory). Step 4 will wire these into the FOB
   reflex loop.
8. Tool calls: OpenAI `response.function_call_arguments.done` → translator
   → tap `function_call` → core's `tool_registry` dispatches (via the
   TapClient's new side-channel mpsc) → `function_call_output` reply →
   translator → OpenAI `conversation.item.create`.
9. `DELETE /v1/sessions/:id` or peer-close → `Closing`: slice-2's
   unmodified teardown sequence fires — the TapEngine's `session_end` →
   brain process's `session.delete` to OpenAI → WS close on both legs.

### 5.2 The tool-call side-channel (the only brown-binary wiring)

Slice-2's `TapClient` runs the WSS pump loop with two mpsc channels
(`tx_pcm_in` for inbound audio to brain, `rx_audio_out` for outbound
audio from brain). Slice-3 adds a third:

- `tx_function_call: mpsc::Sender<FunctionCallEvent>` — the TapClient
  emits a `FunctionCallEvent` whenever it observes a tap `function_call`
  message on its inbound stream. The binary's poll task drains this in
  the same cycle it drains the existing `flush_tx` side-channel
  (slice-2 §5.3 step 4) — same pattern, one extra channel.
- `rx_function_call_output: mpsc::Receiver<FunctionCallOutputEvent>` —
  the binary writes `function_call_output` events through this channel;
  the TapClient picks them up and sends them as tap WS frames.

The shape keeps the seam test (§7 #5 of done criteria below): all this
lives in the binary's `tap_engine.rs` callers + the new `tool_registry.rs`
module. `loop_driver.rs` and `rtc_session.rs` are untouched.

### 5.3 Brain process startup configuration

| Env var | Purpose | Default |
|---|---|---|
| `RUTSTER_TAP_BIND` | WS server bind addr for the brain process | `127.0.0.1:8082` |
| `OPENAI_API_KEY` | OpenAI Realtime API key (mutually exclusive with `_FILE`) | required unless `--features=mock` |
| `OPENAI_API_KEY_FILE` | Path to a file containing the API key | optional; overrides `_KEY` |
| `OPENAI_REALTIME_MODEL` | OpenAI Realtime model id | `gpt-4o-realtime` (or current equivalent) |
| `OPENAI_REALTIME_VOICE` | Voice for TTS output | `alloy` (OpenAI's default; documented) |
| `RUTSTER_BRAIN_TOOLS` | Comma-separated tool names the brain should advertise in `tools.update` | `hangup` (only tool the core wires; others cause `not_implemented` replies) |

The dev mode (`--features=mock`) doesn't read `OPENAI_API_KEY` and
instruments an in-process mock OpenAI Realtime WS server that generates
canned `response.audio.delta` events on `input_audio_buffer.append` and
asserts on the `session.update` having `turn_detection: null`. Used by:
the slice-3 integration test (no real OpenAI credentials, no network
calls), and the offline dev loop.

---

## 6. The tool registry (`crates/rutster/src/tool_registry.rs`)

A FOB-internal dispatch over the brain's proposed tool calls. The brain
proposes (via `function_call`); the FOB disposes (via `function_call_output`).
Per ADR-0007's spend-gate posture: the brain never holds the wire,
structurally.

### 6.1 Trait + registry shape

```rust
#[async_trait]
pub trait Tool: Send + Sync {
    /// Tool name as the brain will reference it in function_call events.
    fn name(&self) -> &str;
    /// JSON-schema description the registry sends to the brain on tools.update.
    fn schema(&self) -> serde_json::Value;
    /// Execute the tool; return a result that the registry will serialize
    /// into the function_call_output payload.
    async fn call(&self, args: serde_json::Value) -> ToolResult;
}

pub enum ToolResult {
    Ok(serde_json::Value),
    Error(String),
    NotImplemented,
}

pub struct ToolRegistry {
    tools: Vec<Box<dyn Tool>>,
}

impl ToolRegistry {
    pub fn new() -> Self;
    pub fn register(&mut self, tool: Box<dyn Tool>);
    /// Dispatch by tool name; returns ToolResult::NotImplemented if unknown.
    pub async fn dispatch(&self, name: &str, args: serde_json::Value) -> ToolResult;
    /// Used on startup + tools.update: enumerate the registered catalog.
    pub fn catalog(&self) -> Vec<serde_json::Value>;
}
```

### 6.2 The `hangup` tool (the only wired impl in slice-3)

A `Tool` impl that on `call`:
1. Looks up the `ChannelId` (passed via the dispatch context — the
   `tool_registry` is keyed by `ChannelId`, one registry per active
   channel).
2. Fires `AppState::close(channel_id).await` — the existing slice-2
   teardown path (sets `Closing`, sends `session_end`, aborts the engine).
3. Returns `ToolResult::Ok({"channel_state": "Closing"})` so the brain
   gets confirmation.

Other tool names (anything in `RUTSTER_BRAIN_TOOLS` the brain advertises
that isn't `hangup`) → the registry returns `ToolResult::NotImplemented`:
the brain process forwards the reply as an OpenAI
`conversation.item.create` with the not_implemented status, and the model
is free to continue.

### 6.3 What slice-3 does NOT wire

- No `transfer` tool (escalation rung 2 — separate slice).
- No `lookup` / CRM-integration tools (business-logic integrations are
  green-zone + future-rung).
- No spend/abuse-triggered tools (step 6).
- No dynamic tool registration at runtime (the catalog is fixed at
  startup from `RUTSTER_BRAIN_TOOLS`).

If a future brain proposes a tool not in the registry: `not_implemented`
reply; the model is free to retry with different arguments or give up.
The registry's job is to be the auditable dispatch boundary, not to be
extensible mid-call.

---

## 7. CI, dev loop, testing

### 7.1 New `[workspace.dependencies]` (Cargo.toml)

- `async-trait = "0.1"` (for the `Tool` trait's async fn).
- `tokio-tungstenite`, `serde_json`, `tracing`, `url` — already pulled
  by slice-2; reused.

Member crate `rutster-brain-realtime` references these with
`dep.workspace = true`.

### 7.2 CI (`.github/workflows/ci.yml`)

Unchanged structure: `cargo fmt --check`, `cargo clippy -D warnings`,
`cargo test --all`, `cargo deny check`. The new `rutster-brain-realtime`
crate joins `--all`. **No Python in CI** — the OpenAI Realtime Python
reference brain is README-documented only. The Rust
`rutster-brain-realtime`'s in-process `MockRealtimeBrain` powers the
integration test; no real OpenAI credentials needed.

### 7.3 Dev loop

- `cargo run -p rutster-brain-realtime --features=mock` → starts the
  brain process with an in-process mock OpenAI Realtime (on `:8082`):
  no API key required, no network calls to OpenAI.
- `cargo run -p rutster-brain-realtime` (with `OPENAI_API_KEY` set) →
  starts the brain process with the real OpenAI Realtime.
- `cargo run` (or `cargo run -p rutster`) → starts the axum signaling
  server on `0.0.0.0:8080`, dials out to `$RUTSTER_TAP_URL` on each
  session.
- Browser → `http://localhost:8080/` → click "Start call" → grant mic →
  speak → hear AI reply through the brain process.
- `RUST_LOG=rutster=debug cargo run` for verbose tracing including
  tap + tool-call events.
- `--features=echo` on the binary (slice-2 inherited) → bypasses the
  brain entirely, routes audio through `EchoAudioPipe`.

### 7.4 Testing strategy

- **Unit tests in `rutster-brain-realtime`:**
  - Protocol-event translation round-trips (tap `audio_in` ↔ OpenAI
    `input_audio_buffer.append`; tap `function_call` ↔ OpenAI
    `response.function_call_arguments.done`; tap `function_call_output`
    ↔ OpenAI `conversation.item.create`).
  - `session.update` body asserts `turn_detection: null` (S4 decision
    encoded as a test).
  - `tools.update` serialization round-trips.
  - Error path: OpenAI-side WS error → tap `error` event forwarding.
- **Unit tests in `rutster-tap`:**
  - New event types (de)serialization round-trips with golden JSON
    fixtures in `tests/fixtures/`.
  - Slice-2's existing echo-brain tests stay green (the new event types
    are ignored by the echo brain — forwards-compat).
- **Unit tests in `rutster` `tool_registry.rs`:**
  - `hangup` tool fires `AppState::close` correctly.
  - Unknown tool name → `ToolResult::NotImplemented`.
  - Registry catalog serialization.
- **Integration test in `rutster` binary crate:** spin up the axum server
  (ephemeral port) + the in-process `MockRealtimeBrain` (ephemeral port) +
  set `RUTSTER_TAP_URL`. Drive a synthetic WebRTC peer (extending
  slice-2's `tap_integration.rs` harness): push PCM into the core via
  the WebRTC peer → assert `audio_out` flows back through the tap (the
  mock brain generates canned `response.audio.delta` on
  `input_audio_buffer.append`) → assert they're re-encoded + pushed to
  str0m. Plus: function-call round-trip (mock brain emits a
  `function_call` for `hangup` → core's `tool_registry.dispatch("hangup")`
  fires → channel state `Closing` → `session_end` over the tap →
  mock brain's `session.delete` recorded).
- **Manual e2e test plan (README):**
  1. `cargo run -p rutster-brain-realtime --features=mock` (mock brain on `:8082`).
  2. `cargo run` (core on `:8080`).
  3. Browser → speak → hear mock-brain reply within ~250 ms (slice-2's
     round-trip; no real OpenAI).
  4. Repeat with the real brain (`cargo run -p rutster-brain-realtime`
     with `OPENAI_API_KEY` set) → end-to-end speech-to-speech with real
     OpenAI Realtime within ~700 ms.
  5. Repeat with the Python brain (`python
     examples/openai_realtime_brain/openai_realtime_brain.py`) → should
     also work (proves the protocol is language-agnostic, same as
     slice-2's Python echo brain).
  6. Function-call: trigger a model behavior that emits a tool call
     (e.g. a test prompt that says "please hang up") → assert the call
     closes + `session_end` over the tap + `session.delete` to OpenAI.
  7. `cargo test --all` green; `cargo fmt --check` /
     `cargo clippy -D warnings` / `cargo deny check` green.

### 7.5 Slice 3 "done" criteria

The slice is complete when, on a clean checkout (+ open pivot / slice-1
review fixes / slice-2 plan-rebaseline PRs merged):

1. `cargo test --all` passes (unit + integration). The new
   `rutster-brain-realtime` crate tests green alongside slice-1/2's suite.
2. `cargo fmt --check`, `cargo clippy -D warnings`, `cargo deny check`
   all pass.
3. `cargo run -p rutster-brain-realtime --features=mock` + `cargo run` →
   browser, speak, hear mock-brain reply within ~250 ms.
4. With `OPENAI_API_KEY` set: real OpenAI Realtime end-to-end within
   ~700 ms.
5. **Both** `rutster-brain-realtime` (Rust, with `--features=mock`) **and**
   `examples/openai_realtime_brain/openai_realtime_brain.py` (Python)
   successfully interop against the core — proves the extended protocol
   is language-agnostic.
6. **The seam test (load-bearing):** `rutster-media`'s `loop_driver.rs`
   and `rtc_session.rs` keep their media-loop trait-method call sites
   **byte-identical** to slice-2 (post-review-fix) baseline. The only
   brown-binary additions are: `crates/rutster/src/tool_registry.rs`
   (new module) + a new side-channel drain in `session_map.rs`. The
   `tap_engine.rs` and `TapClient` get only additive new mpsc channels
   (one new sender + one new receiver); the WSS pump loop's structure
   is unchanged. A `git diff v<slice-2-post-review-fix-tag> --
   crates/rutster-media/src/loop_driver.rs
   crates/rutster-media/src/rtc_session.rs` shows no behavior-changing
   hunks (doc-comment or import changes permitted).
7. **S4 turn-ownership test:** the integration test asserts that the
   `session.update` sent to OpenAI Realtime (or its mock equivalent)
   contains `turn_detection: null`. The brain never auto-barges; the
   core-authoritative playout buffer is the only playout gate.
8. `LEARNING.md` grows ≥3 new pointers: `async-trait` patterns →
   `crates/rutster/src/tool_registry.rs`; OpenAI Realtime adapter →
   `crates/rutster-brain-realtime/src/translator.rs`; tap protocol
   extension + forwards-compat → `crates/rutster-tap/src/protocol.rs`.

---

## 8. Open decisions (tracked)

- **`response.audio.delta` batching.** OpenAI sends many small delta
  events per response; the translator MAY batch into the slice-2
  `audio_out` 20 ms frame, or pass each delta through immediately as
  its own `audio_out`. Batching reduces WS round-trips but adds latency.
  Track + revisit after latency measurement with the real brain.
- **Tool-registry extensibility.** Slice-3 fixes the catalog at startup
  via `RUTSTER_BRAIN_TOOLS`. Runtime-extensible registration (new tools
  added mid-call) is a future-rung concern — would need a tool-register
  API event. Track.
- **API-key rotation.** Slice-3 reads the key once at startup; a key
  rotation requires a brain-process restart. KMS / Vault integration
  lands with the real trust boundary (step 6). Track.
- **Voice + persona selection.** `OPENAI_REALTIME_VOICE` is the only
  persona knob now; future brain processes may carry full prompt /
  system-message customization (rung 2+). Track.
- **Multi-brain routing.** Slice-3 ships one brain process: OpenAI
  Realtime. A Deepgram+LLM+TTS composite adapter or a self-hosted
  open-weights brain would be its own crate
  (`rutster-brain-deepgram`, `rutster-brain-local`) — the tap protocol
  is brain-agnostic by design. Track + revisit when a second brain
  lands.

---

## 9. Out-of-scope re-check (against AGENTS.md + ADR-0007/0008)

| Item | Status |
|---|---|
| Dedicated timing thread for media loop | Still step 4. |
| TLS on the HTTP signaling surface | Still step 5. |
| Authn / authz / multi-tenancy on `/v1/sessions` | Still step 6. |
| Trickle ICE | Unchanged. |
| Barge-in / VAD-driven playout kill | **Step 4.** Slice-3 pre-paves the `speech_started` / `speech_stopped` advisory event seam so step 4 lands cleanly. |
| PSTN trunk / rented transport | Still step 5. |
| Spend cap / abuse gate | Still step 6. |
| CDR / event bus / OTel beyond per-Channel tracing | Still step 5. |
| Browser automation / Playwright | Still post-slice-1. |
| Docker / compose | Still later-rung. |
| Transfer / park / pickup / barge (call features) | Still escalation rung 2. |

If an agent proposes adding any of these in slice 3, the right answer is
"no, see the slice-3 spec §1.2."

---

## 10. Key design decisions (summary of the brainstorming session)

| Decision | Choice | Rejected alternatives | Why |
|---|---|---|---|
| Brain target | **OpenAI Realtime NOW; design for multi-brain later.** Adapter trait shape is the tap protocol itself, not a Rust trait. | Trait crate `rutster-brain` + impl crate; Deepgram+LLM+TTS now | Slice-2's protocol-as-contract framing already delivers "multi-brain later" without a speculative trait; YAGNI on abstraction. A Deepgram+LLM+TTS composite brain's friction will be in its own internal topology (composing three async WS/HTTP streams), not in `trait BrainAdapter` ergonomics — a trait wouldn't ease that anyway. |
| Cool scope | **Round-trip speech + interruption signals (advisory) + function-calling plumbing** | Speech only; speech + interruption; speech + interruption + function calling | Function calling is the gateway to escalation (rung 2 — `hangup` is the only wired tool; `transfer` lands later). Plumbing the FOB dispatch contract in slice-3 means step-4 barge-in composes on a clean foundation; deferring it would force step-4 to retrofit the seam. |
| Function-call handler | **Built-in tool registry in the core (FOB)** | Outbound tool-server URL (second egress); adapter handles function calls itself; plumbing only | Per ADR-0007: "rutster mediates both the provider call-control API and the brain tap, so the brain never holds the wire." The FOB disposes; the brain proposes. `hangup` is the only wired tool; others reply `not_implemented` honestly. The registry's job is the auditable dispatch boundary, not extensibility. |
| Turn-detection ownership (S4) | **Core-authoritative, brain VAD off** (`session.update` with `turn_detection: null`) | Accept split-brain (OpenAI VAD on; core-authoritative too → disagreement); defer the decision to step 4 | ADR-0008 makes the FOB/green-zone split mechanical: reflex loop is FOB, brain is green-zone, playout is core-authoritative. The only doctrine-consistent choice is to disable OpenAI's server-side VAD and let the FOB own turn-taking. Nailing it down at slice-3's `session.update` means step-4 barge-in composes cleanly. |
| API-key posture | **Env var + file-path override.** `OPENAI_API_KEY` env default; `OPENAI_API_KEY_FILE` overrides. | Env-only; KMS-required-now | Matches slice-2's "no auth yet" stance cleanly. KMS / Vault lands with the real trust boundary (step 6). The file-path override makes secret-manager injection (k8s secrets, Vault agent) trivial when that layer exists. |
| Dev mode | **`--features=mock` in-process mocked OpenAI Realtime** | No dev mock (always require real OpenAI); external mock server | The slice-3 dev loop must be zero-OpenAI-credentials + zero-network-calls-to-OpenAI. The integration test uses the same mock. Same pattern as slice-2's `EchoServer`. |
| Workspace shape | **One new crate `rutster-brain-realtime` (library + binary), mirroring `rutster-tap-echo`** | Put the adapter in `examples/`; trait crate + impl crate | Slices-2's precedent. Real workspace member runs fmt/clippy/test/deny like everyone else. Reuses `rutster-tap`'s protocol types — the contract test that the wire types are reusable from outside the core. A future second brain is its own crate, same shape. |
| Brain process port + protocol | **`ws://127.0.0.1:8082/realtime` (slice-2's echo brain defaults to `:8081/echo`)** | Single brain port with feature toggle | Two brain processes coexist (operator picks which to run via `RUTSTER_TAP_URL`). The default URL stays `ws://127.0.0.1:8081/echo` (slice-2 documented); slice-3 adds `ws://127.0.0.1:8082/realtime` as the documented OpenAI-brain default. |
| Tap protocol extension posture | **Additive v1 events** (not v2) | v2 protocol bump; OpenAI Realtime event-schema adoption | Slice-2 §3.4's "unknown type → log + count + drop" rule makes additive events free — old echo brains ignore them. No wire-format break, no version negotiation needed. A v2 binary mode (raw LE i16 over WS binary frames) remains a future-rung optimization per slice-2 §9. |

---

## 11. References

- [README.md](../../../README.md) — north star, capability ladder
- [ARCHITECTURE.md §"Agent tap"](../../ARCHITECTURE.md) — the presumptive
  tap shape this slice hardens against a real brain
- [PORT_PLAN.md](../../PORT_PLAN.md) — capability checklist + thin-slice
  phasing; §10 "WASM demoted, agent tap is the extension point"
- [ADR-0002](../../adr/0002-north-star-and-fused-core.md) — fused vertical
- [ADR-0007](../../adr/0007-trunk-rented-transport.md) — rent the trunk;
  the brain is unaffected (PSTN reaches the reflex loop via media-leg
  ingress, not via the tap)
- [ADR-0008](../../adr/0008-fob-and-green-zone.md) — FOB / green-zone
  doctrine; the brain is green-zone, the reflex loop is FOB, the tap is
  core-authoritative (the S4 turn-ownership decision follows)
- [Slice 1 — WebRTC media loopback](2026-06-28-slice-1-webrtc-loopback-design.md)
  — this slice's foundation
- [Slice 2 — the agent tap](2026-06-28-slice-2-agent-tap-design.md) — the
  tap interface + the `TapAudioPipe` seam this slice composes on
- [Vision-revision spec](2026-06-26-vision-revision-design.md) — the
  pressure-test that produced the architecture
- [Default UI design](2026-06-29-default-ui-design.md) — the operator
  console design record (later-rung, not slice-3's concern)
- [Vision sanity-check review S4](../reviews/2026-06-28-vision-sanity-check.md)
  — the original finding that named the S4 turn-ownership decision;
  slice-3 §4.3 resolves it under ADR-0008