rutster/docs/superpowers/specs/2026-06-30-slice-3-realtime-brain-design.md

Rutster slice 3 — Swap echo for OpenAI Realtime: the brain lands

• Status: Draft (pending review)
• Date: 2026-06-30
• Spearhead step: 3 of 6 (vision-revision §10 / PORT_PLAN "Phasing")
• Origin: brainstorming session 2026-06-30, resumed after the 2026-06-29 strategic pivot (ADR-0007 + ADR-0008).
• Depends on: slice 2 — the agent tap (must be implemented + green; the tap protocol and TapAudioPipe seam are the foundation this slice swaps content into).
• Related: ADR-0002 (fused vertical), ADR-0004 (GPL-3.0-or-later), ADR-0008 (the brain is green-zone; the reflex loop is FOB — load-bearing for the S4 turn-ownership decision below), ARCHITECTURE.md §"Agent tap".

---

TL;DR

Stand up spearhead step 3: swap slice-2's echo brain for a real speech-to-speech brain — OpenAI Realtime — reached through slice-2's existing tap. The core dials out (core-as-client; brain-as-server; still no inbound tap port on the core), slice-2's tap protocol carries audio + new event types (additive, v1, forwards-compatible), and the brain process translates between our protocol and OpenAI Realtime's event taxonomy.

Slice 3 proves agent integration: the same WSS plumbing that today echoes will, in step 4, carry VAD/barge-in signals back from the brain to the core's reflex loop (the FOB). It deliberately defers real barge-in / VAD-driven playout kill (step 4), the PSTN trunk (step 5), and spend control (step 6) — but it pre-paves the speech_started / speech_stopped event seam so step 4 lands cleanly.

The seam slice-2 pre-paved (AudioSource / AudioSink traits + the TapAudioPipe shape + the TapEngine task) is the test of this slice: RtcSession's media-loop path changes by zero lines; the TapEngine's spawn / reconnect / teardown logic is untouched; only the tap protocol module (rutster-tap/src/protocol.rs) grows new event types and a new tool-registry module (crates/rutster/src/tool_registry.rs) lands in the binary.

---

1. Scope

1.1 In scope

• Implementation of spearhead step 3: WebRTC peer → core terminates DTLS-SRTP, decodes Opus → canonical PCM @ 24 kHz mono, ships PCM over WSS to an external OpenAI Realtime brain process, receives PCM back, encodes + plays out via str0m. The user speaks and hears the AI reply through an end-to-end speech-to-speech loop within ~700 ms (slice-1's 200 ms + tap round-trip + OpenAI latency + 100 ms playout buffer headroom).
• crates/rutster-brain-realtime — a new workspace member; library + binary. Dual purpose like slice-2's rutster-tap-echo: a standalone dev-loop binary (cargo run -p rutster-brain-realtime) and an in-process MockRealtimeBrain for integration tests (no network calls to OpenAI). Default port ws://127.0.0.1:8082/realtime (the slice-2 echo brain defaults to :8081/echo; the two coexist).
• Tap protocol extension (additive, v1, forwards-compatible):
  • speech_started, speech_stopped (brain → core, advisory).
  • function_call (brain → core: tool name + args).
  • function_call_output (core → brain: status + result).
  • tools.update (brain → core: tool catalog so the core can validate). Old echo brains ignore the new types per slice-2's "unknown type → log + count + drop" rule (§3.4 of the slice-2 spec).
• In-boundary tool registry in the brown binary (crates/rutster/src/tool_registry.rs). hangup is the only wired tool — fires the existing Channel: Connected → Closing path. Other tool names reply status: "not_implemented". The brain's tools.update event declares the catalog; the registry validates function_call events against it before dispatch.
• OpenAI Realtime translation layer in rutster-brain-realtime: tap events ↔ OpenAI Realtime events. Audio is 24 kHz mono PCM inside base64 LE i16 — matching slice-1's canonical tap format exactly, no resample.
• S4 turn-ownership decision (load-bearing per ADR-0008): OpenAI Realtime's session.update is sent with turn_detection: null (disabled). The core drives turn-taking through its own (FOB) reflex loop in step 4; OpenAI is treated as a speech-to-speech transducer. The speech_started / speech_stopped events are caught and forwarded through the tap protocol as advisory signals so step 4 can use them, but OpenAI does not auto-bar the brain's audio_out — the core-authoritative playout buffer (slice 2 §4.1) is the only thing that gates playout.
• Two-source config for the brain process: OPENAI_API_KEY env default + OPENAI_API_KEY_FILE path override. OPENAI_REALTIME_MODEL env (default: gpt-4o-realtime or current equivalent; documented).
• --features=mock dev mode on the brain binary: runs the brain process with an in-process mocked Realtime (no API key, no network calls to OpenAI) — for offline dev loop + integration tests.
• New workspace deps: tokio-tungstenite (already pulled by slice-2), reqwest or serde_json (already pulled), and no new workspace member-deps beyond what slice-2 already pinned. Reuse slice-2's protocol types from rutster-tap.

1.2 Out of scope (with scheduled return)

Deferred item	Returns in	Why deferred
Real barge-in / VAD-driven playout kill	Step 4	Slice-3 pre-paves the speech_started / speech_stopped advisory event seam; step 4 wires the FOB reflex loop to act on them. The core-authoritative playout buffer from slice-2 §4.1 is already in place.
PSTN trunk / rented-transport integration	Step 5	ADR-0007's rented CPaaS raw-media fork lands the phone number; the brain is unaffected.
Spend cap / abuse gate	Step 6	The brain has no spend surface yet — OpenAI bills the operator directly. In-boundary spend pacing lands with rutster-spend in step 6.
Multi-brain routing / per-tenant brain selection	Step 6	One tap URL per call in slice-3 (env + per-call override). The slice-3 brain process is OpenAI-Realtime-only. A Deepgram+LLM+TTS composite adapter or a self-hosted open-weights brain is a future-rung concern; the tap protocol is brain-agnostic by design.
API-key rotation / KMS integration	Step 6 + later	OPENAI_API_KEY (env) + OPENAI_API_KEY_FILE (path) is the dev posture. KMS / Vault integration lands with the real trust boundary (step 6).
TLS on the tap and on the OpenAI leg	Step 6	Slice-2 rejects wss:// URLs at session-create (deferred to step 6); the OpenAI Realtime leg is wss:// and uses rustls-native roots (no cert pinning in slice-3 — defer to step 6).
Authentication / authorization on the tap_url override	Step 6	Inherits slice-2's "no auth yet" posture.
Re-INVITE / session migration / resumability	Later	Refresh the page → new session, same as slice-1/2. OpenAI Realtime session is per-call; no in-flight call preservation across server restart.
CDR / event bus / OTel beyond per-Channel tracing spans	Step 5	Single peer + single brain; no fanout yet. Tool-call events go to logs + counters only.
Quality dashboard (containment, escalation reasons)	Capability ladder rung 3	The current build target proves agent integration, not analytics.
Audio resampling for brains using 16 kHz	Future	OpenAI Realtime uses 24 kHz mono, matching slice-1's canonical tap format. Will be needed if a future brain uses 16 kHz; the translator resamples at that time.
response.audio.delta batching optimization	Future	OpenAI sends many small delta events; the translator MAY batch into the slice-2 audio_out 20 ms frame. Optimization optional for v1; tracked.

1.3 What this slice does NOT prove

It does not prove: a real barge-in reflex (only the event seam), latency determinism under reflex timing, PSTN trunking, spending controls, multi-tenancy on the tap URL, API-key rotation, or wss:// TLS posture in production. It proves only agent integration: the OpenAI Realtime adapter as green-zone (per ADR-0008), the tap protocol extension carrying audio + interruption signals + function-call plumbing, the in-boundary tool registry dispatching hangup cleanly, and the seam test (slice-2's RtcSession accepts the new brain with zero internal change).

---

2. Workspace layout (delta on slice-2)

One new workspace member; one new module in the binary; additive extension to rutster-tap's protocol module.

rutster/
├── Cargo.toml                          # +rutster-brain-realtime in members
├── crates/
│   ├── rutster/                         # binary: +tool_registry module
│   │   ├── src/main.rs
│   │   ├── src/session_map.rs          # +tool-call side-channel drain
│   │   ├── src/routes.rs               # unchanged shape
│   │   ├── src/tap_engine.rs           # UNCHANGED — the seam test
│   │   ├── src/tool_registry.rs        # NEW: Tool trait, hangup tool
│   │   └── static/index.html           # minor: surface brain connection status
│   ├── rutster-media/                   # UNCHANGED — the seam test
│   │   └── src/loop_driver.rs          # UNCHANGED
│   ├── rutster-call-model/              # UNCHANGED
│   ├── rutster-tap/                     # +additive protocol event types
│   │   └── src/protocol.rs             # +speech_started/stopped, function_call/output, tools.update
│   ├── rutster-tap-echo/                # UNCHANGED (still works against extended protocol)
│   ├── rutster-brain-realtime/          # NEW crate
│   │   ├── Cargo.toml                  # deps: rutster-tap, tokio, tokio-tungstenite, serde_json, tracing, url
│   │   ├── src/lib.rs                  # lib + MockRealtimeBrain for tests
│   │   ├── src/main.rs                 # standalone binary: ws://127.0.0.1:8082/realtime + OpenAI WS client
│   │   ├── src/translator.rs           # tap ⇄ OpenAI event translation
│   │   └── src/openai_client.rs         # wss://api.openai.com/v1/realtime client
│   ├── rutster-trunk/                   # STUB (unchanged)
│   └── rutster-spend/                   # STUB (unchanged)
└── examples/
    └── echo_brain/                      # unchanged (Python reference; ignores new events)

2.1 Dependency direction

• rutster-brain-realtime → rutster-tap (for protocol types — same re-export pattern slice-2 established for rutster-tap-echo).
• rutster-brain-realtime is its own workspace member that's both a binary (dev loop) and a library (test fixture). The library re-exports a MockRealtimeBrain for use by integration tests in the binary crate.
• rutster (binary) gains tool_registry.rs as a sibling of tap_engine.rs — the new module dispatches tool-call events the TapClient observes via the new side-channel mpsc.
• rutster-tap's protocol module is the contract both the brown core and the new brain process share; new event types added here are re-used by rutster-brain-realtime (the contract test that the wire types are reusable from outside the core, exactly as rutster-tap-echo did).
• rutster-media and rutster-call-model are untouched. loop_driver.rs and rtc_session.rs are byte-identical to slice-2 (post-review-fix) baseline.

2.2 Why one new crate for the OpenAI Realtime brain (not an examples/ file)

Mirrors slice-2's dual-purpose pattern for rutster-tap-echo: a real workspace member that runs cargo fmt, cargo clippy -D warnings, cargo test, and cargo deny check like every other crate. It reuses rutster-tap's protocol types — the contract test that the wire types are reusable from outside the core, which is what makes the tap a real extension point and not a closed system.

A Python reference brain (examples/openai_realtime_brain/openai_realtime_brain.py) is the canonical foreign-language brain demo for OpenAI Realtime specifically (slice-2 already shipped the echo brain in Python). This file is not in CI (Python would violate the zero-non-Rust-dev-deps dev loop). README-documented runnable: pip install websockets openai and run.

2.3 Why the brain is green-zone (per ADR-0008)

ADR-0008 classifies "the agent brain" explicitly under green-zone: it's not hot-path inside the boundary (its round-trip is hundreds of ms; the timing-critical work happens in the FOB reflex loop), not security-constitutive (the core-authoritative playout buffer structurally prevents the brain from flooding the wire), and not differentiating (the tap-as-open-protocol is the differentiator; any brain speaking it works). So the OpenAI Realtime adapter lives outside the FOB trust boundary — its own process, its own API key, its own failure domain. The brain is reached by the FOB; it does not live in it. The only FOB-side additions in slice-3 are:

• rutster-tap/src/protocol.rs — additive event types (protocol extension, not new behavior — the existing TapClient + TapAudioPipe surface is unchanged).
• crates/rutster/src/tool_registry.rs — in-boundary tool dispatch (a security-constitutive capability: the brain proposes tool calls; the FOB disposes. Per ADR-0007's spend-gate posture: "rutster mediates both the provider call-control API and the brain tap, so the brain never holds the wire").

---

3. Tap wire protocol extension (rutster-tap/src/protocol.rs)

The slice-2 v1 protocol (envelope: {v, type, seq, ts}). The new event types are additive — slice-2's §3.4 "unknown type → log + count + drop" rule means an old brain (slice-2's rutster-tap-echo) ignores them. Slice-2's envelope invariants (per-direction seq, advisory ts, samples: 480 for audio frames, explicit LE byte order) carry over unchanged.

3.1 Envelope (slice-2's, unchanged)

{
  "v": 1,
  "type": "<event_name>",
  "seq": <uint>,
  "ts": <uint>
}

3.2 New events — brain → core (advisory)

type	payload fields	when
speech_started	{}	brain detected user speech started (translated from OpenAI input_audio_buffer.speech_started; advisory — the core’s FOB reflex loop, when wired in step 4, may use it for barge-in)
speech_stopped	{}	brain detected user speech stopped (translated from OpenAI input_audio_buffer.speech_stopped; advisory)
tools.update	{ "tools": [{ "name": "<slug>", "description": "...", "parameters": {...} }] }	brain declares its tool catalog (sent on hello ack, re-sent on tool catalog changes). The core's tool registry uses this to validate function_call events.
function_call	{ "id": "<uuid>", "name": "<slug>", "args": { ... } }	brain wants the core to execute a tool (translated from OpenAI response.function_call_arguments.done). The core dispatches via the tool registry and replies via function_call_output.

3.3 New events — core → brain

type	payload fields	when
function_call_output	{ "id": "<uuid>", "status": "ok"|"error"|"not_implemented", "result": { ... } }	core's tool-registry reply. Translated by the brain process into an OpenAI conversation.item.create with type: function_call_output so OpenAI continues the conversation appropriately.

3.4 Invariants

• Tool-call id: function_call and function_call_output carry the same id (a UUID minted by the brain; OpenAI calls it call_id, we translate verbatim). Mismatches are logged + counted; the tool-registry dispatch keys off the id.
• Tool validation: the core's tool registry validates function_call events against the most recent tools.update catalog. An unknown tool name → function_call_output with status: "not_implemented" (not error; distinguishing "the FOB doesn't know this tool" from "the tool failed"). The catalog is brain-authoritative — the brain declares it; the core merely checks dispatch.
• Advisory interruption events: speech_started / speech_stopped are advisory only in slice-3. The core logs + counts them; the FOB reflex loop lands in step 4 and will use them. No slice-3 code path acts on them in the hot media loop — loop_driver.rs is byte-identical to slice-2 (post-review-fix) baseline.
• Forward-compat: unknown type values continue to be logged + counted + dropped (slice-2 §3.4). The echo brain from slice-2 ignores every new type this slice introduces; it echoes audio unchanged.

3.5 Byte-order + audio format (unchanged from slice-2)

PCM inside the base64 payload is explicit little-endian i16 bytes (v1 wire contract, slice-2 §3.4). OpenAI Realtime's audio events also use 24 kHz mono PCM inside base64 LE i16 — no transcoding, no resample, no endianness swap. The translator passes the audio payload through to the WS frame verbatim. This is a happy convergence that simplifies the translator; it is not a coincidence (slice-1's choice of 24 kHz mono was made with this ecosystem in mind).

---

4. The OpenAI Realtime translation layer (rutster-brain-realtime/src/translator.rs)

Pure (no schema of its own beyond the new tap event types); maps event names + payload shapes between the two wire protocols. Owns no call state beyond the OpenAI Realtime session/connection state.

4.1 The two-leg brain process

                WS server side                        WS client side
        (core-as-client dials here)         (brain-as-client dials OpenAI)
                  ┌──────────────────┐                              ┌──────────┐
   tap protocol   │  rutster-brain-  │     translation layer         │  OpenAI  │
   core ◄────────┤  realtime         ├────────────────────────────►  │ Realtime │
  (audio_in/out,  │                  │   tap event ⇄ OpenAI event   │  API WS  │
  speech_started, │                  │                              │ (wss://api.openai.com)│
  func_call, ...) └──────────────────┘                              └──────────┘
                        │
                        │ brain process also owns:
                        │  • OpenAI session.update (turn_detection: null — S4)
                        │  • interruption-signal forwarding (OpenAI → tap)
                        │  • function_call forwarding (bidirectional)
                        │  • tools.update on startup (catalog to core)

The translation layer is pure (no schema of its own beyond the new tap event types); it maps event names + payload shapes between the two wire protocols. It owns no call state beyond OpenAI's session/connection state.

4.2 Event mapping table

Tap protocol	OpenAI Realtime	Notes
hello	session.update (modalities: ["audio", "text"], audio config, turn_detection: null)	Sent on handshake + reconnect; the S4 decision is encoded here
audio_in (base64 PCM LE i16)	input_audio_buffer.append (base64, same wire shape)	Pass-through; no transcoding
audio_out (from brain → core, advisory)	response.audio.delta events	The brain receives, formats, sends to core via tap_audio_out mpsc
speech_started (brain → core advisory)	input_audio_buffer.speech_started	Caught and forwarded as advisory
speech_stopped (brain → core advisory)	input_audio_buffer.speech_stopped	Caught and forwarded as advisory
function_call (brain → core)	response.function_call_arguments.done (OpenAI emits) → translator formats as tap function_call	Brain's translator extracts call_id, name, arguments; the core dispatches via tool registry
function_call_output (core → brain)	conversation.item.create with item.type: function_call_output, call_id: <id>, output: <result-json>	Closes the tool-call loop
bye / session_end	session.delete + WS close	Graceful teardown

4.3 The S4 turn-ownership decision (load-bearing per ADR-0008)

ARCHITECTURE.md is right that the core should be authoritative on VAD/barge-in ("the tap carries the results of reflexes, not the responsibility"; "AudioOut advisory / core-authoritative"). But the most likely first brain — OpenAI Realtime — does its own server-side VAD and turn detection by default. Integrating step 3–4 means either: disable the brain's turn detection and feed it clean, locally-detected turns (preserves the reflex-authoritative principle, but is more integration work and fights the API's grain), or accept split-brain turn-taking where local VAD and the brain's VAD can disagree (double-triggers, dropped barge-ins).

Decision implied: make "who owns turn detection" an explicit decision in the step-3 (brain) design doc, defaulting to core-authoritative, brain VAD off, with the integration cost budgeted. Don't let it be discovered at wiring time. — Claude, vision-sanity-check S4

ADR-0008's FOB/green-zone split makes this decidable without ambiguity:

• The reflex loop is FOB (hot-path + differentiating — turn-taking, VAD-driven barge-in, jitter, pacing).
• The brain is green-zone (per ADR-0008's explicit classification).
• The tap is core-authoritative (slice-2 §4.1 — the playout buffer structurally prevents the brain from gating playout).

The only doctrine-consistent posture is: OpenAI Realtime's server-side turn detection is disabled (session.update with turn_detection: null), and the FOB owns turn-taking. The speech_started / speech_stopped events are forwarded as advisory signals so step 4 can use them as one input to the FOB reflex loop — but OpenAI does not gate playout, ever. The core-authoritative playout buffer (slice-2 §4.1) is the only thing that gates playout in slice-3 (and continues to be in step 4).

Why this is the right call now (rather than deferring to step 4): leaving OpenAI's turn detection on would mean slice-3's demo "works by accident" — the slice-4 barge-in integration would then have to disable it mid-flight, with unpredictable behavioral changes. By nailing it down at slice-3's session.update, the brain process never wires up the wrong posture and step 4's barge-in work composes cleanly on top.

4.4 Failure mode + reconnect

• Tap-side WS reconnect (brain unreachable): slice-2's TapEngine::spawn_tap_engine already does bounded-backoff reconnect (250 ms → 500 ms → 1 s → 2 s → cap 5 s, infinite retries, re-hellos with the same session_id). The brain process being unreachable looks identical to slice-2's echo brain being unreachable — the slice-2 reconnect path is unchanged. The seam test (slice-2 §8.5 #6) holds: tap_engine.rs is byte-identical.
• OpenAI-side WS failure: the brain process's own concern — it enters its own bounded-backoff reconnect to OpenAI, and forwards a tap error event ({ "code": "brain_upstream_disconnect", "message": "..." }) to the core as an FYI. The core logs + counts; the call stays up; the tap-side playout goes silent (slice-2's underflow path) until OpenAI reconnects. The slice-3 brain's OpenAI-side reconnect is independent of the slice-2 tap-side reconnect — two distinct failure surfaces.
• API-key invalid (OpenAI returns 401): the brain process emits a tap error event ({ "code": "brain_auth_failed", "message": "..." }) and exits the OpenAI leg. The core's response is unchanged from the upstream-disconnect case (logs + silence + reconnect path). Operator intervention; the call stays up.
• Tool-registry dispatch failure: a tool-registry error (e.g., hangup fires but the channel state machine rejects the transition because the call is already Closing) returns a function_call_output with status: "error" and a result body explaining. The brain forwards the error to OpenAI; OpenAI may or may not retry depending on its own logic. No core-side retry; the model gets one chance.

---

5. Lifecycle & integration

5.1 Session lifecycle (slice-3 delta on slice-2)

1. POST /v1/sessions — body still optionally carries tap_url. If absent, falls back to RUTSTER_TAP_URL env. Default now documented as either ws://127.0.0.1:8081/echo (slice-2 Rust echo brain) or ws://127.0.0.1:8082/realtime (slice-3 OpenAI Realtime brain); the operator picks which brain to run.
2. POST /v1/sessions/:id/offer — unchanged from slice-1/2; SDP answer.
3. On Connected: the binary's poll task observes the state transition
   • spawns the TapEngine task (unchanged from slice-2). The TapEngine connects to the brain process's WS server.
4. Brain process startup: connects to wss://api.openai.com/v1/realtime, sends session.update with turn_detection: null, waits for OpenAI session.created ack.
5. Brain process WS server accepts the core's tap WS; hello exchange; brain process sends tools.update with the catalog (currently: hangup only, plus any tool schemas the brain process's startup config declares — see §6).
6. Audio flows: core decodes Opus → PCM → audio_in to brain → translator formats as input_audio_buffer.append → OpenAI processes → response.audio.delta → translator formats as audio_out → core's playout ring → str0m encode.
7. Interruption signals: OpenAI input_audio_buffer.speech_started / .speech_stopped → translator → tap speech_started / speech_stopped → core logs + counts (advisory). Step 4 will wire these into the FOB reflex loop.
8. Tool calls: OpenAI response.function_call_arguments.done → translator → tap function_call → core's tool_registry dispatches (via the TapClient's new side-channel mpsc) → function_call_output reply → translator → OpenAI conversation.item.create.
9. DELETE /v1/sessions/:id or peer-close → Closing: slice-2's unmodified teardown sequence fires — the TapEngine's session_end → brain process's session.delete to OpenAI → WS close on both legs.

5.2 The tool-call side-channel (the only brown-binary wiring)

Slice-2's TapClient runs the WSS pump loop with two mpsc channels (tx_pcm_in for inbound audio to brain, rx_audio_out for outbound audio from brain). Slice-3 adds a third:

• tx_function_call: mpsc::Sender<FunctionCallEvent> — the TapClient emits a FunctionCallEvent whenever it observes a tap function_call message on its inbound stream. The binary's poll task drains this in the same cycle it drains the existing flush_tx side-channel (slice-2 §5.3 step 4) — same pattern, one extra channel.
• rx_function_call_output: mpsc::Receiver<FunctionCallOutputEvent> — the binary writes function_call_output events through this channel; the TapClient picks them up and sends them as tap WS frames.

The shape keeps the seam test (§7 #5 of done criteria below): all this lives in the binary's tap_engine.rs callers + the new tool_registry.rs module. loop_driver.rs and rtc_session.rs are untouched.

5.3 Brain process startup configuration

Env var	Purpose	Default
RUTSTER_TAP_BIND	WS server bind addr for the brain process	127.0.0.1:8082
OPENAI_API_KEY	OpenAI Realtime API key (mutually exclusive with _FILE)	required unless --features=mock
OPENAI_API_KEY_FILE	Path to a file containing the API key	optional; overrides _KEY
OPENAI_REALTIME_MODEL	OpenAI Realtime model id	gpt-4o-realtime (or current equivalent)
OPENAI_REALTIME_VOICE	Voice for TTS output	alloy (OpenAI's default; documented)
RUTSTER_BRAIN_TOOLS	Comma-separated tool names the brain should advertise in tools.update	hangup (only tool the core wires; others cause not_implemented replies)

The dev mode (--features=mock) doesn't read OPENAI_API_KEY and instruments an in-process mock OpenAI Realtime WS server that generates canned response.audio.delta events on input_audio_buffer.append and asserts on the session.update having turn_detection: null. Used by: the slice-3 integration test (no real OpenAI credentials, no network calls), and the offline dev loop.

---

6. The tool registry (crates/rutster/src/tool_registry.rs)

A FOB-internal dispatch over the brain's proposed tool calls. The brain proposes (via function_call); the FOB disposes (via function_call_output). Per ADR-0007's spend-gate posture: the brain never holds the wire, structurally.

6.1 Trait + registry shape

#[async_trait]
pub trait Tool: Send + Sync {
    /// Tool name as the brain will reference it in function_call events.
    fn name(&self) -> &str;
    /// JSON-schema description the registry sends to the brain on tools.update.
    fn schema(&self) -> serde_json::Value;
    /// Execute the tool; return a result that the registry will serialize
    /// into the function_call_output payload.
    async fn call(&self, args: serde_json::Value) -> ToolResult;
}

pub enum ToolResult {
    Ok(serde_json::Value),
    Error(String),
    NotImplemented,
}

pub struct ToolRegistry {
    tools: Vec<Box<dyn Tool>>,
}

impl ToolRegistry {
    pub fn new() -> Self;
    pub fn register(&mut self, tool: Box<dyn Tool>);
    /// Dispatch by tool name; returns ToolResult::NotImplemented if unknown.
    pub async fn dispatch(&self, name: &str, args: serde_json::Value) -> ToolResult;
    /// Used on startup + tools.update: enumerate the registered catalog.
    pub fn catalog(&self) -> Vec<serde_json::Value>;
}

6.2 The hangup tool (the only wired impl in slice-3)

A Tool impl that on call:

1. Looks up the ChannelId (passed via the dispatch context — the tool_registry is keyed by ChannelId, one registry per active channel).
2. Fires AppState::close(channel_id).await — the existing slice-2 teardown path (sets Closing, sends session_end, aborts the engine).
3. Returns ToolResult::Ok({"channel_state": "Closing"}) so the brain gets confirmation.

Other tool names (anything in RUTSTER_BRAIN_TOOLS the brain advertises that isn't hangup) → the registry returns ToolResult::NotImplemented: the brain process forwards the reply as an OpenAI conversation.item.create with the not_implemented status, and the model is free to continue.

6.3 What slice-3 does NOT wire

• No transfer tool (escalation rung 2 — separate slice).
• No lookup / CRM-integration tools (business-logic integrations are green-zone + future-rung).
• No spend/abuse-triggered tools (step 6).
• No dynamic tool registration at runtime (the catalog is fixed at startup from RUTSTER_BRAIN_TOOLS).

If a future brain proposes a tool not in the registry: not_implemented reply; the model is free to retry with different arguments or give up. The registry's job is to be the auditable dispatch boundary, not to be extensible mid-call.

---

7. CI, dev loop, testing

7.1 New [workspace.dependencies] (Cargo.toml)

• async-trait = "0.1" (for the Tool trait's async fn).
• tokio-tungstenite, serde_json, tracing, url — already pulled by slice-2; reused.

Member crate rutster-brain-realtime references these with dep.workspace = true.

7.2 CI (.github/workflows/ci.yml)

Unchanged structure: cargo fmt --check, cargo clippy -D warnings, cargo test --all, cargo deny check. The new rutster-brain-realtime crate joins --all. No Python in CI — the OpenAI Realtime Python reference brain is README-documented only. The Rust rutster-brain-realtime's in-process MockRealtimeBrain powers the integration test; no real OpenAI credentials needed.

7.3 Dev loop

• cargo run -p rutster-brain-realtime --features=mock → starts the brain process with an in-process mock OpenAI Realtime (on :8082): no API key required, no network calls to OpenAI.
• cargo run -p rutster-brain-realtime (with OPENAI_API_KEY set) → starts the brain process with the real OpenAI Realtime.
• cargo run (or cargo run -p rutster) → starts the axum signaling server on 0.0.0.0:8080, dials out to $RUTSTER_TAP_URL on each session.
• Browser → http://localhost:8080/ → click "Start call" → grant mic → speak → hear AI reply through the brain process.
• RUST_LOG=rutster=debug cargo run for verbose tracing including tap + tool-call events.
• --features=echo on the binary (slice-2 inherited) → bypasses the brain entirely, routes audio through EchoAudioPipe.

7.4 Testing strategy

• Unit tests in rutster-brain-realtime:
  • Protocol-event translation round-trips (tap audio_in ↔ OpenAI input_audio_buffer.append; tap function_call ↔ OpenAI response.function_call_arguments.done; tap function_call_output ↔ OpenAI conversation.item.create).
  • session.update body asserts turn_detection: null (S4 decision encoded as a test).
  • tools.update serialization round-trips.
  • Error path: OpenAI-side WS error → tap error event forwarding.
• Unit tests in rutster-tap:
  • New event types (de)serialization round-trips with golden JSON fixtures in tests/fixtures/.
  • Slice-2's existing echo-brain tests stay green (the new event types are ignored by the echo brain — forwards-compat).
• Unit tests in rutster tool_registry.rs:
  • hangup tool fires AppState::close correctly.
  • Unknown tool name → ToolResult::NotImplemented.
  • Registry catalog serialization.
• Integration test in rutster binary crate: spin up the axum server (ephemeral port) + the in-process MockRealtimeBrain (ephemeral port) + set RUTSTER_TAP_URL. Drive a synthetic WebRTC peer (extending slice-2's tap_integration.rs harness): push PCM into the core via the WebRTC peer → assert audio_out flows back through the tap (the mock brain generates canned response.audio.delta on input_audio_buffer.append) → assert they're re-encoded + pushed to str0m. Plus: function-call round-trip (mock brain emits a function_call for hangup → core's tool_registry.dispatch("hangup") fires → channel state Closing → session_end over the tap → mock brain's session.delete recorded).
• Manual e2e test plan (README):
  1. cargo run -p rutster-brain-realtime --features=mock (mock brain on :8082).
  2. cargo run (core on :8080).
  3. Browser → speak → hear mock-brain reply within ~250 ms (slice-2's round-trip; no real OpenAI).
  4. Repeat with the real brain (cargo run -p rutster-brain-realtime with OPENAI_API_KEY set) → end-to-end speech-to-speech with real OpenAI Realtime within ~700 ms.
  5. Repeat with the Python brain (python examples/openai_realtime_brain/openai_realtime_brain.py) → should also work (proves the protocol is language-agnostic, same as slice-2's Python echo brain).
  6. Function-call: trigger a model behavior that emits a tool call (e.g. a test prompt that says "please hang up") → assert the call closes + session_end over the tap + session.delete to OpenAI.
  7. cargo test --all green; cargo fmt --check / cargo clippy -D warnings / cargo deny check green.

7.5 Slice 3 "done" criteria

The slice is complete when, on a clean checkout (+ open pivot / slice-1 review fixes / slice-2 plan-rebaseline PRs merged):

1. cargo test --all passes (unit + integration). The new rutster-brain-realtime crate tests green alongside slice-1/2's suite.
2. cargo fmt --check, cargo clippy -D warnings, cargo deny check all pass.
3. cargo run -p rutster-brain-realtime --features=mock + cargo run → browser, speak, hear mock-brain reply within ~250 ms.
4. With OPENAI_API_KEY set: real OpenAI Realtime end-to-end within ~700 ms.
5. Both rutster-brain-realtime (Rust, with --features=mock) and examples/openai_realtime_brain/openai_realtime_brain.py (Python) successfully interop against the core — proves the extended protocol is language-agnostic.
6. The seam test (load-bearing): rutster-media's loop_driver.rs and rtc_session.rs keep their media-loop trait-method call sites byte-identical to slice-2 (post-review-fix) baseline. The only brown-binary additions are: crates/rutster/src/tool_registry.rs (new module) + a new side-channel drain in session_map.rs. The tap_engine.rs and TapClient get only additive new mpsc channels (one new sender + one new receiver); the WSS pump loop's structure is unchanged. A git diff v<slice-2-post-review-fix-tag> -- crates/rutster-media/src/loop_driver.rs crates/rutster-media/src/rtc_session.rs shows no behavior-changing hunks (doc-comment or import changes permitted).
7. S4 turn-ownership test: the integration test asserts that the session.update sent to OpenAI Realtime (or its mock equivalent) contains turn_detection: null. The brain never auto-barges; the core-authoritative playout buffer is the only playout gate.
8. LEARNING.md grows ≥3 new pointers: async-trait patterns → crates/rutster/src/tool_registry.rs; OpenAI Realtime adapter → crates/rutster-brain-realtime/src/translator.rs; tap protocol extension + forwards-compat → crates/rutster-tap/src/protocol.rs.

---

8. Open decisions (tracked)

• response.audio.delta batching. OpenAI sends many small delta events per response; the translator MAY batch into the slice-2 audio_out 20 ms frame, or pass each delta through immediately as its own audio_out. Batching reduces WS round-trips but adds latency. Track + revisit after latency measurement with the real brain.
• Tool-registry extensibility. Slice-3 fixes the catalog at startup via RUTSTER_BRAIN_TOOLS. Runtime-extensible registration (new tools added mid-call) is a future-rung concern — would need a tool-register API event. Track.
• API-key rotation. Slice-3 reads the key once at startup; a key rotation requires a brain-process restart. KMS / Vault integration lands with the real trust boundary (step 6). Track.
• Voice + persona selection. OPENAI_REALTIME_VOICE is the only persona knob now; future brain processes may carry full prompt / system-message customization (rung 2+). Track.
• Multi-brain routing. Slice-3 ships one brain process: OpenAI Realtime. A Deepgram+LLM+TTS composite adapter or a self-hosted open-weights brain would be its own crate (rutster-brain-deepgram, rutster-brain-local) — the tap protocol is brain-agnostic by design. Track + revisit when a second brain lands.

---

9. Out-of-scope re-check (against AGENTS.md + ADR-0007/0008)

Item	Status
Dedicated timing thread for media loop	Still step 4.
TLS on the HTTP signaling surface	Still step 5.
Authn / authz / multi-tenancy on /v1/sessions	Still step 6.
Trickle ICE	Unchanged.
Barge-in / VAD-driven playout kill	Step 4. Slice-3 pre-paves the speech_started / speech_stopped advisory event seam so step 4 lands cleanly.
PSTN trunk / rented transport	Still step 5.
Spend cap / abuse gate	Still step 6.
CDR / event bus / OTel beyond per-Channel tracing	Still step 5.
Browser automation / Playwright	Still post-slice-1.
Docker / compose	Still later-rung.
Transfer / park / pickup / barge (call features)	Still escalation rung 2.

If an agent proposes adding any of these in slice 3, the right answer is "no, see the slice-3 spec §1.2."

---

10. Key design decisions (summary of the brainstorming session)

Decision	Choice	Rejected alternatives	Why
Brain target	OpenAI Realtime NOW; design for multi-brain later. Adapter trait shape is the tap protocol itself, not a Rust trait.	Trait crate rutster-brain + impl crate; Deepgram+LLM+TTS now	Slice-2's protocol-as-contract framing already delivers "multi-brain later" without a speculative trait; YAGNI on abstraction. A Deepgram+LLM+TTS composite brain's friction will be in its own internal topology (composing three async WS/HTTP streams), not in trait BrainAdapter ergonomics — a trait wouldn't ease that anyway.
Cool scope	Round-trip speech + interruption signals (advisory) + function-calling plumbing	Speech only; speech + interruption; speech + interruption + function calling	Function calling is the gateway to escalation (rung 2 — hangup is the only wired tool; transfer lands later). Plumbing the FOB dispatch contract in slice-3 means step-4 barge-in composes on a clean foundation; deferring it would force step-4 to retrofit the seam.
Function-call handler	Built-in tool registry in the core (FOB)	Outbound tool-server URL (second egress); adapter handles function calls itself; plumbing only	Per ADR-0007: "rutster mediates both the provider call-control API and the brain tap, so the brain never holds the wire." The FOB disposes; the brain proposes. hangup is the only wired tool; others reply not_implemented honestly. The registry's job is the auditable dispatch boundary, not extensibility.
Turn-detection ownership (S4)	Core-authoritative, brain VAD off (session.update with turn_detection: null)	Accept split-brain (OpenAI VAD on; core-authoritative too → disagreement); defer the decision to step 4	ADR-0008 makes the FOB/green-zone split mechanical: reflex loop is FOB, brain is green-zone, playout is core-authoritative. The only doctrine-consistent choice is to disable OpenAI's server-side VAD and let the FOB own turn-taking. Nailing it down at slice-3's session.update means step-4 barge-in composes cleanly.
API-key posture	Env var + file-path override. OPENAI_API_KEY env default; OPENAI_API_KEY_FILE overrides.	Env-only; KMS-required-now	Matches slice-2's "no auth yet" stance cleanly. KMS / Vault lands with the real trust boundary (step 6). The file-path override makes secret-manager injection (k8s secrets, Vault agent) trivial when that layer exists.
Dev mode	--features=mock in-process mocked OpenAI Realtime	No dev mock (always require real OpenAI); external mock server	The slice-3 dev loop must be zero-OpenAI-credentials + zero-network-calls-to-OpenAI. The integration test uses the same mock. Same pattern as slice-2's EchoServer.
Workspace shape	One new crate rutster-brain-realtime (library + binary), mirroring rutster-tap-echo	Put the adapter in examples/; trait crate + impl crate	Slices-2's precedent. Real workspace member runs fmt/clippy/test/deny like everyone else. Reuses rutster-tap's protocol types — the contract test that the wire types are reusable from outside the core. A future second brain is its own crate, same shape.
Brain process port + protocol	ws://127.0.0.1:8082/realtime (slice-2's echo brain defaults to :8081/echo)	Single brain port with feature toggle	Two brain processes coexist (operator picks which to run via RUTSTER_TAP_URL). The default URL stays ws://127.0.0.1:8081/echo (slice-2 documented); slice-3 adds ws://127.0.0.1:8082/realtime as the documented OpenAI-brain default.
Tap protocol extension posture	Additive v1 events (not v2)	v2 protocol bump; OpenAI Realtime event-schema adoption	Slice-2 §3.4's "unknown type → log + count + drop" rule makes additive events free — old echo brains ignore them. No wire-format break, no version negotiation needed. A v2 binary mode (raw LE i16 over WS binary frames) remains a future-rung optimization per slice-2 §9.

---

11. References

• README.md — north star, capability ladder
• ARCHITECTURE.md §"Agent tap" — the presumptive tap shape this slice hardens against a real brain
• PORT_PLAN.md — capability checklist + thin-slice phasing; §10 "WASM demoted, agent tap is the extension point"
• ADR-0002 — fused vertical
• ADR-0007 — rent the trunk; the brain is unaffected (PSTN reaches the reflex loop via media-leg ingress, not via the tap)
• ADR-0008 — FOB / green-zone doctrine; the brain is green-zone, the reflex loop is FOB, the tap is core-authoritative (the S4 turn-ownership decision follows)
• Slice 1 — WebRTC media loopback — this slice's foundation
• Slice 2 — the agent tap — the tap interface + the TapAudioPipe seam this slice composes on
• Vision-revision spec — the pressure-test that produced the architecture
• Default UI design — the operator console design record (later-rung, not slice-3's concern)
• Vision sanity-check review S4 — the original finding that named the S4 turn-ownership decision; slice-3 §4.3 resolves it under ADR-0008