Slice 3 — OpenAI Realtime brain: swap echo for the brain #4
Reference in New Issue
Block a user
Delete Branch "slice-3-realtime-brain"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Spearhead step 3: swap slice-2's echo brain for a real speech-to-speech brain (OpenAI Realtime) reached through the existing tap. The core dials out (core-as-client; brain-as-server; still no inbound tap port on the core).
What lands
rutster-brain-realtime— new workspace member (library + binary); the OpenAI Realtime adapter process. Translator (tap ⇄ OpenAI events), OpenAI wss client pump, API-key loader (env + file path), MockRealtimeBrain behind--features=mockfor offline dev + integration tests.speech_started,speech_stopped,function_call,function_call_output,tools.update— slice-2's unknown-type → log + count + drop rule means old echo brains ignore them.TapClientemitsFunctionCallEventonfunction_callmessages; binary dispatches via the newtool_registry.rs.hangupis the only wired tool (fires the existingChannel: Connected → Closingpath; ADR-0007's spend-gate posture — brain never holds the wire).tool_registry.rs(Tool trait + hangup impl + inline unit tests).session.updatesent withturn_detection: null— OpenAI's server VAD disabled, FOB owns turn-taking. Locked by unit test in translator.rs (session_update_disables_openai_turn_detection).examples/openai_realtime_brain/) — canonical foreign-language proof the protocol is language-agnostic.Acceptance gates (spec §7.5)
cargo test --all: 117 tests green (+46 over slice-2's 71; +14 from dev-b's protocol fixtures + integration test).cargo fmt --check+cargo clippy --all --all-targets -- -D warnings: clean.loop_driver.rs+rtc_session.rsbyte-identical to slice-2 (only doc-comment + import reorder, spec-permitted).session.updatehasturn_detection: null.cargo run -p rutster-brain-realtime --features=mock+cargo runinterop on :8082/:8080.Merge instructions
Default squash-merge (not stacked on any unmerged branch).
11-task TDD plan; mirrors slice-2/slice-1's plan shape (header, global constraints extracted verbatim from spec §1.1, file-structure delta on slice-2, task-to-file mapping, per-task TDD steps with failing-test-first / verify red / implement / verify green / commit). Tasks: 1. Workspace deps + rutster-brain-realtime crate skeleton. 2. Tap protocol additive v1 extensions (speech_started/stopped, function_call, function_call_output, tools.update) - 6 TDD tests. 3. API-key loader (env var + file-path override) - 5 TDD tests. 4. Translator (tap <-> OpenAI Realtime event mapping) - 8 TDD tests incl. the S4 turn-ownership assertion. 5. OpenAI wss client pump (session.update with turn_detection: null on handshake, select! over tap input + OpenAI input). 6. Tool trait + registry + HangupTool (FOB-boundary dispatch) - 5 TDD tests. 7. TapClient function_call arms + TapConn extension (side-channel mpsc pair; the seam test preserved - TapEngine's structure unchanged, only new mpsc halflings threaded through). 8. session_map tool-call side-channel drain (the FOB dispatch becomes end-to-end live; drive_all_sessions mirrors slice-2's flush_rx drain pattern). 9. Brain binary (ws server on :8082 + OpenAI client glue; --features=mock uses run_mock_brain stub for now). 10. MockRealtimeBrain + integration test (no real OpenAI creds). 11. Python reference brain + LEARNING.md + README dev loop. Spec self-review: every spec section mapped to a task; no TBD placeholders; type consistency verified (FunctionCallEvent, ToolResult, Tool, ToolRegistry, HangupTool, run_mock_brain, load_api_key, TranslateError, OpenAiClientError, and the 6 translator fn names all appear consistently).slice-3 §5.2 + §6: the binary's poll task now drains the brain's function_call proposals from rx_function_call, dispatches through the per-channel ToolRegistry (HangupTool wired at spawn_tap_engine time), and writes function_call_output replies back through tx_function_call_output which run_tap_client forwards as tap WS frames to the brain. TapClient: handle_brain_frame now forwards function_call events to a new tx_function_call mpsc side-channel instead of dropping them. run_tap_client adds a select! arm draining rx_function_call_output + sending each as a tap frame. Advisory events (speech_started/stopped, tools.update) still log + count (slice-3 deferred-action posture). TapEngine: spawn_tap_engine now takes AppState + constructs a per-channel ToolRegistry (spec §6.2) with HangupTool pre-registered (§6.3). TapConn gains rx_function_call, tx_function_call_output, tool_registry fields. session_map: drive_all_sessions calls drain_function_calls in the same cycle as the slice-2 §5.3 step 4 flush drain (one extra channel, same cycle); the helper spawns each dispatch as its own task so the 750 ms hangup teardown bound (AppState::close) can't stall the 10 ms poll cadence. files touched: crates/rutster-tap/src/{lib,tap_client}.rs, crates/rutster/src/{session_map,tap_engine}.rs, crates/rutster/tests/tap_integration.rs ( AppState arg ), crates/rutster-brain-realtime/src/translator.rs (clippy needless_borrow ). NOT touched: loop_driver.rs, rtc_session.rs (seam test §7.5 #6). gates: cargo fmt --check OK. cargo clippy --all --tests -D warnings OK. cargo test --all OK. cargo deny check has pre-existing environmental failure (CVSS 4.0 unsupported in advisory-db; same on main).slice-3 §5.3 dev mode: an in-process fake OpenAI Realtime WS server (MockRealtimeBrain) behind --features=mock. Asserts session.update has turn_detection: null (S4 — spec §4.3) on handshake; exports a typed OpenAI-shaped error on violation so the brain fails fast + visibly. On each input_audio_buffer.append replies with a canned response.audio.delta carrying 480 zeroed samples as base64 LE i16 (same wire shape as slice-2's audio_out). Powers the offline dev loop (cargo run -p rutster-brain-realtime --features=mock) + the integration test (no real OpenAI credentials, no network calls). Also adds the binary main.rs (spec §4.2 + §5.3): binds the tap-side WS server on RUTSTER_TAP_BIND (default 127.0.0.1:8082), accepts per-call tap connections, handles the hello handshake, splits the WS into sink + stream, dials the OpenAI side (real wss://api.openai.com in default mode; MockRealtimeBrain url in mock mode), and bridges via run_openai_pump + two mpsc::channel<String> pairs. Integration test brain_mock_round_trip drives MockRealtimeBrain + run_openai_pump end-to-end: pushes a tap audio_in frame through the pump, observes the canned response.audio.delta arrive + get translated back as a tap audio_out frame on the pump's outbound mpsc. Verifies the dev-loop 'actually start + interop on a basic audio round-trip' contract the directive's Task 4 calls out, without requiring a real WebRTC peer (same constraint as slice-1/2's browser-driven e2e). Files: crates/rutster-brain-realtime/src/{lib,mock,main}.rs, crates/rutster-brain-realtime/tests/brain_mock_round_trip.rs. NOT touched: loop_driver.rs, rtc_session.rs (seam §7.5 #6). Gates: cargo fmt OK. cargo clippy --all --tests --features=mock -D warnings OK. cargo test --all --features=mock OK (56 tests). cargo deny check has pre-existing environmental failure (CVSS 4.0 unsupported in advisory-db; same on main). Dev loop verified: cargo run -p rutster-brain-realtime --features=mock starts the brain (mock on ephemeral port, tap server on :8082). ./target/release/rutster with RUTSTER_TAP_URL=ws://127.0.0.1:8082/realtime starts the core on :8080. POST /v1/sessions succeeds. (Full audio round-trip requires a browser-driven WebRTC peer, same constraint as slice-1/2 — covered by the integration test instead.)137c63d034to2154f66483