6e729ad670
Closes the ground-truth loop. deploy/opencode-plugin/router-outcome.js hooks tool.execute.after, watches for test and build commands, and reports pass/fail to /outcome. opencode already runs your tests; this is what makes the result reach routing. Command detection is deliberately narrow -- pytest, npm test, cargo, go, ruff, mypy, tsc and friends. A failing `ls` says nothing about model quality, and a false signal is worse than none because it trains the router on noise. Verdict comes from exit status plus text signatures for tools that exit 0 while reporting failures, with "0 failed" and "no errors" guarded against. The router being unreachable never breaks a session. Attribution is the hard part, and two assumptions failed under test. The first fingerprint design keyed on the system prompt. One real opencode run produced TWO distinct keys, because it runs several agents with different prompts -- so that fingerprint identifies AGENTS, not sessions, and would have refused every single run forever. A permanent false positive dressed as safety. The second assumption was that opencode states its project root up front. Capturing a real request showed it does not. Directory is now derived from the file paths an agent touches across the whole conversation, counting every ancestor so the shared project root wins over any one subdirectory, and stripping trailing filenames so a file is never mistaken for a directory. A single mention is not enough; corroboration is required. When a report cannot be matched by directory and more than one conversation was active in the window, /outcome answers 409 and records nothing. Refusing beats guessing: a misattributed failure penalizes a model for work it never did, and this project has already recorded false failures twice from harness bugs that took measurement to catch. The window is 120 seconds, not 30 minutes. At 30 it swept in traffic from earlier in the same work session and refused a legitimate report -- observed directly, not theorised. Tests 232 -> 243. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xTPER7K8fNyKiuqNvTCTa
100 lines
3.8 KiB
Python
100 lines
3.8 KiB
Python
"""Tests for session identity — telling concurrent clients apart.
|
|
|
|
The router has to attribute an outcome report to the right conversation
|
|
without the client cooperating. Getting this wrong is worse than getting no
|
|
report at all: a misattributed test failure penalizes a model for work it
|
|
never did, and this project has already recorded false failures twice from
|
|
harness bugs that took measurement to catch.
|
|
|
|
So the rule under test is: identify exactly when possible, refuse when not.
|
|
"""
|
|
|
|
from dispatcher import session_directory, session_fingerprint
|
|
|
|
|
|
SYS_A = "You are opencode. Project root: /home/alee/Sources/6krrt"
|
|
SYS_B = "You are opencode. Project root: /tmp/otherproj"
|
|
|
|
|
|
def _conv(system, *turns):
|
|
return [{"role": "system", "content": system}] + [
|
|
{"role": "user", "content": t} for t in turns
|
|
]
|
|
|
|
|
|
# --- fingerprints ---------------------------------------------------------
|
|
|
|
def test_two_sessions_are_distinguishable():
|
|
assert session_fingerprint(_conv(SYS_A)) != session_fingerprint(_conv(SYS_B))
|
|
|
|
|
|
def test_a_fingerprint_is_stable_as_a_session_grows():
|
|
# It must survive the conversation accumulating turns, or every message
|
|
# would look like a new session
|
|
short = session_fingerprint(_conv(SYS_A, "first"))
|
|
long = session_fingerprint(_conv(SYS_A, "first", "second", "third"))
|
|
assert short == long
|
|
|
|
|
|
def test_an_empty_conversation_has_no_fingerprint():
|
|
assert session_fingerprint([]) is None
|
|
assert session_fingerprint([{"role": "user", "content": " "}]) is None
|
|
|
|
|
|
def test_multimodal_content_still_fingerprints():
|
|
msgs = [{"role": "system", "content": [{"type": "text", "text": SYS_A}]}]
|
|
assert session_fingerprint(msgs) is not None
|
|
|
|
|
|
# --- directory extraction -------------------------------------------------
|
|
|
|
def test_a_single_mention_is_not_enough_to_claim_a_directory():
|
|
# One stray path proves nothing. Requiring corroboration keeps an
|
|
# unrelated path in a chat message from hijacking attribution.
|
|
assert session_directory(_conv("see /home/alee/Sources/6krrt once")) is None
|
|
|
|
|
|
def test_no_directory_when_the_prompt_names_none():
|
|
assert session_directory(_conv("You are a helpful assistant.")) is None
|
|
|
|
|
|
def test_only_the_opening_messages_are_scanned():
|
|
# A path mentioned deep in a long conversation is not the project root
|
|
msgs = _conv("You are an assistant.", "x", "y") + [
|
|
{"role": "user", "content": "check /tmp/some/other/path"}
|
|
]
|
|
assert session_directory(msgs) is None
|
|
|
|
|
|
# --- directory from the whole conversation --------------------------------
|
|
|
|
def test_directory_comes_from_the_files_the_agent_touches():
|
|
# opencode does not state its project root up front — verified by
|
|
# capturing a real request — but a coding agent names files inside its
|
|
# project constantly.
|
|
conv = [
|
|
{"role": "assistant", "content": "reading /tmp/proj/inventory.py"},
|
|
{"role": "user", "content": "ran /tmp/proj/test_inventory.py"},
|
|
{"role": "assistant", "content": "editing /tmp/proj/inventory.py"},
|
|
]
|
|
assert session_directory(conv) == "/tmp/proj"
|
|
|
|
|
|
def test_a_filename_is_never_mistaken_for_a_directory():
|
|
conv = [{"role": "assistant", "content": "/tmp/proj/only.py " * 3}]
|
|
assert session_directory(conv) == "/tmp/proj"
|
|
|
|
|
|
def test_the_shared_root_wins_over_any_one_subdirectory():
|
|
conv = [{"role": "assistant", "content":
|
|
"/home/alee/Sources/6krrt/routing.py "
|
|
"/home/alee/Sources/6krrt/tests/test_routing.py "
|
|
"/home/alee/Sources/6krrt/scoring.py"}]
|
|
assert session_directory(conv) == "/home/alee/Sources/6krrt"
|
|
|
|
|
|
def test_two_projects_stay_distinct():
|
|
a = [{"role": "assistant", "content": "/tmp/projA/x.py /tmp/projA/y.py /tmp/projA/z.py"}]
|
|
b = [{"role": "assistant", "content": "/tmp/projB/x.py /tmp/projB/y.py /tmp/projB/z.py"}]
|
|
assert session_directory(a) != session_directory(b)
|