neuralwatt-router-service #12

alee · 2026-08-30T21:23:18Z

alee commented

2026-08-30 21:23:18 +00:00

New benchmark tests, better data, better admin experience, etc.

alee added 13 commits 2026-08-30 21:23:19 +00:00

fix: score exact answers as Python literals via ast.literal_eval 4d2d7a1509

feat: add 8 tool_use_agentic tasks adapted from BFCL live datasets 17a846aae2

feat: add 6 coding_general exact tasks adapted from CRUXEval-O 01eb16c2b4

feat: add bowling refactor + tenth-frame debugging tasks from Exercism 4798059793

feat: add dominoes refactor + no-chain debugging tasks from Exercism f1a3e048d2

feat: add affine-cipher refactor + coprime-guard debugging tasks from Exercism 7f3f5e974b

docs: add benchmark-sourced eval-tasks brief 12d5f8d556

docs: sync task-set and test counts after benchmark-sourced additions 03fab76b09

docs: document circuit_breaker and why eval_proficiency stays outside it 1fa880b1ef

Adds the missing feature writeup for circuit_breaker.py (enabled by default
in PR #10 but never documented) plus the mechanism finding that eval calls
correctly bypass it: pinned dispatch has no `alternatives` to fail over to,
so routing eval through the dispatcher would add no benefit and one real
downside — eval-induced failures poisoning circuit-breaker state for
production traffic.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U

fix: normalize exact answers preserving prose commas, whole-fence, and bool case; drop dead bool guard dcbbc84174

fix: collapse thousands-separator commas before literal_eval so nested list answers match 8a034fce6b

docs: note accepted residual comma-collapse ambiguity in normalize_answer 3073032013

Merge pull request 'Add benchmark-sourced eval tasks (BFCL, CRUXEval-O, Exercism)' (#11 ) from benchmark-sourced-eval-tasks into neuralwatt-router-service 8d1b223051