8d1b223051
Merge pull request 'Add benchmark-sourced eval tasks (BFCL, CRUXEval-O, Exercism)' ( #11 ) from benchmark-sourced-eval-tasks into neuralwatt-router-service
...
Reviewed-on: #11
2026-08-30 21:22:38 +00:00
adlee-was-taken
3073032013
docs: note accepted residual comma-collapse ambiguity in normalize_answer
2026-08-30 17:17:34 -04:00
adlee-was-taken
8a034fce6b
fix: collapse thousands-separator commas before literal_eval so nested list answers match
2026-08-30 17:06:31 -04:00
adlee-was-taken
dcbbc84174
fix: normalize exact answers preserving prose commas, whole-fence, and bool case; drop dead bool guard
2026-08-30 16:36:36 -04:00
adlee-was-taken
1fa880b1ef
docs: document circuit_breaker and why eval_proficiency stays outside it
...
Adds the missing feature writeup for circuit_breaker.py (enabled by default
in PR #10 but never documented) plus the mechanism finding that eval calls
correctly bypass it: pinned dispatch has no `alternatives` to fail over to,
so routing eval through the dispatcher would add no benefit and one real
downside — eval-induced failures poisoning circuit-breaker state for
production traffic.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com >
Claude-Session: https://claude.ai/code/session_01VRQXz5SYZYVWscxS1QqF6U
2026-08-30 16:13:08 -04:00
adlee-was-taken
03fab76b09
docs: sync task-set and test counts after benchmark-sourced additions
2026-08-30 16:03:34 -04:00
adlee-was-taken
12d5f8d556
docs: add benchmark-sourced eval-tasks brief
2026-08-30 15:58:01 -04:00
adlee-was-taken
7f3f5e974b
feat: add affine-cipher refactor + coprime-guard debugging tasks from Exercism
2026-08-30 14:21:12 -04:00
adlee-was-taken
f1a3e048d2
feat: add dominoes refactor + no-chain debugging tasks from Exercism
2026-08-30 14:15:41 -04:00
adlee-was-taken
4798059793
feat: add bowling refactor + tenth-frame debugging tasks from Exercism
2026-08-30 14:10:54 -04:00
adlee-was-taken
01eb16c2b4
feat: add 6 coding_general exact tasks adapted from CRUXEval-O
2026-08-30 13:32:39 -04:00
adlee-was-taken
17a846aae2
feat: add 8 tool_use_agentic tasks adapted from BFCL live datasets
2026-08-30 13:12:43 -04:00
adlee-was-taken
4d2d7a1509
fix: score exact answers as Python literals via ast.literal_eval
2026-08-30 13:06:04 -04:00
