9d894da726
judge_for exists because a model scoring its own prose is a known bias and the default judge is in the evaluated set. It fell back to a single hardcoded ALTERNATE_JUDGE = "qwen3.6-35b" -- which is also in that set, so with `--judge-model qwen3.6-35b` the guard fired on qwen3.6-35b and returned qwen3.6-35b. The one case it exists to prevent was the one case it produced. It degraded silently, which is the part that matters: a self-graded score looks exactly like any other judge output, so nothing in the run would say so. Now it tries the configured judge first and then a list of alternates, taking the first from a different family, and returns None when every candidate shares the model's family -- the caller skips the task rather than recording a self-graded sample. That is the same rule score_judge already applies to a judge reply it cannot parse: no sample beats a false one. Matching stays on the family via parse_base_model_id, so a -fast row is not judged by its reasoning-on sibling either. Same weights, same conflict. Also renames the judge-task COUNT computed before the loop, which shared the name `judged` with the per-task judge result inside it. Harmless today only because the count is consumed before the loop starts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WSkcSD2Jzkxo1Kw27ecfXJ
22 KiB
22 KiB