adlee-was-taken 0b85b6366c feat(eval): promote held-out set to evals/heldout.yaml ...

Copy plans/local-decision-classifier-heldout.yaml (30 tasks) to
evals/heldout.yaml unchanged, and add a loader test asserting the
file loads 30 scoreable tasks with the expected categories.

The held-out set is weak: single author, short prompts, no true
distributional shift from the training set. Treat 100% as a ceiling,
not a forecast -- it cannot measure generalization.

2026-09-28 19:56:55 -04:00

4.2 KiB

Raw Permalink Blame History

View Raw