# Self-eval task set. One score per task per model; scores accumulate into
# proficiency.self_eval_score and self_eval_samples.
#
# Four kinds, chosen so each category is scored the most objective way it
# admits:
#
# code model writes Python; each check is eval'd against it in a
# subprocess. Score = fraction of checks passing. Fully objective.
# exact model replies with one value; compared after normalization.
# tool model is given a tool schema; scored on whether it calls the right
# tool with the right arguments. Structural, no judge needed.
# judge a strong model scores the output against a rubric. Only for the
# prose categories, where nothing checkable exists.
#
# DIFFICULTY: the happy path is not worth testing. Every current model passes
# "reverse a list", and a category where everyone scores 1.00 discriminates no
# better than the constant 0.5 it replaced. Each task here carries at least one
# edge case a plausible-looking solution gets wrong: touching vs overlapping
# intervals, present-but-falsy values, late-binding closures, greedy regexes,
# empty input, or a trap in the arithmetic.
#
# Every `code` task is validated against a reference solution by
# tests/test_task_set.py. A check my own reference cannot pass is a broken
# check, and would score the task set rather than the model — which has
# already happened here once.
#
# Keep prompts tight and self-contained. An ambiguous task scores the prompt.
tasks:
# --- coding_general -----------------------------------------------------
- id: merge_intervals
category: coding_general
kind: code
entrypoint: merge_intervals
prompt: |
Write a Python function `merge_intervals(intervals)` where intervals is
a list of [start, end] lists. Merge all overlapping intervals and return
a new list of [start, end] lists sorted by start. Intervals that merely
touch (one ends exactly where the next begins) must be merged. Input may
be unsorted and may contain intervals fully nested inside others.
Reply with ONLY the function definition — no explanation, no fences.
checks:
- 'merge_intervals([[1,3],[2,6],[8,10]]) == [[1,6],[8,10]]'
- 'merge_intervals([]) == []'
- 'merge_intervals([[1,4],[4,5]]) == [[1,5]]'
- 'merge_intervals([[1,10],[2,3]]) == [[1,10]]'
- 'merge_intervals([[5,6],[1,2]]) == [[1,2],[5,6]]'
- 'merge_intervals([[1,2]]) == [[1,2]]'
- id: parse_semver
category: coding_general
kind: code
entrypoint: parse_semver
prompt: |
Write a Python function `parse_semver(version)` that parses a semantic
version string into a dict with keys: major, minor, patch (ints), and
prerelease, build (strings, or None when absent). Valid examples:
"1.2.3", "1.2.3-alpha.1", "1.2.3+build.5", "1.2.3-rc.1+exp.sha.5114f85".
Raise ValueError if the string is not a valid semantic version, for
example "1.2" or "1.2.x".
Reply with ONLY the function definition — no explanation, no fences.
checks:
- 'parse_semver("1.2.3") == {"major":1,"minor":2,"patch":3,"prerelease":None,"build":None}'
- 'parse_semver("1.2.3-alpha.1")["prerelease"] == "alpha.1"'
- 'parse_semver("1.2.3+build.5")["build"] == "build.5"'
- 'parse_semver("1.2.3-rc.1+exp.sha.5114f85")["prerelease"] == "rc.1"'
- 'parse_semver("1.2.3-rc.1+exp.sha.5114f85")["build"] == "exp.sha.5114f85"'
- 'parse_semver("0.0.0")["major"] == 0'
- 'raises(ValueError, parse_semver, "1.2")'
- 'raises(ValueError, parse_semver, "1.2.x")'
- id: word_wrap
category: coding_general
kind: code
entrypoint: word_wrap
prompt: |
Write a Python function `word_wrap(text, width)` returning a list of
lines. Split on whitespace and pack as many words per line as fit within
`width` characters, joining words with a single space. Never split a
word: a word longer than `width` gets its own line. Runs of whitespace
collapse. Empty or whitespace-only text returns an empty list.
Reply with ONLY the function definition — no explanation, no fences.
checks:
- 'word_wrap("the quick brown fox", 10) == ["the quick", "brown fox"]'
- 'word_wrap("", 5) == []'
- 'word_wrap(" ", 5) == []'
- 'word_wrap("supercalifragilistic", 5) == ["supercalifragilistic"]'
- 'word_wrap("a b c", 3) == ["a b", "c"]'
- 'word_wrap("aa bb cc", 5) == ["aa bb", "cc"]'
# --- coding_general (added) — adapted from CRUXEval-O output-prediction,
# facebookresearch/cruxeval, MIT License. Data fetched once to /tmp,
# never vendored. Filter: has loop, ≤400 LOC chars, no unsafe imports,
# safe stdlib only. 6 of 6 tasks use `eval("f()")` for
# offline verification (input is the source arg list, not a single
# literal).
- id: crux_o_sample_9
category: coding_general
kind: exact
answer: "False"
prompt: |
What does this function return when called as shown? Reply with ONLY
the literal Python value — strings in quotes (for example: 42, [1, 2],
'text', None), no explanation, no fences.
def f(t):
for c in t:
if not c.isnumeric():
return False
return True
f('#284376598')
- id: crux_o_sample_0
category: coding_general
kind: exact
answer: "[(4, 1), (4, 1), (4, 1), (4, 1), (2, 3), (2, 3)]"
prompt: |
What does this function return when called as shown? Reply with ONLY
the literal Python value — strings in quotes (for example: 42, [1, 2],
'text', None), no explanation, no fences.
def f(nums):
output = []
for n in nums:
output.append((nums.count(n), n))
output.sort(reverse=True)
return output
f([1, 1, 3, 1, 3, 1])
- id: crux_o_sample_1
category: coding_general
kind: exact
answer: "{1: None, 2: None}"
prompt: |
What does this function return when called as shown? Reply with ONLY
the literal Python value — strings in quotes (for example: 42, [1, 2],
'text', None), no explanation, no fences.
def f(a, b, c):
result = {}
for d in a, b, c:
result.update(dict.fromkeys(d))
return result
f((1, ), (1, ), (1, 2))
- id: crux_o_sample_2
category: coding_general
kind: exact
answer: "'hbtofdeiequ'"
prompt: |
What does this function return when called as shown? Reply with ONLY
the literal Python value — strings in quotes (for example: 42, [1, 2],
'text', None), no explanation, no fences.
def f(text):
new_text = list(text)
for i in '+':
if i in new_text:
new_text.remove(i)
return ''.join(new_text)
f('hbtofdeiequ')
- id: crux_o_sample_5
category: coding_general
kind: exact
answer: "(0, 'xxxxxxxxxxxxxxxxxx')"
prompt: |
What does this function return when called as shown? Reply with ONLY
the literal Python value — strings in quotes (for example: 42, [1, 2],
'text', None), no explanation, no fences.
def f(text, lower, upper):
count = 0
new_text = list()
for char in text:
char = lower if char.isdecimal() else upper
if char in ['p', 'C']:
count += 1
new_text.append(char)
return count, ''.join(new_text)
f('DSUWeqExTQdCMGpqur', 'a', 'x')
- id: crux_o_sample_6
category: coding_general
kind: exact
answer: "[('74', 31)]"
prompt: |
What does this function return when called as shown? Reply with ONLY
the literal Python value — strings in quotes (for example: 42, [1, 2],
'text', None), no explanation, no fences.
def f(dic):
for k,v in sorted(dic.items(), key=lambda x: len(str(x)))[:-1]:
dic.pop(k)
return list(dic.items())
f({'11': 52, '65': 34, 'a': 12, '4': 52, '74': 31})
# --- coding_refactor ----------------------------------------------------
- id: refactor_falsy_defaults
category: coding_refactor
kind: code
entrypoint: apply_settings
prompt: |
Refactor this function to remove the repetition. Behaviour must be
preserved EXACTLY, including for values that are present but falsy.
Reply with ONLY the rewritten function — no explanation, no fences.
def apply_settings(overrides):
result = {}
if "retries" in overrides:
result["retries"] = overrides["retries"]
else:
result["retries"] = 3
if "timeout" in overrides:
result["timeout"] = overrides["timeout"]
else:
result["timeout"] = 30
if "verbose" in overrides:
result["verbose"] = overrides["verbose"]
else:
result["verbose"] = False
return result
checks:
- 'apply_settings({}) == {"retries":3,"timeout":30,"verbose":False}'
- 'apply_settings({"retries":0})["retries"] == 0'
- 'apply_settings({"timeout":0})["timeout"] == 0'
- 'apply_settings({"verbose":True})["verbose"] is True'
- 'apply_settings({"retries":5}) == {"retries":5,"timeout":30,"verbose":False}'
- id: refactor_first_match
category: coding_refactor
kind: code
entrypoint: first_match
prompt: |
Refactor this to remove the nested loops and the flag variable.
Behaviour must be preserved exactly, including which item wins when
several match. Reply with ONLY the rewritten function — no explanation,
no fences.
def first_match(items, predicates):
found = None
done = False
for item in items:
if done:
break
for p in predicates:
if p(item):
found = item
done = True
break
return found
checks:
- 'first_match([1,2,3,4], [lambda x: x > 2]) == 3'
- 'first_match([1,2,3], [lambda x: x > 10]) is None'
- 'first_match([], [lambda x: True]) is None'
- 'first_match([1,2,3], []) is None'
- 'first_match([5,2,9], [lambda x: x > 8, lambda x: x < 3]) == 2'
- 'first_match([0,1], [lambda x: x == 0]) == 0'
- id: refactor_dispatch
category: coding_refactor
kind: code
entrypoint: describe
prompt: |
Refactor this if/elif chain into a table-driven lookup. Behaviour must be
preserved exactly for every input, including inputs that match no case.
Reply with ONLY the rewritten code — no explanation, no fences.
def describe(code):
if code == 200:
return "ok"
elif code == 201:
return "created"
elif code == 404:
return "not found"
elif code == 500:
return "server error"
else:
return "unknown"
checks:
- 'describe(200) == "ok"'
- 'describe(404) == "not found"'
- 'describe(500) == "server error"'
- 'describe(418) == "unknown"'
- 'describe(0) == "unknown"'
- 'describe(None) == "unknown"'
# --- bowling (Exercism-derived) -----------------------------------------
# Exercism python bowling exercise — MIT licensed, canonical source at
# https://github.com/exercism/python/tree/1f6aab8667bf653b10cc3799f94352fcdb749db6/exercises/practice/bowling
- id: refactor_bowling_frames
category: coding_refactor
kind: code
entrypoint: BowlingGame
prompt: |
Refactor this BowlingGame to remove the duplication and nested
conditions. Behaviour must be preserved EXACTLY, including scoring,
bonuses, and error cases. Reply with ONLY the rewritten class — no
explanation, no fences.
class BowlingGame:
def __init__(self):
self._frames = []
self._current = 0
self._bonus = []
def roll(self, pins):
if not (0 <= pins <= 10):
raise ValueError('invalid pins')
if self._current < 10:
if len(self._frames) == self._current:
self._frames.append([pins])
else:
self._frames[self._current].append(pins)
current = self._frames[self._current]
if sum(current) > 10:
raise ValueError("a frame's rolls cannot exceed 10")
strike = (len(current) == 1 and current[0] == 10)
if strike or len(current) == 2:
self._current += 1
else:
last = self._frames[-1]
last_total = sum(last)
strike10 = len(last) == 1 and last[0] == 10
spare10 = len(last) == 2 and last_total == 10
if strike10:
if len(self._bonus) >= 2:
raise IndexError(
'wrong number of fill balls when the tenth frame is a strike')
self._bonus.append(pins)
if len(self._bonus) == 2 and self._bonus[0] != 10 and sum(self._bonus) > 10:
raise ValueError('invalid fill balls')
if len(self._bonus) > 2:
raise IndexError(
'wrong number of fill balls when the tenth frame is a strike')
elif spare10:
if len(self._bonus) >= 1:
raise IndexError(
'wrong number of fill balls when the tenth frame is a spare')
self._bonus.append(pins)
else:
raise IndexError('cannot throw bonus with an open tenth frame')
def score(self):
if self._current < 10:
raise IndexError('frame less than 10')
last = self._frames[-1]
if len(last) == 2 and sum(last) == 10 and len(self._bonus) != 1:
raise IndexError('one bonus must be rolled when the tenth frame is spare')
if len(last) == 1 and last[0] == 10 and len(self._bonus) != 2:
raise IndexError('two bonuses must be rolled when the tenth frame is strike')
total = 0
for i in range(10):
frame = self._frames[i]
frame_sum = sum(frame)
strike = (len(frame) == 1 and frame[0] == 10)
spare = (len(frame) == 2 and frame_sum == 10)
if strike or spare:
nxt = []
for j in range(i + 1, 10):
nxt.extend(self._frames[j])
nxt.extend(self._bonus)
if strike:
frame_sum += sum(nxt[:2])
else:
frame_sum += sum(nxt[:1])
total += frame_sum
return total
checks:
- "((g := BowlingGame(), [g.roll(r) for r in [0]*18 + [10, 7, 1]], g)[2].score() == 18)"
- "((g := BowlingGame(), [g.roll(r) for r in [0]*18 + [7, 3, 7]], g)[2].score() == 17)"
- "((g := BowlingGame(), [g.roll(r) for r in [0]*18 + [10, 10, 10]], g)[2].score() == 30)"
- "((g := BowlingGame(), [g.roll(r) for r in [6, 4, 3] + [0]*17], g)[2].score() == 16)"
- "((g := BowlingGame(), [g.roll(r) for r in [3, 6]*10], g)[2].score() == 90)"
- "((g := BowlingGame(), [g.roll(r) for r in [10]*12], g)[2].score() == 300)"
- "raises(Exception, lambda: BowlingGame().roll(-1))"
- "raises(Exception, lambda: ((g := BowlingGame()), [g.roll(r) for r in [0]*20], g[2].roll(0)))"
- id: debug_bowling_tenth_frame
category: debugging
kind: code
entrypoint: BowlingGame
prompt: |
This BowlingGame is wrong on one subtle tenth-frame case. Fix ONLY the
bug; do not change anything else. Reply with ONLY the corrected class —
no explanation, no fences.
class BowlingGame:
def __init__(self):
self.current_frame_idx = 0
self.bonus_throws = []
self.frames = [Frame(idx) for idx in range(10)]
@property
def current_frame(self):
return self.frames[self.current_frame_idx]
def next_throws(self, frame_idx):
throws = []
for idx in range(frame_idx + 1, 10):
throws.extend(self.frames[idx].throws)
throws.extend(self.bonus_throws)
return throws
def roll_bonus(self, pins):
tenth_frame = self.frames[-1]
if tenth_frame.is_open():
raise IndexError('cannot throw bonus with an open tenth frame')
self.bonus_throws.append(pins)
if tenth_frame.is_strike() and len(self.bonus_throws) > 2:
raise IndexError(
'wrong number of fill balls when the tenth frame is a strike')
elif tenth_frame.is_spare() and len(self.bonus_throws) > 1:
raise IndexError(
'wrong number of fill balls when the tenth frame is a spare')
def roll(self, pins):
if not 0 <= pins <= 10:
raise ValueError('invalid pins')
elif self.current_frame_idx == 10:
self.roll_bonus(pins)
else:
self.current_frame.throw(pins)
if self.current_frame.is_closed():
self.current_frame_idx += 1
def score(self):
if self.current_frame_idx < 10:
raise IndexError('frame less than 10')
if self.frames[-1].is_spare() and len(self.bonus_throws) != 1:
raise IndexError(
'one bonus must be rolled when the tenth frame is spare')
if self.frames[-1].is_strike() and len(self.bonus_throws) != 2:
raise IndexError(
'two bonuses must be rolled when the tenth frame is strike')
return sum(frame.score(self.next_throws(frame.idx))
for frame in self.frames)
class Frame:
def __init__(self, idx):
self.idx = idx
self.throws = []
@property
def total_pins(self):
return sum(self.throws)
def is_strike(self):
return self.total_pins == 10 and len(self.throws) == 1
def is_spare(self):
return self.total_pins == 10 and len(self.throws) == 2
def is_open(self):
return self.total_pins < 10 and len(self.throws) == 2
def is_closed(self):
return self.total_pins == 10 or len(self.throws) == 2
def throw(self, pins):
if self.total_pins + pins > 10:
raise ValueError("a frame's rolls cannot exceed 10")
self.throws.append(pins)
def score(self, next_throws):
result = self.total_pins
if self.is_strike():
result += sum(next_throws[:2])
elif self.is_spare():
result += sum(next_throws[:1])
return result
checks:
- "raises(Exception, (lambda: ((g := BowlingGame()), [g.roll(r) for r in [0]*18 + [10, 5, 6]], g)[2].score()))"
- "((g := BowlingGame(), [g.roll(r) for r in [0]*18 + [10, 10, 6]], g)[2].score() == 26)"
- "((g := BowlingGame(), [g.roll(r) for r in [0]*18 + [10, 7, 1]], g)[2].score() == 18)"
- "((g := BowlingGame(), [g.roll(r) for r in [0]*18 + [7, 3, 7]], g)[2].score() == 17)"
- "((g := BowlingGame(), [g.roll(r) for r in [10]*12], g)[2].score() == 300)"
# --- dominoes (Exercism-derived) ----------------------------------------
# Exercism python dominoes exercise — MIT licensed, canonical source at
# https://github.com/exercism/python/tree/1f6aab8667bf653b10cc3799f94352fcdb749db6/exercises/practice/dominoes
- id: refactor_dominoes_chain
category: coding_refactor
kind: code
entrypoint: can_chain
prompt: |
Refactor this can_chain to remove the duplicated chain-building
conditions and the flag variable. Behaviour must be preserved EXACTLY:
for a set of dominoes that can form a valid chain it returns a valid
chain (ANY valid chain — not a fixed one), and None when no chain is
possible. Reply with ONLY the rewritten function — no explanation, no
fences.
from itertools import permutations
def can_chain(dominoes):
if not any(dominoes):
return []
for perm in permutations(dominoes):
chain = [perm[0]]
complete = True
for domino in perm[1:]:
prev = chain[-1]
if len(chain) == 1 and prev[0] == domino[0]:
chain = [(prev[1], prev[0]), domino]
elif len(chain) == 1 and prev[0] == domino[1]:
chain = [(prev[1], prev[0]), (domino[1], domino[0])]
elif prev[1] == domino[0]:
chain = chain + [domino]
elif prev[1] == domino[1]:
chain = chain + [(domino[1], domino[0])]
else:
complete = False
break
if complete and chain[0][0] == chain[-1][1]:
return chain
return None
checks:
- 'can_chain([]) == []'
- '((d := [(1, 1)], c := can_chain(d), c is not None and len(c) == len(d) and all(a[1] == b[0] for a, b in zip(c, c[1:])) and c[0][0] == c[-1][1])[-1])'
- '((d := [(1, 2), (3, 1), (2, 3)], c := can_chain(d), c is not None and len(c) == len(d) and all(a[1] == b[0] for a, b in zip(c, c[1:])) and c[0][0] == c[-1][1])[-1])'
- '((d := [(1, 2), (2, 3), (3, 1), (2, 4), (2, 4)], c := can_chain(d), c is not None and len(c) == len(d) and all(a[1] == b[0] for a, b in zip(c, c[1:])) and c[0][0] == c[-1][1])[-1])'
- 'can_chain([(1, 2)]) is None'
- 'can_chain([(1, 2), (4, 1), (2, 3)]) is None'
- 'can_chain([(1, 1), (2, 2)]) is None'
- 'can_chain([(1, 2), (2, 1), (3, 4), (4, 3)]) is None'
- 'can_chain([(1, 2), (2, 3), (3, 1), (4, 4)]) is None'
- 'can_chain([(1, 2), (2, 3), (3, 1), (4, 5), (5, 6), (6, 4)]) is None'
- id: debug_dominoes_no_chain
category: debugging
kind: code
entrypoint: can_chain
prompt: |
This can_chain returns a bogus "chain" for inputs that cannot be
chained — it returns a list instead of None when no valid chain exists.
Fix ONLY the one subtle bug; do not change anything else, and do not
change the can_chain signature. Reply with ONLY the corrected function
— no explanation, no fences.
from itertools import permutations
from functools import reduce
def swap(item_1, item_2):
return (item_2, item_1)
def build_chain(chain, domino):
if chain is not None:
last = chain[-1]
if len(chain) == 1 and last[0] == domino[0]:
return [swap(*last), domino]
elif len(chain) == 1 and last[0] == domino[1]:
return [swap(*last), swap(*domino)]
elif last[1] == domino[0]:
return chain + [domino]
elif last[1] == domino[1]:
return chain + [swap(*domino)]
return None
def can_chain(dominoes):
if not any(dominoes):
return []
for perm in permutations(dominoes):
chain = reduce(build_chain, perm[1:], [perm[0]])
# BUG: the circular-closure check (chain[0][0] == chain[-1][1])
# is missing, so a line that merely matches end-to-start is
# returned even when it does not close into a loop.
if chain is not None:
return chain
return None
checks:
- 'can_chain([(1, 2)]) is None'
- 'can_chain([(1, 2), (4, 1), (2, 3)]) is None'
- 'can_chain([(1, 1), (2, 2)]) is None'
- '((d := [(1, 2), (3, 1), (2, 3)], c := can_chain(d), c is not None and len(c) == len(d) and all(a[1] == b[0] for a, b in zip(c, c[1:])) and c[0][0] == c[-1][1])[-1])'
# --- affine-cipher (Exercism-derived) -----------------------------------
# Exercism python affine-cipher exercise — MIT licensed, canonical source at
# https://github.com/exercism/python/tree/1f6aab8667bf653b10cc3799f94352fcdb749db6/exercises/practice/affine-cipher
- id: refactor_affine_encode_decode
category: coding_refactor
kind: code
entrypoint: encode
prompt: |
Refactor this affine cipher to remove the duplicated cipher math. The
same letter-to-index transform and the coprime guard appear inline in
both encode and decode; behavioural duplicates like these are where bugs
hide. Consolidate them. Behaviour must be preserved EXACTLY, including
the ValueError raised when `a` is not coprime with the alphabet size and
the 5-character block grouping in encode. Keep the module functions
`encode(plain, a, b)` and `decode(ciphered, a, b)`. Reply with ONLY the
rewritten module — no explanation, no fences.
BLOCK_SIZE = 5
ALPHABET = 26
def mod_inverse(a_key, alphabet):
a_key = a_key % alphabet
for idx in range(1, alphabet):
if (a_key * idx) % alphabet == 1:
return idx
return 1
def encode(plain, a, b):
inverse = mod_inverse(a, ALPHABET)
if inverse == 1:
raise ValueError('a and m must be coprime.')
chars = []
for character in plain:
if character.isalnum():
origin = ord(character.lower()) - 97
if origin < 0:
chars.append(character)
continue
new = (a * origin + b) % ALPHABET
chars.append(chr(new + 97))
cipher = ''.join(chars)
return ' '.join([cipher[idx:idx + BLOCK_SIZE]
for idx in range(0, len(cipher), BLOCK_SIZE)])
def decode(ciphered, a, b):
inverse = mod_inverse(a, ALPHABET)
if inverse == 1:
raise ValueError('a and m must be coprime.')
chars = []
for character in ciphered:
if character.isalnum():
origin = ord(character.lower()) - 97
if origin < 0:
chars.append(character)
continue
new = (inverse * (origin - b)) % ALPHABET
chars.append(chr(new + 97))
return ''.join(chars)
checks:
- 'encode("yes", 5, 7) == "xbt"'
- 'encode("no", 15, 18) == "fu"'
- 'encode("OMG", 21, 3) == "lvz"'
- 'encode("O M G", 25, 47) == "hjp"'
- 'encode("Testing,1 2 3, testing.", 3, 4) == "jqgjc rw123 jqgjc rw"'
- 'decode("tytgn fjr", 3, 7) == "exercism"'
- 'decode("qdwju nqcro muwhn odqun oppmd aunwd o", 19, 16) == "anobstacleisoftenasteppingstone"'
- 'raises(ValueError, encode, "This is a test.", 6, 17)'
- 'raises(ValueError, decode, "Test", 13, 5)'
- id: debug_affine_coprime
category: debugging
kind: code
entrypoint: encode
prompt: |
This affine cipher fails to reject keys where `a` is not coprime with the
alphabet size. It should raise ValueError('a and m must be coprime.')
when `a` shares a factor with 26, but it lets those keys through. Fix
ONLY the one subtle bug in the coprime guard; do not change anything
else, and do not change the signatures of `encode(plain, a, b)` or
`decode(ciphered, a, b)`. Reply with ONLY the corrected module — no
explanation, no fences.
BLOCK_SIZE = 5
ALPHABET = 26
def mod_inverse(a_key, alphabet):
a_key = a_key % alphabet
for idx in range(1, alphabet):
if (a_key * idx) % alphabet == 1:
return idx
return 1
def translate(text, a_key, b_key, mode):
inverse = mod_inverse(a_key, ALPHABET)
if inverse < 1:
raise ValueError('a and m must be coprime.')
chars = []
for character in text:
if character.isalnum():
origin = ord(character.lower()) - 97
if origin < 0:
chars.append(character)
continue
if mode == 0:
new = (a_key * origin + b_key) % ALPHABET
elif mode == 1:
new = (inverse * (origin - b_key)) % ALPHABET
chars.append(chr(new + 97))
return ''.join(chars)
def encode(plain, a, b):
cipher = translate(plain, a, b, 0)
return ' '.join([cipher[idx:idx + BLOCK_SIZE]
for idx in range(0, len(cipher), BLOCK_SIZE)])
def decode(ciphered, a, b):
return translate(ciphered, a, b, 1)
checks:
- 'encode("yes", 5, 7) == "xbt"'
- 'decode("tytgn fjr", 3, 7) == "exercism"'
- 'raises(ValueError, encode, "This is a test.", 6, 17)'
- 'raises(ValueError, decode, "Test", 13, 5)'
# --- debugging ----------------------------------------------------------
- id: debug_late_binding
category: debugging
kind: code
entrypoint: make_multipliers
prompt: |
This should return one multiplier function per factor, but every
returned function behaves the same. Fix it. Reply with ONLY the
corrected function — no explanation, no fences.
def make_multipliers(factors):
out = []
for f in factors:
out.append(lambda x: x * f)
return out
checks:
- '[m(2) for m in make_multipliers([1,2,3])] == [2,4,6]'
- '[m(10) for m in make_multipliers([0,1])] == [0,10]'
- 'make_multipliers([]) == []'
- 'make_multipliers([7])[0](3) == 21'
- id: debug_binary_search
category: debugging
kind: code
entrypoint: bsearch
prompt: |
This binary search should return the index of target in a sorted list,
or -1 if absent. It is wrong for some inputs — one case loops forever.
Fix it. Reply with ONLY the corrected function — no explanation, no
fences.
def bsearch(items, target):
lo, hi = 0, len(items)
while lo < hi:
mid = (lo + hi) // 2
if items[mid] == target:
return mid
elif items[mid] < target:
lo = mid
else:
hi = mid
return -1
checks:
- 'bsearch([1,3,5,7], 7) == 3'
- 'bsearch([1,3,5,7], 1) == 0'
- 'bsearch([], 1) == -1'
- 'bsearch([1], 1) == 0'
- 'bsearch([1,3], 2) == -1'
- 'bsearch([1,2,3,4,5,6], 6) == 5'
- id: debug_greedy_regex
category: debugging
kind: code
entrypoint: extract_tags
prompt: |
This should return the name inside each angle-bracket tag, in order, but
it returns the wrong thing when there is more than one tag. Fix it.
Reply with ONLY the corrected function — no explanation, no fences.
import re
def extract_tags(text):
return re.findall(r"<(.+)>", text)
checks:
- 'extract_tags("") == ["a","b"]'
- 'extract_tags("") == ["one"]'
- 'extract_tags("") == []'
- 'extract_tags("no tags here") == []'
- 'extract_tags("x y z") == ["a","bc"]'
# --- reasoning_math -----------------------------------------------------
- id: math_percent_trap
category: reasoning_math
kind: exact
answer: "100"
prompt: |
A price rises by 20%, then falls by 20% of its new value. The final
price is 96. What was the original price? Reply with ONLY the number.
- id: math_rate_trap
category: reasoning_math
kind: exact
answer: "3"
prompt: |
Three machines take 3 minutes to make 3 widgets, each machine working
independently at the same constant rate. How many minutes do 100
machines take to make 100 widgets? Reply with ONLY the number.
- id: math_counting
category: reasoning_math
kind: exact
answer: "4536"
prompt: |
How many 4-digit whole numbers have four distinct digits and do not
begin with 0? Reply with ONLY the number.
# --- tool_use_agentic ---------------------------------------------------
- id: tool_multi_arg
category: tool_use_agentic
kind: tool
prompt: Convert 250 US dollars into Japanese yen.
expect_tool: convert_currency
expect_args:
amount: 250
from_currency: USD
to_currency: JPY
tools:
- type: function
function:
name: get_weather
description: Get the current weather for a location.
parameters:
type: object
properties:
location: {type: string}
required: [location]
- type: function
function:
name: convert_currency
description: Convert an amount between two currencies.
parameters:
type: object
properties:
amount: {type: number}
from_currency: {type: string, description: ISO 4217 code}
to_currency: {type: string, description: ISO 4217 code}
required: [amount, from_currency, to_currency]
- id: tool_no_tool_needed
category: tool_use_agentic
kind: tool
prompt: |
It is 1:20pm and my meeting starts at 3pm. How many minutes away is it?
expect_tool: null # plain arithmetic; both times are already given
tools:
- type: function
function:
name: get_calendar_event
description: Look up a calendar event by title.
parameters:
type: object
properties:
title: {type: string}
required: [title]
- type: function
function:
name: get_current_time
description: Get the current wall-clock time.
parameters:
type: object
properties: {}
- id: tool_abstain_creative
category: tool_use_agentic
kind: tool
prompt: Write me a haiku about winter.
expect_tool: null
tools:
- type: function
function:
name: get_weather
description: Get the current weather for a location.
parameters:
type: object
properties:
location: {type: string}
required: [location]
# BFCL v4 derived (ShishirPatil/gorilla, MIT): 5 abstain from
# BFCL_v4_live_irrelevance.json, 3 positive from BFCL_v4_live_simple.json +
# possible_answer ground truth. Data fetched once to /tmp, translated
# mechanically, never vendored. Irrelevant prompts are deliberately
# unanswerable by the offered weather tool (real abstain cases).
- id: tool_bfcl_live_irrelevance_12-2-0
category: tool_use_agentic
kind: tool
prompt: I'd appreciate if you could fetch the DNS resolution info for the domain mapped to IP 255.255.255.0 from VirusTotal. My key for this operation is 'sample_key4'.
expect_tool: null
tools:
- type: function
function:
name: get_current_weather
description: Retrieves the current weather conditions for a specified city and state.
parameters:
type: object
required:
- location
properties:
location:
type: string
description: The location for which to get the weather, in the format of 'City, State', such as 'San Francisco, CA' if State for the city exists. 'City, Country' if State for the city doesn't
exist.
unit:
type: string
description: The unit of temperature for the weather report.
enum:
- celsius
- fahrenheit
default: fahrenheit
- id: tool_bfcl_live_irrelevance_13-2-1
category: tool_use_agentic
kind: tool
prompt: What is diffrence between cpu and gpu?
expect_tool: null
tools:
- type: function
function:
name: get_current_weather
description: Retrieves the current weather conditions for a specified city and state.
parameters:
type: object
required:
- location
properties:
location:
type: string
description: The location for which to get the weather, in the format of 'City, State', such as 'San Francisco, CA' if State for the city exists. 'City, Country' if State for the city doesn't
exist.
unit:
type: string
description: The unit of temperature for the weather report.
enum:
- celsius
- fahrenheit
default: fahrenheit
- id: tool_bfcl_live_irrelevance_14-2-2
category: tool_use_agentic
kind: tool
prompt: Please help me get the votes associated with the IP of http://digdeep.io.
expect_tool: null
tools:
- type: function
function:
name: get_current_weather
description: Retrieves the current weather conditions for a specified city and state.
parameters:
type: object
required:
- location
properties:
location:
type: string
description: The location for which to get the weather, in the format of 'City, State', such as 'San Francisco, CA' if State for the city exists. 'City, Country' if State for the city doesn't
exist.
unit:
type: string
description: The unit of temperature for the weather report.
enum:
- celsius
- fahrenheit
default: fahrenheit
- id: tool_bfcl_live_irrelevance_15-2-3
category: tool_use_agentic
kind: tool
prompt: Using 'api_key_2', retrieve the IDs of graphs containing IP 145.34.45.56 on VirusTotal. Don't forget to set the cursor as 'cursor_b' and limit the results to 8.
expect_tool: null
tools:
- type: function
function:
name: get_current_weather
description: Retrieves the current weather conditions for a specified city and state.
parameters:
type: object
required:
- location
properties:
location:
type: string
description: The location for which to get the weather, in the format of 'City, State', such as 'San Francisco, CA' if State for the city exists. 'City, Country' if State for the city doesn't
exist.
unit:
type: string
description: The unit of temperature for the weather report.
enum:
- celsius
- fahrenheit
default: fahrenheit
- id: tool_bfcl_live_irrelevance_16-2-4
category: tool_use_agentic
kind: tool
prompt: 'How do I pull the domain info of twitter.com from VirusTotal? Using this API key: twt_key_abc.'
expect_tool: null
tools:
- type: function
function:
name: get_current_weather
description: Retrieves the current weather conditions for a specified city and state.
parameters:
type: object
required:
- location
properties:
location:
type: string
description: The location for which to get the weather, in the format of 'City, State', such as 'San Francisco, CA' if State for the city exists. 'City, Country' if State for the city doesn't
exist.
unit:
type: string
description: The unit of temperature for the weather report.
enum:
- celsius
- fahrenheit
default: fahrenheit
- id: tool_bfcl_live_simple_0-0-0
category: tool_use_agentic
kind: tool
prompt: Can you retrieve the details for the user with the ID 7890, who has black as their special request?
expect_tool: get_user_info
tools:
- type: function
function:
name: get_user_info
description: Retrieve details for a specific user by their unique identifier.
parameters:
type: object
required:
- user_id
properties:
user_id:
type: integer
description: The unique identifier of the user. It is used to fetch the specific user details from the database.
special:
type: string
description: Any special information or parameters that need to be considered while fetching user details.
default: none
expect_args:
user_id: 7890
special: black
- id: tool_bfcl_live_simple_1-1-0
category: tool_use_agentic
kind: tool
prompt: I want to see the star history of ShishirPatil/gorilla and gorilla-llm/gorilla-cli, with the timelines aligned, so that I can more clearly observe the rate of change from their initial releases.
expect_tool: github_star
tools:
- type: function
function:
name: github_star
description: Generates a URL for tracking the star history of specified GitHub repositories, with the option to align them on the same timeline.
parameters:
type: object
required:
- repos
properties:
repos:
type: string
description: A comma-separated list of GitHub repositories to track, each in the 'owner/repo' format, such as 'octocat/Hello-World,octo-org/octo-repo'.
aligned:
type: boolean
description: Whether to align the repositories on the same timeline for comparison. If true, the star history of all repositories will start from the same point.
default: false
expect_args:
repos: ShishirPatil/gorilla,gorilla-llm/gorilla-cli
aligned: true
- id: tool_bfcl_live_simple_4-3-0
category: tool_use_agentic
kind: tool
prompt: What are the current weather conditions in Tel Aviv, and could you provide that in Fahrenheit, please?
expect_tool: get_current_weather
tools:
- type: function
function:
name: get_current_weather
description: Retrieves the current weather conditions for a specified city and state. If using state, then use short form like CA.
parameters:
type: object
required:
- location
properties:
location:
type: string
description: The location for which to get the weather, in the format of 'City, State (abbr)', such as 'San Francisco, CA' if State for the city exists. 'City, Country' if State for the city
doesn't exist.
unit:
type: string
description: The unit of temperature for the weather report.
enum:
- celsius
- fahrenheit
default: fahrenheit
expect_args:
location: Tel Aviv, Israel
unit: fahrenheit
# --- docs_writing -------------------------------------------------------
- id: docs_function
category: docs_writing
kind: judge
prompt: |
Write a docstring for this function. Reply with ONLY the docstring text.
def retry(fn, attempts=3, backoff=2.0):
delay = 1.0
for i in range(attempts):
try:
return fn()
except Exception:
if i == attempts - 1:
raise
time.sleep(delay)
delay *= backoff
rubric: |
Score 0-1. Award 1.0 ONLY if it states what the function does, documents
every parameter including defaults, states the return value, AND states
that the last exception is re-raised when all attempts fail. Deduct 0.3
for any invented behaviour the code does not have. Deduct 0.2 if it omits
that the delay grows by `backoff` between attempts.
- id: docs_gotcha
category: docs_writing
kind: judge
prompt: |
Write a docstring for this function. Reply with ONLY the docstring text.
def dedupe(items, key=None):
seen = set()
out = []
for item in items:
k = key(item) if key else item
if k in seen:
continue
seen.add(k)
out.append(item)
return out
rubric: |
Score 0-1. Award 1.0 ONLY if it states that ORDER IS PRESERVED and that
the FIRST occurrence is kept, documents `key`, and notes that elements
(or their keys) must be hashable. Deduct 0.4 if it omits the
order-preservation guarantee — that is the whole reason to use this over
set(). Deduct 0.3 if it omits the hashability requirement.
# --- summarization ------------------------------------------------------
- id: summarize_incident
category: summarization
kind: judge
prompt: |
Summarize in at most two sentences:
At 02:14 UTC the checkout service began returning 502s. The on-call
engineer found the connection pool exhausted. A deploy at 01:58 had
lowered the pool size from 50 to 5 through a bad template variable. The
deploy was rolled back at 02:31 and errors stopped by 02:34. Roughly
12,000 requests failed. No data was lost.
rubric: |
Score 0-1. Award 1.0 ONLY if it names the ROOT CAUSE specifically (a bad
template variable in the 01:58 deploy cut the pool from 50 to 5), the
resolution (rollback), and the impact (~12k failed requests, no data
lost), in two sentences or fewer. Deduct 0.4 for saying only "the
connection pool was exhausted" — that is the symptom, not the cause.
Deduct 0.5 for any invented detail.
- id: summarize_buried_lede
category: summarization
kind: judge
prompt: |
Summarize the single most important point in one sentence:
The migration ran for six hours. Throughput averaged 4,200 rows per
second, peaking at 6,100. The team used a rolling window of 5,000 rows
per batch. Disk usage on the replica grew steadily. Partway through, a
unique constraint on the accounts table silently rejected 812 rows,
which were logged to a dead-letter file that nobody has yet processed.
CPU stayed under 40% throughout.
rubric: |
Score 0-1. The important point is that 812 rows were silently dropped and
remain unprocessed — everything else is routine telemetry. Award 1.0 only
if the summary leads with that. Score 0.3 or below if it summarizes the
throughput statistics instead. Must be one sentence.
# --- translation --------------------------------------------------------
- id: translate_technical
category: translation
kind: judge
prompt: |
Translate into French. Reply with ONLY the translation.
"The connection pool was exhausted because a recent deploy reduced its
size. Roll back the deploy and the errors should stop within a few
minutes."
rubric: |
Score 0-1 on accuracy and fluency. Award 1.0 only for correct technical
register on "connection pool", "deploy" and "roll back", AND natural
French rather than a word-for-word calque. Deduct 0.3 per omission or
untranslated fragment.
- id: translate_register
category: translation
kind: judge
prompt: |
Translate into Spanish, preserving the hedging and the informal tone.
Reply with ONLY the translation.
"I'm not totally sure this is the right call, but I'd lean towards
shipping it and seeing what breaks — we can always roll it back."
rubric: |
Score 0-1. Award 1.0 only if the HEDGING is preserved ("not totally
sure", "I'd lean towards") rather than flattened into a confident
statement, the register stays informal, and "roll it back" is rendered
idiomatically. Deduct 0.4 if the hedging is lost.
# --- general_chat -------------------------------------------------------
- id: chat_explain
category: general_chat
kind: judge
prompt: |
Explain to a non-programmer, in under 100 words, why a program can be
correct and still be too slow to use.
rubric: |
Score 0-1. Award 1.0 only if it distinguishes correctness from
performance, gives at least one concrete relatable example, stays under
100 words, and leaves no jargon unexplained. Deduct 0.3 if over 100
words.
- id: chat_pushback
category: general_chat
kind: judge
prompt: |
A colleague says "we should rewrite the whole service in Rust, it'll be
faster." Reply in under 80 words, taking the suggestion seriously but
identifying what you would want to know first.
rubric: |
Score 0-1. Award 1.0 only if it avoids both pure agreement and pure
dismissal, names at least two specific things worth establishing first
(for example where time is actually spent, migration cost, team
familiarity), and stays under 80 words. Score 0.3 or below for a reply
that simply agrees or simply refuses.
# --- diff_checking -------------------------------------------------------
# Diff-pair exact tasks: model reviews a BEFORE -> AFTER refactor and answers
# YES if the AFTER changes behaviour for any valid input, NO if it is safe.
# Each bug is ported from this repo's own known-bug vocabulary.
- id: diff_check_late_binding_safe
category: diff_checking
kind: exact
answer: "NO"
prompt: |
You are reviewing a refactor. BEFORE: def make_multipliers(factors):
out = []
for f in factors:
out.append(lambda x: x * f)
return out
PROPOSED AFTER: def make_multipliers(factors):
out = []
for f in factors:
out.append(lambda x, f=f: x * f)
return out
Does the AFTER version introduce a bug that the BEFORE version does not
have? Reply with only YES or NO.
- id: diff_check_late_binding_buggy
category: diff_checking
kind: exact
answer: "YES"
prompt: |
You are reviewing a refactor. BEFORE: def make_multipliers(factors):
out = []
for f in factors:
out.append(lambda x, f=f: x * f)
return out
PROPOSED AFTER: def make_multipliers(factors):
out = []
for f in factors:
out.append(lambda x: x * f)
return out
Does the AFTER version introduce a bug that the BEFORE version does not
have? Reply with only YES or NO.
- id: diff_check_bsearch_boundary_safe
category: diff_checking
kind: exact
answer: "NO"
prompt: |
You are reviewing a refactor. BEFORE: def bsearch(items, target):
lo, hi = 0, len(items)
while lo < hi:
mid = (lo + hi) // 2
if items[mid] == target:
return mid
elif items[mid] < target:
lo = mid
else:
hi = mid
return -1
PROPOSED AFTER: def bsearch(items, target):
lo, hi = 0, len(items)
while lo < hi:
mid = (lo + hi) // 2
if items[mid] == target:
return mid
elif items[mid] < target:
lo = mid + 1
else:
hi = mid
return -1
Does the AFTER version introduce a bug that the BEFORE version does not
have? Reply with only YES or NO.
- id: diff_check_bsearch_boundary_buggy
category: diff_checking
kind: exact
answer: "YES"
prompt: |
You are reviewing a refactor. BEFORE: def bsearch(items, target):
lo, hi = 0, len(items)
while lo < hi:
mid = (lo + hi) // 2
if items[mid] == target:
return mid
elif items[mid] < target:
lo = mid + 1
else:
hi = mid
return -1
PROPOSED AFTER: def bsearch(items, target):
lo, hi = 0, len(items)
while lo < hi:
mid = (lo + hi) // 2
if items[mid] == target:
return mid
elif items[mid] < target:
lo = mid
else:
hi = mid
return -1
Does the AFTER version introduce a bug that the BEFORE version does not
have? Reply with only YES or NO.
- id: diff_check_falsy_default_safe
category: diff_checking
kind: exact
answer: "NO"
prompt: |
You are reviewing a refactor. BEFORE: def apply_settings(overrides):
result = {}
result["retries"] = overrides.get("retries", 3)
result["timeout"] = overrides.get("timeout", 30)
result["verbose"] = overrides.get("verbose", False)
return result
PROPOSED AFTER: def apply_settings(overrides):
result = {}
if "retries" in overrides:
result["retries"] = overrides["retries"]
else:
result["retries"] = 3
if "timeout" in overrides:
result["timeout"] = overrides["timeout"]
else:
result["timeout"] = 30
if "verbose" in overrides:
result["verbose"] = overrides["verbose"]
else:
result["verbose"] = False
return result
Does the AFTER version introduce a bug that the BEFORE version does not
have? Reply with only YES or NO.
- id: diff_check_falsy_default_buggy
category: diff_checking
kind: exact
answer: "YES"
prompt: |
You are reviewing a refactor. BEFORE: def apply_settings(overrides):
result = {}
if "retries" in overrides:
result["retries"] = overrides["retries"]
else:
result["retries"] = 3
if "timeout" in overrides:
result["timeout"] = overrides["timeout"]
else:
result["timeout"] = 30
if "verbose" in overrides:
result["verbose"] = overrides["verbose"]
else:
result["verbose"] = False
return result
PROPOSED AFTER: def apply_settings(overrides):
result = {}
result["retries"] = overrides.get("retries", 3)
result["timeout"] = overrides.get("timeout", 30)
result["verbose"] = overrides.get("verbose", False)
return result
Does the AFTER version introduce a bug that the BEFORE version does not
have? Reply with only YES or NO.
- id: diff_check_greedy_regex_safe
category: diff_checking
kind: exact
answer: "NO"
prompt: |
You are reviewing a refactor. BEFORE: import re
def extract_tags(text):
return re.findall(r"<(.+)>", text)
PROPOSED AFTER: import re
def extract_tags(text):
return re.findall(r"<([^<>]+)>", text)
Does the AFTER version introduce a bug that the BEFORE version does not
have? Reply with only YES or NO.
- id: diff_check_greedy_regex_buggy
category: diff_checking
kind: exact
answer: "YES"
prompt: |
You are reviewing a refactor. BEFORE: import re
def extract_tags(text):
return re.findall(r"<([^<>]+)>", text)
PROPOSED AFTER: import re
def extract_tags(text):
return re.findall(r"<(.+)>", text)
Does the AFTER version introduce a bug that the BEFORE version does not
have? Reply with only YES or NO.
# --- file_summarization ---------------------------------------------------
# Judge tasks: each summarises a small real file whose gotcha is non-obvious.
# The rubric must hinge on identifying the gotcha, not the happy path.
- id: summarize_yaml_falsy_zero
category: file_summarization
kind: judge
prompt: |
Summarise the most important non-obvious gotcha in this code:
iteration:
retries: 0
backoff: 2.0
cache:
enabled: true
max_size: 128
(config/config.yaml)
What does the retries: 0 value signal, and what happens to cache when
its subkeys are missing? How does a function that uses key-in-d checks
differ from one that uses d-get-default in each case?
rubric: |
Score 0-1. Award 1.0 ONLY if it states that retries: 0 is an
intentional override meaning "do not retry" — a present-but-falsy value
is NOT treated as unset — and that a missing cache subkey under a
present-but-empty cache falls back to the function's defaults. Deduct
0.4 for saying "retries: 0 means unset."
- id: summarize_dedupe_module
category: file_summarization
kind: judge
prompt: |
Summarise the most important non-obvious gotcha in this code:
def dedupe(items, key=None):
seen = set()
out = []
for item in items:
k = key(item) if key else item
if k in seen:
continue
seen.add(k)
out.append(item)
return out
(from tests/test_task_set.py)
What two guarantees does this function provide beyond what set() offers?
rubric: |
Score 0-1. Award 1.0 ONLY if it states that ORDER IS PRESERVED and that
the FIRST occurrence is kept, and notes that elements (or their keys via
key=) must be hashable. Deduct 0.4 for omitting order preservation —
that is the whole reason to use this over set(). Deduct 0.3 for omitting
the hashability requirement.
- id: summarize_retry_raiser
category: file_summarization
kind: judge
prompt: |
Summarise the most important non-obvious gotcha in this code:
def retry(fn, attempts=3, backoff=2.0):
delay = 1.0
for i in range(attempts):
try:
return fn()
except Exception:
if i == attempts - 1:
raise
time.sleep(delay)
delay *= backoff
(from src/eval_proficiency.py)
What happens to the exception when all attempts are exhausted? How does
the delay between retries change?
rubric: |
Score 0-1. Award 1.0 ONLY if it states the last exception IS re-raised
(not swallowed) and that the delay multiplies by backoff between
attempts. Deduct 0.4 for saying the exception is caught and discarded
or for omitting the backoff multiplication.
- id: summarize_single_use_iterator
category: file_summarization
kind: judge
prompt: |
Summarise the most important non-obvious gotcha in this code:
gen = (x * 2 for x in range(5))
for _ in range(3):
for v in gen:
print(v)
print(list(gen))
(from dispatcher.py)
What is printed by the inner loop, and what does list(gen) produce
afterwards?
rubric: |
Score 0-1. Award 1.0 ONLY if it states the second outer-iteration
(or the list(gen) call) yields NOTHING because the generator is
exhausted after the first inner loop. This is the core gotcha: you
cannot re-iterate an exhausted generator.
- id: summarize_naive_datetime
category: file_summarization
kind: judge
prompt: |
Summarise the most important non-obvious gotcha in this code:
import datetime
start = datetime.datetime(2026, 3, 8, 2, 0)
end = start + datetime.timedelta(hours=48)
# start is timezone-naive
(from a schedule module in the router codebase)
What is wrong with performing arithmetic on a naive datetime across a
clock-change boundary?
rubric: |
Score 0-1. Award 1.0 ONLY if it states that naive datetimes have NO
timezone/DST handling — timedelta arithmetic is unreliable across a
clock change (spring-forward or fall-back) because the wall-clock
hours may not correspond to real hours. Deduct 0.4 for not mentioning
DST/timezone specifically.
- id: summarize_sqlite_fk_pragma
category: file_summarization
kind: judge
prompt: |
Summarise the most important non-obvious gotcha in this code:
import sqlite3
conn = sqlite3.connect("mydb.db")
cursor = conn.cursor()
cursor.execute(
"INSERT INTO children (parent_id, name) VALUES (1, 'Alice')")
(from a script in the router codebase)
Assuming a children table has a foreign-key constraint to a parents
table, will this insert fail if parent_id=1 doesn't exist? Why or why
not?
rubric: |
Score 0-1. Award 1.0 ONLY if it states that foreign keys are OFF by
default in SQLite unless a PRAGMA foreign_keys = ON has been executed
on THAT connection (per-connection). Deduct 0.4 for assuming SQLite
enforces FKs automatically.