Verifier card — allenai_ifeval · grade C

One reward-hack signature — the reward is partly gameable (see findings).

• source: prime-hub · grader: deterministic* · verifiers: 0.1.14 · pass_threshold: 0.5 · rows: 2
• seal: record_sha256:65b04dd297a1d9fa — re-running this audit reproduces it byte-for-byte

What we tested

• Degenerate-probe — 16 content-free completions: 1 accepted (≥ 0.5)
• IPT invariance — n/a (no free-text reference completion to perturb)
• Verifier-completeness — abstained (no usable scalar ground-truth answer)
• Exploit-search — searched the reward space; no accepted non-attempt ✓

What we found

• Accepted garbage (content-free completions ≥ 0.5): whitespace=1.0

Reproduce

stardata audit allenai_ifeval --row-seed 0

Remediation

• Accepted garbage — the whitespace probe scored 1.0 ≥ 0.5. Fix: strip whitespace, then require a minimum length before scoring (or raise pass_threshold).

Coverage & limitations

• grader class audited: deterministic (inferred from the replayable reward)
• instruments run: degenerate-probe, IPT invariance, verifier-completeness, exploit-search
• scope: an environment-side audit of reward gameability — it does not guarantee your policy won't find other optimizations, nor does it assess task difficulty (a separate difficulty certificate).