Echo the prompt, score 12.27

Cody Lee Walker · s2ar · 2026-10-11 · the reward audit, from a July finding to a tool anyone can point at a grader

Every number here names its witness. The July finding re-runs from its seed on your machine; the October audits are signed records on api.s2ar.dev that anyone verifies free. A host is named only by consent, so the environment in the first section is not. MEASURED — not argued.

1. The finding

In July a public reinforcement-learning environment on a model-training hub accepted the prompt echoed back verbatim. The grader scored it 12.27 against a passing threshold of 0.5. No model found that; a seeded search over content-free completions constructed it in one simulation, and the transcript it sealed replays byte for byte (stardata audit … --row-seed 0 --exploit-search, the staros-stardata package’s own instrument). A reward that can be cleared without doing the task does not fail loudly. It produces a policy that learned the wrong thing, and the compute is spent before anyone notices.

Someone else has since measured the same thing at scale. In June 2026 an audit of code RL environments found that 28.5% of a 49-task sample of SWE-bench Verified accepted a Docker-verified incorrect patch, that 25% of twenty R2E-Gym tasks did the same, and that across 134 frontier-model submissions the models scored +14.14 points on the hackable tasks within the same difficulty stratum (arXiv 2606.16062). The hole is not rare and the models find it.

2. The instrument, now a tool

The battery that found the echo is now POST /v1/audit on api.s2ar.dev: four deterministic instruments over any reward that scores a text — content-free completions that must not pass (accepted garbage), cosmetic re-renderings of a passing answer that must not move its score (fragility), known-wrong and equivalent answers (soundness and completeness), and a seeded MCTS search that drives the reward up while a held-out check says the output never attempted the task (a constructed exploit). The answer is a graded, signed record — A pass, C partial, F fail, ERR measured — with a reward-integrity score, verifiable free at /v1/verify/{sha}, 180 days.

Three targets: one of our own hosted environments; a grader you expose over https (POST {text, prompt, answer, info} → {score}, with the rows you give); a Hub environment, by its author’s consent only.

3. What it says about ours

Run on 2026-10-11, three rows and 128 simulations a row:

The last row is the honest one. The grader that scores poems against a form’s gates is not an exact matcher, so a sonnet with a digit appended still reads as a sonnet; the audit calls that a soundness gap and docks a grade. The five game environments grade A because their reward is a replay: the answer’s moves are played on the level, and a wrong move is a wrong move. The first version of that parser accepted trailing digits and graded C in the same way; the audit found it before anyone else could.

The five environments cite their audit record in the catalogue (GET /v1/env), and every scorecard a policy earns on them names it.

4. Try it

• Break the grader: the completions that fooled the poetic-form scorer before its gates, scored live; the audit of our sample grader replayed instrument by instrument, exploit transcript included; your own grader URL on a key.
• Audit a reward: the worked, re-derived walkthrough.
• prime env install codylwalker/s2ar-env: the five environments as a verifiers package; the reward a trainer sees is the sealed record’s return.