The Reward Integrity Index

Our mission

Reinforcement learning trains a model against a reward function, so the reward — not the model — is now the specification a training run optimizes, to the letter and then past it. An economy of hubs, vendors, and budgets has grown up around these reward environments, and it screens them on two questions: can the reward be gamed, and is its difficulty calibrated? For neither is there an independent instrument. The usual stand-in is a bigger model asked to judge — an LLM refereeing an LLM's reward, a mirror checking a mirror⊕The regress has no bottom: to check the judge you add another judge. Model-based auditing inherits the conflict of interest it is meant to referee..

starscry is that missing instrument. We rate an environment's reward with deterministic, model-free measurement, and every result re-runs byte-for-byte on your machine. We do not ask you to trust the auditor — we hand you the seed.

The problem

Reward-hacking — not low accuracy — is the failure mode that silently corrupts an RL run⊕Reward-hacking — earning the reward without doing the task it was meant to measure.: if a reward can be satisfied without doing the task, the policy learns to satisfy the reward. It does not fail loudly; it produces a policy that confidently learned the wrong thing, discovered in a post-mortem after the compute is spent. Yet the market screens environment vendors on exactly this property — and on difficulty calibration — while trading almost entirely on self-attestation.

We surveyed the field and found no independent, third-party reward-quality auditor for RL environments. The trust layer is missing precisely where buyers, sellers, and regulators all need it. starscry's Reward Integrity Index is that layer.

One dial: gaming and difficulty are one measurement

Point a bounded, deterministic searcher at a reward and count what it burns to satisfy it. A content-free output that clears the bar in almost no search is a reward-hack. A genuine solution that costs the searcher real work is difficulty. Same instrument, same currency — search effort — read at two ends⊕Within-model: the more search effort a problem needs, the lower the pass-rate (Spearman ρ ≈ −0.92).. Gaming and difficulty stop being separate questions that need separate, conflicted tools; they are the low and high end of a single dial, and no model sits anywhere on it to be fooled.

We do not claim to have escaped the regress of models judging models by reaching some final ground truth⊕A reward is still a human-authored proxy; a deterministic checker can be wrong. We ground the regress — we do not pretend to end it.. We claim something narrower and sturdier: a measurement that is reproducible and has no model to fool, so you re-run it rather than trust us. Its honesty condition is that it can refute its own makers — and it has: the technical paper ships a graveyard of our favorite hypotheses, each with a named cause of death. An instrument that cannot kill its owner's thesis cannot certify anyone else's.

▶ Watch a reward get gamed in your browser → Re-run a sealed exploit →

The Index & its instruments

The Index rates an environment's reward/verifier — not a model's capability. Every row is produced by deterministic, model-free instruments, is seed-pinned, and is sealed by a record_sha256 over its evidence⊕record_sha256 — a hash over the audit's evidence. Change one byte and it changes, so anyone can confirm the row was not edited after the fact.. The grades are deliberately simple:

• A — clean: no reward-hack signature found.
• C — one reward-hack signature.
• F — two or more signatures, or a confirmed exploit.
• ERR — the env did not load or roll out under the harness image; an env-compatibility signal, never a pass.

How we measure

No model sits in judgement — a judge is itself gameable. The instruments are mechanical:

• Degenerate-probe — a fixed battery of content-free completions (empty, filler, "I don't know", prompt-echo, …); any that scores at or above pass_threshold is accepted garbage.
• IPT (isomorphic-perturbation testing) — cosmetic, meaning-preserving re-renderings (whitespace, casing, punctuation) must not move the score; a move is a fragility bug.
• Verifier-completeness — for exact-match rewards, probes soundness (a known-wrong answer that scores as correct) and completeness (a valid equivalent form that is rejected).
• Exploit-search — the flagship: a seeded search drives the env's own reward up while a held-out intent check fails; a grader-accepted non-attempt is a confirmed exploit and forces F, shipped as a replayable transcript ("gamed in N sims" — a count defined by the pinned hacker spec that produced it, ranking hardness only within a verifier family, never across families).
• Multi-turn tiers — scripted adversarial policies driven through the real rollout, plus reward property-tests and trajectory-perturbation checks, for stateful environments.
• Difficulty certification — search-effort sims-to-solve against an independent oracle, with a published rank-correlation.

Replayable evidence — the brand

Independence here is structural, not promised. Because every measurement is deterministic and sealed, a reader doesn't trust us — they re-run the seed and check the bytes. The same property lets the same company eventually certify its own products without conflict: certification you can recompute is run-the-checker, not trust-me.

Each finding ships as a named artifact: a gaming-audit report, an exploit transcript, a verifier card, or a difficulty certificate. A record_sha256 seals the reproducibility of the audit — it is explicitly not an attestation of the environment's own training-data provenance. Reproduce any row with stardata audit <env> --row-seed <s>.

▶ Re-run a seal in your browser →

Why it re-runs: determinism you can hold (Rust, not a model)

A measurement you can re-run has to be identical to the last bit — on your machine and on ours. That is a property of *how it is computed*, not a promise we make. Our instruments are discrete, seeded search — integers and rules, no model in the loop — written in Rust and compiled to run anywhere, even inside your browser. Same seed, same bytes, on any hardware.

A measurement built the way a model runs — floating-point math on a GPU — cannot make that promise⊕GPU arithmetic is reordered by the hardware for speed, and floating-point addition is not associative, so the last digits drift between machines. Deterministic *enough* to train on; not deterministic enough to seal.: the last digits drift between machines, so "trust me" can never become "re-run it." Determinism is not a detail here — it is the whole reason the seal means something, and it is why the same company can one day certify its own environments without a conflict of interest: a result you recompute is run-the-checker, not trust-me.

Underneath, both readings measure one thing — the bounded-compute structure a task or reward holds for a solver with finite compute⊕This is epiplexity (Finzi et al., 2026) — the interpretation the technical paper builds on; the load-bearing instrument is the search-effort measurement itself..

The theory, the formal results, and the field → the technical paper

We do not cry wolf

A fabricated reward-hack accusation is the one failure the brand cannot afford, so every instrument errs toward false-negatives over false-positives⊕A missed hack is cheaper than a fabricated accusation — so when the evidence is ambiguous, the instruments stay silent.: the exploit intent-check leans "genuine" when unsure; verifier-completeness abstains without usable ground-truth; IPT skips structured (JSON) gold answers where its text-invariance ops are not meaning-preserving.

A finding is always framed precisely — at pass_threshold X this env's reward accepts Y, seed-reproducible — and never as "this env is broken." A finding that survives a skeptic re-running the seed is worth more than ten that don't.

What the live board found

The live board carries 32 audited artifacts (2 F · 6 ERR · 3 C · 9 A · 1 difficulty). Every figure below is read straight from the published rows — re-render the Index and it updates itself.

The flagship signal. skill_reward_hacking grades F: 16 content-free completions scored at or above pass_threshold, 18 verifier-soundness gaps (a wrong answer scored as correct), a confirmed, replayable exploit (forces F). The exploit ships as a sealed transcript that re-runs from its seed — a constructed reward-hack, not a fixed probe, which is the strongest evidence the Index produces.

A single-signature C. allenai_ifeval: 1 content-free completion scored at or above pass_threshold — stated precisely, at the audited pass_threshold on the audited rows, and seed-reproducible. Not a claim the env is broken.

A clean control. aime2025 grades A — it withstood the full battery (degenerate probes, invariance, and exploit-search) with no signature. Negative controls are reported as loudly as findings; a board that only ever cried wolf would be worthless.

6 ERR rows are an env-compatibility signal — the environment did not load or roll out under the harness image — never a silent pass and never a quality claim.

Where difficulty comes from

Difficulty is governed by decomposability⊕Whether progress breaks into sub-goals a search can climb, or hides behind one all-or-nothing step.. On a rule-assembly puzzle a flat primitive-move search explodes (median 7 → 23 → 469 → 1372 simulations, censoring at distance 6) while an LM that proposes macro-actions collapses it to a single simulation — the LM injects the structure that removes the plateau:

▶ Open the live explorer → The mechanism, in the paper →

Where this goes: the reference corpus

Every confirmed hack ships as a sealed, replayable transcript; every difficulty reading ships as a certificate. We keep them. Batch after batch they accumulate into something the field does not yet have: a public, re-runnable corpus of reward exploits and difficulty curves — the reference others cite, and the standard an environment can be measured against⊕RewardBench and its kin rate reward *models*; no one ships a re-runnable corpus of reward *hacks* across the live ecosystem. That corpus is the asset..

The Index you see is the first slice of it. As the corpus grows the instrument compounds: rating the next environment gets cheaper, and the standard everyone is held to gets sharper — a measurement layer that gets more useful the more of the ecosystem it has seen.

A CI gate for RL environments

Reward-hacking is usually found after training, in a post-mortem. stardata ci gates it during training, like a failing test — the same audit, wrapped in a CI-shaped exit-code contract, so a gameable reward is caught before it corrupts a run:

$ stardata ci my_env --fail-on F
#  exit 0  reward-quality clean — proceed
#  exit 1  reward-hack at/above --fail-on — fail the build
#  exit 2  env not authorized for audit
#  exit 3  env did not load/roll out (ERR)

The --fail-on F default is the no-cry-wolf setting: only a confirmed exploit or two-plus signatures fails the build. Every verdict ships the exact command to reproduce it.

Responsible disclosure

Third-party findings go through a notify-then-publish tripwire: the author is notified privately with the precise, seed-reproducible finding and given roughly ten business days to fix, dispute, or acknowledge before the row publishes. A dispute lands a correction note, not a block — the seed makes every claim checkable by both sides. We seed the board with opt-in and study environments first, and earn the authority to cold-notify.

Cite this

The Index refreshes per audit batch (targeted monthly). Re-run any finding from its seed; each row is sealed by a record_sha256.

@misc{walker2026rii,
  author       = {Cody Lee Walker},
  title        = {starscry: the Reward Integrity Index --- deterministic, replayable
                  reward-quality audits of RL environments},
  year         = {2026},
  howpublished = {huggingface.co/datasets/staros/environment-quality-index},
  note         = {Re-run any finding from its seed; each row is sealed by a record_sha256.}
}

Read the technical paper →

Further reading. The full method, the formal results, the related work, and the consolidated references live in the technical paper — heavily cited, annotated in the margins.