Certified Isomorphic Compression: Deterministic Witnesses Bracket What a Vision-LLM Still Reads
Cody Lee Walker · StarOS / stardata · 2026-07-09 · working preprint
Companion to the starlens compression service (api.s2ar.dev) and the stardata environment-quality preprint [16]. Every empirical number below is generated from a sealed, content-addressed calibration cache and replays at zero API cost from the cited harness command; the generated
docs/research/data/numbers.jsonis the single source every figure and sentence traces to. Don't trust us — re-run the seed.
Abstract
Vision-language models bill image inputs by dimensions, not bytes: a provider resizes the image to its internal working resolution and charges per pixel area or tile. Downscaling an image is only useful if the model's answer survives. Every existing vision-token-reduction method reports an aggregate benchmark average; none tells whether this image, compressed this far, still says the same thing to this model. We present certified isomorphic compression: a deterministic witness measures the drift a compression induces in what the model reads, at the provider's own working resolution ("acuity"), and a sealed, seed-replayable certificate carries the verdict — including the certified refusal when no compression is safe. A compression format is a theory of its receiver made executable; ours measures the receiver, per model, and ships the measurement with every request.
Seven contributions, every number replayable from a sealed cache of 12123 VLM calls.
(1) The isomorphic-perturbation primitive: drift = D(witness(X), witness(T(X))) under a
meaning-preserving transform, with witness self-qualification — shared with concurrent
reward-auditing work [16].
(2) A measured witness bake-off across eleven production VLMs. On our synthetic corpora,
RMSE-style witnesses over-certify small/dense text for every model measured (7 items at 100% of
readers) and over-certify smooth renders by 3–7× (band-safety geomean 0.15–0.37), while a
gradient-energy witness is near-perfectly safe there (0.95–1.0). The negatives ride with the
positives: on text RMSE under-certifies more often than it over-certifies, and under a Holm
correction only the game-frame safety cells stay significant.
(3) A 110-item license-verified public calibration corpus (documents, receipts, websites,
charts, photos, code, dashboards — every image redistributable, per-item attribution) that
quadruples the cliff-item base, more than halves the CI widths — and reorders the witness
hierarchy: witness safety is content-class-relative. Gradient energy's render-safety does not
transfer (geomean 0.57–0.66 on the public mix); SSIM, mid-pack before, leads the general-purpose
field (0.92–0.93); and because the two fail on disjoint content classes, a composite witness
— certify only what grad AND ssim both accept — is near-safe on every corpus measured (geomean
0.98–1.0).
(4) An accuracy-vs-savings frontier with a 175-item held-out split never used for fitting. The
frontier caught our own routing failing twice: the RMSE fallback collapsed on renders (0.37–0.62
accuracy), and its gradient replacement over-certified small text past the OCR gate. The shipped
composite routing retains 0.96–0.97 of answers out-of-sample at 12–22% billed savings, where
the naive RMSE band loses 42–50% of answers; JPEG requantization preserves answers but saves
zero tokens; OpenAI's detail:low saves 85% with no per-item verdict.
(5) A cross-model transfer matrix: bands are reader-relative and transfer asymmetrically down a
robustness ordering — compression certified for the most robust reader breaks other models to
0.58–0.71 accuracy, while every band transfers to the robust reader at 0.96 or above (rounded).
(6) Calibration cheap enough to run per model release: adaptive binary search, ~50% fewer
calls, ~95% band-exact, tens of cents per model; k-repeat checks find temperature-0 answers
perfectly stable at the scales measured.
(7) The coverage-guarantee upgrade, measured to its edge: split-conformal tolerances [19]
built on a sound witness-band score carry their finite-sample guarantee out-of-sample only in
the tight-α regime (GPT-4o-mini: a measured 4.6% violation rate at a 5% target while certifying
87 of 131 items) — and fail through the witness's blind spots everywhere else, the same
held-out code panels defeating every model's certificate. Production therefore ships a measured
operating curve (tolerance → violation, refusal, savings), not guarantee language.
1. Introduction
Three facts shape the economics of vision-LLM inputs. First, images are billed by dimensions: Anthropic charges ≈ w·h/750 tokens after a long-edge fit (current documentation expresses this as 28-px patches, ≈ w·h/784 — the same shape [20]), OpenAI by 512-px tile after a short-edge fit, Gemini by 768-px tile [20]. Bytes are irrelevant — a JPEG can grow in bytes while its token cost drops 98%. Second, fresh images cannot ride prompt caching: a screenshot-per-turn agent pays full price every turn, exactly where text-side caching saves nothing. Third, the answer to "how far can I compress?" is not a property of the image alone — it depends on what question is asked and which model reads it [6].
The existing token-reduction literature answers with aggregates. Text-side prompt compression (LLMLingua [4, 5], productized middleware) is text-only. Provider-side vision-token pruning (PVC [3], the survey in [17]) is baked into checkpoints you must run yourself. Query-conditioned resolution selection (CARES [1], input-adaptive preprocessing keyed on edge density [2]) is the closest prior art — and reports benchmark-average quality, never a per-request guarantee. A recent market sweep found the same two structural gaps everywhere: nobody reduces image token cost as a drop-in request-path transform, and nobody attaches a per-request correctness instrument to the transform they do apply.
We build the instrument. The design goal is the one that made deterministic reward-auditing
credible [16]: a measurement the customer can recompute. A compression ships with a certificate
— witness, drift, tolerance, acuity, target model, seed, and a record_sha256 sealing the
arrangement — and a certified refusal ("this image cannot be compressed without changing what a
reader extracts") is a first-class result, not an error — an abstention in the selective-prediction
sense [18]. One definition up front: throughout, "certified" warrants a measured, replayable
verdict of a named witness at a named tolerance for a named reader — it does not warrant answer
preservation on unmeasured content, and the per-model tolerances of §5 are central-tendency
calibrations, not coverage guarantees (§6 measures the conformal upgrade [19]: the guarantee
transfers to held-out data only in the tight-α regime, and fails through the witness's blind
spots everywhere else). This paper reports what we measured while making that certificate honest: which cheap
witnesses can stand behind it, on which content, for which models, and at what calibration cost.
The strongest evidence that the instrument works is that it caught us: §6 documents our own
shipped routing failing off-corpus in exactly the way the bake-off predicts, and the fix it forced.
2. The primitive
Everything reduces to one operation, shared verbatim with the reward-gameability instrument in [16]:
drift = D( witness(X), witness(T(X)) ) # T = a meaning-preserving transform
verdict = "isomorphic" if drift <= tolerance else "moved"
For compression, X is the input image, T is a downscale (or a re-encode), and the witness is a
cheap deterministic proxy for what the model extracts. A witness is not a lens on a pre-existing
quantity: each witness constitutes a different "what the model reads" — RMSE reads gross
luminance, OCR reads print-legible text, gradient energy reads high-frequency detail — which is why
witness choice, not witness accuracy, turns out to be the operative decision (§4). Four design
elements matter:
Acuity. Every witness reads the image through the provider's internal resize (Anthropic ≈ 1568-px long edge with a ~1.15-megapixel cap; OpenAI ≈ 768-px short-edge fit; Gemini 768-px tiles). Drift only counts if it changes what the model actually receives. §5 measures this dial directly: the same image and witness certify an 8.3× different band depending on acuity alone. In the terms of the earliest hidden-surface taxonomy, the criterion is image-space, not object-space [21]: correctness is defined at the resolvable raster of the interface at hand — here the model's internal resize — not at the source's resolution.
The isomorphic band. A binary search over the linear downscale factor finds the deepest scale whose drift stays within tolerance — with a fast-refusal probe (if a near-identity x0.97 downscale already moves the witness, the band is empty and the original is returned after one probe).
Witness self-qualification. Before trusting a witness on an image, we apply the primitive to
the witness itself: if it drifts under the x0.97 near-identity perturbation, it is noise on this
content (tesseract "reading" a texture drifts heavily; real document text barely moves) and is
disqualified. probe="auto" routes images with substantial stable text to the OCR witness and
everything else to RMSE.
The seal. The certificate records source/compressed hashes, per-provider token savings, the
target model profile, {probe, drift, tolerance, acuity, deterministic}, and a seed, then seals a
canonical serialization under record_sha256. Deterministic witnesses carry the full
re-run-the-seed guarantee; the model-in-loop witness (§3) is labeled deterministic:false and is
used for calibration, never per-request certification. Compression itself is deterministic —
byte-stable outputs keep provider prompt caches warm. The seal binds the record, not the apparatus:
verification also assumes the pinned witness stack (tesseract build, resize kernel) survives
alongside the certificate — those pins live in the repository, outside the hash.
3. Instruments and corpora
Eight witnesses, ordered by trust:
| witness | extract | deterministic | measured role |
|---|---|---|---|
recon |
RMSE at acuity | yes | permissive general-purpose default |
phash |
8×8 average-hash Hamming | yes | gross-structure sanity |
dhash |
difference-hash Hamming | yes | bands coincide with phash on all 50 corpus images |
grad |
high-frequency (gradient) energy at acuity | yes | the measured-safe witness for renders/UI (§4) |
ssim |
1 - mean SSIM (luma, 7-px window) at acuity | yes | the human-perceptual baseline; the safest general single witness on public content (§4) |
dual |
grad AND ssim must both accept (max of tolerance-normalized drifts) | yes | the composite: near-safe on every corpus because its legs fail on disjoint classes (§4) |
ocr |
tesseract text, edit distance | yes (pinned engine) | conservative on substantial text; blind elsewhere (§4) |
vlm |
a real model's answer, scored for accuracy | no (model-in-loop) | calibration ground truth only |
Five VQA corpora (each item: image, question, exact ground truth). Synthetic-era: text/UI
— 26 items spanning dense terminal text, chart reading, form fields, and 4 real application
screenshots; gesture — 23 game-frame items (marker counts, spatial reads); starlab — 13
physics-render items (texture and artifact reads). Public-era: public — 110 calibration items
sampled from license-verified redistributable datasets (DUDE documents, CORD/SROIE receipts,
WebSight rendered websites, ChartX charts, VizWiz photos) plus two self-rendered families (code
panels, mixed dashboards), with per-item provenance and upstream licenses shipped alongside; and
heldout — 175 items from the same class mix, disjoint by id and image hash, never used to
fit any tolerance (the rule is pre-registered and test-enforced). An item only carries fidelity
signal if the model answers it correctly uncompressed (has_signal); a model's break must occur
above the sweep floor to count as a cliff (vlm_cliff) — at the floor, the true cliff is unknown
and the item is excluded rather than counted in our favor.
Ground truth is an answer sweep per (model, item) across ten downscale factors from x1.0 to x0.05 (adaptive binary-search sweeps on the public corpus), cached content-addressed by (image-hash, question, model) — a cache of 12123 sealed calls (sweeps plus the §6 policy arms, the held-out evaluation, k-repeats, and §7 fill-ins) that makes every result in this paper replayable at zero cost. Models whose base (uncompressed) answers are empty or wrong on every item are excluded before analysis — they carry no fidelity signal.
Honest methods — three artifacts our own pipeline manufactured, and their guards. (1) An early
run "measured" a dramatic per-model instability: one frontier model appeared to flip on a Stroop
item under mild compression. The cliff was fabricated by the agreement metric — raw string
agreement read "GREEN" vs "The word is **GREEN**" as a moved answer. Bands are now
accuracy-based whenever ground truth exists (isomorphic = still extracts the right value, not the
same phrasing), and the agreement fallback is containment-based (bidirectional substring on
normalized answers — robust to verbosity, blind to negation; it applies only where ground truth is
absent). (2) Items a model cannot read
uncompressed carry no witness evidence (has_signal). (3) A band at the sweep floor is a
measurement artifact, not an over-certification (vlm_cliff). One calibrated model was excluded
outright: qwen3.6-27b returns empty answers on every item (a reasoning-style token-budget burn) —
no signal, no row. We report these because they are the difference between a measured table and a
confident-looking artifact; the round that produced (1) also produced a corrected, smaller
per-model spread.
4. The witness bake-off, measured
The universal finding. Across eleven production VLMs, the RMSE witness (recon) over-certifies
a specific content class — small and dense text — for every model that can read it. Seven text
items are over-certified at 100% of evaluated readers, among them tiny_code and
small_label_number at 11/11 models, dense_config and subscript_number at 10/10:
| item | over-certified | readers evaluated |
|---|---|---|
| tiny_code | 11 | 11 |
| small_label_number | 11 | 11 |
| dense_config | 10 | 10 |
| subscript_number | 10 | 10 |
| low_contrast | 9 | 9 |
| which_color | 5 | 5 |
| count_circles | 4 | 4 |
This is a descriptive per-item consistency, not eleven independent replications: the readers share the items, overlap in training lineage, and two of the seven items are counting/color reads rather than dense text — the class boundary is observed, not pre-registered.
The magnitude is what matters. We score each cliff item with a band-safety ratio —
witness band ÷ model band, capped at 1.0; below 1.0 the witness certified more compression than
the model tolerates. Per-model geometric means on the text corpus run 0.58–0.78 (95% BCa bootstrap
CIs in Figure f5), with single-item failures as deep as tiny_code: RMSE certifies x0.051 where
ten of the eleven models break between x0.2 and x0.4 and the weakest measured reader breaks at
x0.85.
The negative, reported loudly. On the text corpus a per-model sign test does not find recon over-certifying more often than it under-certifies — the under-certifications are the majority for all eleven models (e.g. Claude Sonnet 4.6: 7 over vs 8 under, one-sided p = 0.70). RMSE's failure on text is class-concentrated and magnitude-heavy, not uniform: it is conservative on big-feature content and reliably, deeply wrong on exactly the content class an agent screenshots most — terminal text, dense config, small UI labels.
Renders are worse. On the two render corpora (game frames, physics shots) recon's band-safety
geomean collapses to 0.15–0.37 — it certifies roughly 3–7× more linear compression than the models
tolerate, on most cliff items. The hash witnesses match it (their bands coincide with each other on
all 50 corpus images; we collapse phash/dhash to one column). The gradient-energy witness
(grad) — mean absolute finite-difference of luma at acuity, drift = relative energy change — is
the measured-safe witness: geomean 1.0 [1.0, 1.0] with zero over-certifications on the
gesture corpus for all three models, 0.9859–1.0 on starlab, and 0.95–0.98 on text. Independently,
input-adaptive VLM preprocessing keyed on edge density [2] converges on the same signal from the
efficiency side; we arrive at it from the certification side with per-item measurements.
Inference, scoped honestly. A paired sign-flip permutation test on per-item log ratios supports grad being safer than recon in every (model, corpus) cell measured — p ≤ 0.0004 on gesture, 0.008–0.065 across the text and starlab cells, with three text cells at or above 0.05. All seventeen tests are reported uncorrected and the analysis is exploratory; under a Holm correction only the three gesture cells remain significant at α = 0.05.
The human-perceptual baseline, measured rather than presumed. On the synthetic corpora SSIM's band-safety geomean runs 0.67–0.76 on text and 0.52–0.79 on the render corpora — consistently safer than RMSE, never approaching grad there. The machine reader's metric gap [13, 14] is now a measurement, not an assertion.
The public-corpus reversal: witness safety is content-class-relative. The 110-item public corpus (74/54/59 cliff items for the trio — four times the synthetic base, CI widths less than half) reorders the hierarchy. Band-safety geomeans, Claude Sonnet 4.6:
| witness | synthetic text | game frames | public mix |
|---|---|---|---|
| ocr | 0.79 | — | 0.95 |
| ssim | 0.70 | 0.79 | 0.93 |
| grad | 0.95 | 1.00 | 0.66 |
| recon | 0.60 | 0.15 | 0.64 |
| phash/dhash | 0.35 | 0.12 | 0.22 |
Gradient energy's near-perfect render safety does not transfer: on the public mix it
over-certifies like RMSE — and the failures are not photos but small text that slips the OCR
gate (thermal receipts, dark-background code panels: 10 of 10 code-panel cliffs over-certified).
SSIM's failures concentrate on charts. The two witnesses fail on disjoint content classes —
so their conservative composition (dual: certify only the band both accept) is near-safe
everywhere we can measure: band-safety geomean 0.98–1.0 on the public mix, 1.0 on both render
corpora, 0.99 on synthetic text, with 2–3 residual over-certifications per model (all charts). No
single cheap witness earns the certificate across content classes; a collation of two does. The
sign-test picture also shifts on real content: GPT-4o-mini's recon over-certifications now
significantly outnumber its under-certifications (34 vs 20, one-sided p = 0.038).
The witness sandwich, and where it inverts. On substantial text the intended ordering holds
and is wide: on tiny_code, recon certifies x0.051, typical models break at x0.2, and pinned
tesseract refuses below x0.742 — the OCR witness is conservative by construction and leaves real
savings on the table (production VLMs out-read tesseract on compressed text). But the sandwich has
edges we measured: (a) on text-free content OCR extracts nothing, sees zero drift, and
certifies anything — it over-certifies shape/color items as deeply as recon does text; the
probe="auto" stability gate exists precisely to keep OCR off such content; (b) for the weakest
reader measured (Claude Fable 5 breaks on tiny_code at x0.85), the model's cliff sits above
the OCR band — no deterministic witness in our set brackets that model on that item without the
per-model tolerance of §5. That the weakest compressed-text reader in the table is also its newest
frontier model is itself a finding: capability and compression-robustness dissociate.
5. Per-model calibration, and what it costs
The band is reader-relative. Third-party measurements first suggested this (quarter-resolution document extraction costs Gemini 2.5 Pro 1.7pp but GPT-4o 29.1pp and open 8–17B models ~40pp [6]); our sweeps confirm it per item. We therefore calibrate a per-model tolerance scale: the geometric mean of band-safety ratios, clamped never to loosen beyond the heuristic prior and rounded to 0.05. Eleven models are measured (tiers-v1):
| model | tol_scale | recon geomean [95% CI] | calibration corpus | typical savings (area proxy) |
|---|---|---|---|---|
| Gemini 3.1 Flash Lite | 0.75 | 0.73 [0.63, 0.81] | public | 96% |
| Claude Sonnet 5 | 0.65 | 0.63 [0.54, 0.71] | public | 89% |
| Claude Sonnet 4.6 | 0.6 | 0.64 [0.55, 0.73] | public | 91% |
| GPT-4o-mini | 0.6 | 0.58 [0.47, 0.67] | public | 89% |
| Gemini 3.5 Flash | 0.6 | 0.61 [0.49, 0.71] | public | 88% |
| Claude Fable 5 | 0.6 | 0.61 [0.43, 0.76] | public (stride) | 88% |
| GPT-5.4-mini | 0.6 | 0.59 [0.50, 0.68] | public | 86% |
| Grok 4.5 | 0.5 | 0.68 [0.59, 0.75] | public | 87% |
| Llama 4 Scout | 0.5 | 0.55 [0.46, 0.64] | public | 89% |
| Qwen 3.7 Plus | 0.5 | 0.67 [0.56, 0.75] | public | 91% |
| Llama 3.2 11B | 0.5 | 0.61 [0.50, 0.70] | public | 88% |
Every row now carries public-corpus provenance (tighten-only refit, against both the heuristic prior and the previously shipped value): the dense trio first, then the eight adaptive rows in a breadth pass. The refit moved the loose synthetic-text fits down into a tight 0.5–0.75 band — Gemini 3.5 Flash 0.75 → 0.6, GPT-5.4-mini 0.7 → 0.6 — and the CIs are roughly half their synthetic-text widths (the 3–4× cliff-item base at work). Claude Fable 5, the priciest reader, was measured frugally on a deterministic stratified stride of the corpus (24 cliff items; its CI is the wide one) — the stride is the designed cheap path for frontier-priced models. The public refit also replicates §4's reversal on all eleven models: gradient energy's render-safety collapses to 0.57–0.68 on the public mix for every model measured (permutation vs RMSE: no effect anywhere, smallest p 0.17), while the dual composite holds 0.96–1.0 across the board — the composition argument is not a trio artifact.
The savings column is a pixel-area proxy (1 − band²) — the dimension-reduction the band buys, independent of billing shape. Provider-billed savings are tile-quantized and lower on OpenAI/Gemini billing: §6 measures the same models under their billing shapes, where a 99% area-proxy row can bill out at a third of that. Two denominators, both real; every savings number in this paper now names its own.
The CIs still overlap heavily — even at the public corpus's cliff base, per-model differences are mostly inside noise, and the clamp-to-prior rule reflects that: a measurement this size may tighten a tolerance, never loosen it. What is not inside noise is the direction everything above establishes: every tol_scale is well below 1.0 because the universal small-text class drags every model's geomean down.
Calibration is cheap enough to be routine. The adaptive band finder (binary search over the scale grid, assuming rough monotonicity) uses 4.5 calls per item — 49.6% fewer than the full sweep — and lands the exact same band on 94.9% of the 78 fully-cached validation items (every miss is its own honest signature: a non-monotone accuracy curve). At production prices that is tens of cents per model on the synthetic corpus and single-digit dollars on the 110-item public corpus; the frugal stride cuts a frontier-priced model (Fable 5) to roughly a dollar. A scheduled weekly job keeps the table growing as models ship.
The acuity dial. The same witness, image, and tolerance certify wildly different bands
depending on the resolution the witness reads at: on a real launcher screenshot the recon band
moves from x0.064 at 256-px acuity to x0.531 at 1568-px — an 8.3× band spread, worth roughly 64×
in tokens on per-pixel billing. On the saturated items (dense_config, waterfall_shells) the curve is flat at the
search floor at every acuity — over-certification so deep the acuity dial cannot express it,
which is the witness-choice problem of §4, not a resolution problem.
6. The accuracy-vs-savings frontier
The operational question is not "which witness is prettiest" but "what policy should compress an
agent's screenshots." We compare policies per corpus and model: blind fixed downscales; each
witness's band at default tolerance; three generations of the calibrated auto policy
(auto-v1: stable text → ocr, else RMSE; auto-v2: else gradient energy; auto-v3: else the dual
composite — the shipped routing, and this section's own results are the reason for each
succession); and two industry knobs (JPEG quality, detail:low) measured with fresh calls. Scope:
cells scored on a policy's own calibration corpus are in-sample consistency checks; the held-out
split below is the out-of-sample answer.
Claude Sonnet 4.6, the frontier's cleanest in-sample panel (synthetic text/UI):
| policy | accuracy retained | image tokens saved |
|---|---|---|
| calibrated auto-v1 (tiers-v1) | 1.00 | 71.8% |
| calibrated auto-v2 (tiers-v1) | 1.00 | 17.4% (48% refusals) |
| calibrated auto-v3 (tiers-v1) | 1.00 | 16.5% (56% refusals) |
| ocr band (default tol) | 0.96 | 69.1% |
| recon band (default tol) | 0.84 | 94.0% |
| grad band (default tol) | 0.92 | 27.7% |
| ssim band (default tol) | 0.84 | 71.5% |
| dual band (default tol) | 0.96 | 24.2% |
| fixed x0.4 | 1.00 | 83.8% |
| fixed x0.2 | 0.84 | 95.9% |
| jpeg q25 (full size) | 1.00 | 0.0% |
Four readings, including the ones against us:
- The uncalibrated RMSE band is the trap. It looks spectacular on savings (94.0%) and quietly loses 16% of answers — the over-certification of §4 expressed in dollars-vs-correctness. This is the policy an aggregate-average report would ship.
- A blind fixed scale can match the calibrated point — if you already know which one. Fixed x0.4 retains 100% at 83.8% savings for Sonnet 4.6. But x0.4 is not safe for Fable 5 (0.92) or Llama-family readers (0.96) — the right fixed scale is itself model-relative, which is the calibration story restated. And a fixed scale certifies nothing per item: the calibrated policy's residual failures are identified in advance by a certificate (or a refusal); a fixed scale's failures are discovered in production.
- JPEG requantization is simultaneously harmless and worthless. Answers survive q60 and even q25 essentially untouched (1.00 / 1.00 / 0.96 across the dense trio) — and the token axis does not move, because tokens are dimensions. The knob the industry actually ships optimizes the denominator nobody bills on. (This also validates carrying requantization inside the fidelity loop when minimizing upload bytes: at these qualities the witness tolerance is not consumed by encode artifacts.)
detail:lowis a strong blind baseline — where it really exists. On native OpenAI semantics (GPT-4o-mini) it retains 0.96 accuracy at 85.0% savings: a single fixed operating point, no per-image adaptivity, no certificate, and no equivalent on Anthropic or Gemini APIs. On OpenAI billing it dominates the calibrated policy on raw savings — what the certificate buys there is per-image adaptivity and a verdict, not tokens. We report the equivalent cells for OpenRouter-hosted Llama/Qwen models but do not trust them as evidence: a provider that ignores thedetailfield silently serves the full image, which reproduces exactly the aggregate-average blindness this paper is against. The episode, twice: our own policy failed off-corpus, and the instrument caught it both times. Auto-v1's RMSE fallback — evaluated on the render corpora it was never shown in an earlier draft — collapses to 0.43 accuracy on game frames and 0.62 on physics renders (Sonnet 4.6; GPT-4o-mini worse at 0.37), exactly as §4's bake-off predicts:probe="auto"routed text-free content to the one witness §4 proves over-certifies it. Auto-v2 replaced the fallback with the gradient witness and restored the render cells (Figure f4) — and then the public corpus caught v2's own gap: grad over-certifies small text that slips the OCR gate (§4's reversal), and v2 held only 0.89–0.91 of answers out-of-sample. Auto-v3 replaces the fallback with thedualcomposite. We keep all three generations in the tables because the succession is the paper's thesis pointed at ourselves, twice: a policy evaluated only on its calibration corpus is an aggregate average waiting to fail — ours did, the sealed cache caught it before production did, both times, and each fix was measured below before it shipped.
The held-out verdict. 175 items, same class mix as the public corpus, disjoint, never used to fit anything. Accuracy retained @ provider-billed savings (refusal fraction in parentheses):
| policy | Sonnet 4.6 | GPT-4o-mini | Llama 3.2 11B |
|---|---|---|---|
| calibrated auto-v3 (shipped) | 0.96 @ 22% (43%) | 0.97 @ 12% (47%) | 0.97 @ 14% (48%) |
| calibrated auto-v2 | 0.89 @ 38% (21%) | 0.89 @ 22% (23%) | 0.91 @ 23% (23%) |
| calibrated auto-v1 | 0.85 @ 48% (17%) | 0.90 @ 32% (18%) | 0.89 @ 29% (19%) |
| fixed x0.4 | 0.81 @ 70% | 0.89 @ 42% | 0.81 @ 41% |
| recon band (default tol) | 0.58 @ 88% | 0.58 @ 60% | 0.50 @ 59% |
Three readings. (a) The RMSE trap is worse out-of-sample — it silently loses 42–50% of answers on real content while shipping the biggest savings numbers. (b) The generation ordering holds off-corpus, and v3 recovers most of the gap: 0.96–0.97 retained, against v2's 0.89–0.91 and the in-sample-perfect 1.00 that motivated humility. (c) The remaining trade is explicit: v3 buys its safety with refusals (43–48% of held-out items get "don't compress this" rather than a risky band) and correspondingly modest billed savings — the safety-first operating point. Callers who want the other end of the curve can name a witness and tolerance explicitly; the certificate prices the risk either way.
Pricing the risk: the conformal dial, measured. The per-model tolerance of §5 is a central-tendency fit: the geomean of band-safety ratios says where the witness typically stands relative to the model, not how often a chosen tolerance over-certifies the next item. Selective prediction wants the second number — a dial α such that at tolerance tol(α), at most an α-fraction of fresh items get a bad certificate. Split conformal calibration [19] promises exactly this from one ingredient: an exchangeable per-item score for the largest tolerance that is still safe, whose lower α-quantile on a calibration set becomes tol(α).
We tried the cheap score first, and it is this paper's third self-caught failure. Scoring each item by its witness drift at the model's measured band scale (one witness evaluation per item) assumes drift keeps growing past the band, so that a tolerance just above it certifies only marginally deeper. Real drift curves plateau — SSIM saturates on flat chart backgrounds, gradient energy dips and recovers — and a tolerance a hair above the plateau certifies far past the model's cliff. On the held-out split the resulting "guaranteed" tolerances violated at roughly twice nominal everywhere (Sonnet 4.6: 11.7% of certified items wrong at a 5% target), and at α = 0.2 they certified images at the sweep floor. The sound score drops the assumption and prices the witness-band function itself: sweep a tolerance grid (all local — witness bands cost no API calls) and take the largest prefix at which the witness band never certifies deeper than the model's measured band. Held-out violation rate among certified items, sound score:
| α (target) | Sonnet 4.6 | GPT-4o-mini | Llama 3.2 11B |
|---|---|---|---|
| 0.05 | tol 1.2 → 0.099 ✗ | tol 1.0 → 0.046 ✓ | tol 1.0 → 0.053 ✓ |
| 0.10 | tol 2.5 → 0.181 ✗ | tol 3.0 → 0.132 ✗ | tol 2.0 → 0.187 ✗ |
| 0.20 | tol 3.0 → 0.227 ✗ | tol 5.0 → 0.290 ✗ | tol 4.0 → 0.212 ✗ |
(Llama's 0.053 is 4 violations among 75 certified — inside finite-sample noise of the 0.05
target.) The verdict is a split decision with a legible shape. At α = 0.05 the guarantee
transfers for GPT-4o-mini — 87 of its 131 signal items certified at 18.5% mean billed savings
with a measured 4.6% violation rate — and for Llama; Sonnet breaks (0.099 against 0.05, clear at
n = 111), and every model breaks at looser α. And the residual failures are not noise: the same
six held-out code panels are violated by every model — the dual composite accepts them at or
near the sweep floor while all three models fail — alongside the chart-mark class §4 already
convicts. A conformal guarantee inherits its witness's blind spots: score exchangeability
cannot survive a content pocket the witness cannot see, and no quantile fixes a witness that is
blind on a class. The practical interface is therefore two-tier. In the tight-α regime a
conformal tolerance is a real, honestly-priced loosening — Sonnet's α = 0.05 tolerance of 1.2
lifts held-out billed savings from 21.9% to 34.8% while refusals drop from 43% to 31%, at a
measured 9.9% violation rate against v3's 4.3%. Everywhere else the dial ships as the measured
operating curve — tolerance in, (violation, refusal, savings) out — with no guarantee language
attached: calibration/tiers.json now carries a heldout_curve per fully-measured model, and
the certificates stay sealed and replayable either way.
7. Cross-model band transfer
If a certificate names its target model, the natural question is what happens when a different model reads the compressed image. For the six models with complete sweeps we take model A's measured band per item and score model B's answer at that scale (full coverage, 68 targeted fill-in calls):
Transfer is asymmetric and robustness-ordered. Bands certified for the most
compression-robust reader (Gemini 3.5 Flash) break every other model — read-accuracy 0.58–0.71
across the row. In the other direction, every model's band transfers to Gemini at 0.96 or above
(rounded; the lowest cell is 0.9583). The
frontier-model rows (Sonnet 4.6) transfer at 0.83–1.00; the weakest reader (Fable 5) is the most
frequently broken column. The practical rule the matrix licenses: a certificate is valid
downward in robustness (a band certified for a fragile reader is safe for a robust one) and
invalid upward — which is why the certificate carries target.model and why a multi-model
pipeline must certify for its weakest reader. Part of the pattern is entailed by construction (a
band is defined by its own reader's accuracy, so the diagonal is 1.0 and rough ordering follows
band depth); the informative cells are the off-diagonal failures and the non-monotone exceptions,
which a larger corpus should analyze with per-cell intervals. This is the compression-side sibling
of the scale-transfer result in the difficulty-certification line [16]: the instrument's verdict is
solver-relative, and pretending otherwise is exactly the aggregate-average failure.
8. Related work
Query-conditioned resolution selection. CARES [1] trains a compact selector that predicts the minimal sufficient input resolution per (image, query) and reports large GPT-4o cost cuts; input-adaptive preprocessing for FastVLM [2] keys the reduction on edge density and entropy — independent convergence on the gradient-energy signal our bake-off measures as the safe witness. Both optimize the average; neither emits a per-request, replayable verdict or a refusal. Our per-item sweeps are, to our knowledge, the first published per-model certification measurements for this transform family.
Provider-side token compression. PVC [3] and the surveyed pruning/pooling/resampler methods [17] compress inside the model stack (checkpoint-bound); they change what a provider could bill, not what a client pays today. Text-side prompt compression (LLMLingua [4], LongLLMLingua [5]) is the occupied, text-only sibling. Semantic caching [15] reuses answers across queries at an embedding threshold — the same aggregate-vs-per-request gap in a different subsystem.
Robustness measurement. VAREX [6] measures per-model resolution fragility on document extraction (the third-party numbers our tiers-v0 prior encoded); VLM-RobustBench [8] finds low-level spatial/resampling perturbations degrade VLMs more than photometric ones; and visually prompted benchmarks measurably tilt under JPEG settings that are motivated as imperceptible [9] — our jpeg_family transform exists to certify that axis per request rather than discover it per leaderboard. DeepSeek-OCR [7] publishes a compression-vs-precision curve for text rendered as vision tokens — an isomorphic band measured with a model witness, and (pinned, local) our leading candidate for a mid-sandwich deterministic-given-pin witness.
Machine-oriented quality metrics. SSIM [13] and LPIPS [14] are calibrated to human judgments — and rather than assert the mismatch, §4 measures it: an SSIM witness over-certifies machine readers on every corpus (band-safety geomean 0.52–0.79), better than RMSE, far from the gradient-energy witness. ML-CLIPSim [12] and text-aware learned codecs [10, 11] optimize machine-oriented quality but in bytes — the wrong denominator when billing is dimensional. The certified refusal connects this work to selective prediction [18]: the certificate is an abstention mechanism whose reject decision is deterministic and replayable. Conformal calibration [19] looked like the natural route from §5's central-tendency tolerances to per-request coverage guarantees; §6 measures where that route actually goes — the guarantee transfers out-of-sample only in the tight-α regime, because score exchangeability breaks on content pockets the witness cannot see. Distribution-free coverage is not a patch a wrapper can apply over a blind witness; the witness has to see the class first.
Verification scaling. LLM-as-a-Verifier [22] identifies verification as a first-class scaling
axis and buys verifier quality with inference: continuous scores as the expectation over
scoring-token logits, scaled along granularity, repeated evaluation, and criteria decomposition,
with state-of-the-art best-of-N trajectory selection across coding, robotics, and medical agents.
It is the strongest statement of the model-in-loop pole of the design space this paper's trust
labels split: verdicts are fresh stochastic readings whose quality scales with compute —
deterministic:false in our taxonomy, the class we confine to calibration — and the method
assumes scoring-token logit access that several frontier APIs do not expose, so the measuring
apparatus is itself gated. The programs are complementary across an agentic loop: we certify what
a model reads (inputs); they score what an agent did (outputs).
Model-internal confidence as a fidelity signal. Coupled multimodal samplers read cross-modal
fidelity off the model's own internals: SC-CMJP [24] gates image-token commitment on an
entropy-weighted mixture of self-confidence and attention-weighted text confidence, and re-masks
low-confidence tokens to self-correct — in effect a model-internal witness that text and image
still agree, consulted mid-generation. The same three properties that make internal confidence
the right signal for a sampler make it unusable as a certificate witness here: it requires logit
and attention access that frontier vision APIs do not expose, it is a fresh stochastic reading
rather than a deterministic replay (deterministic:false, the class §3 confines to calibration),
and it cannot be sealed for third-party re-verification. Its benchmarks are, however, kin:
JMaze/JNono score joint text-image outputs against algorithmic ground truth — the same
no-human-labels discipline as our §5 simulation census, and (as rendered thin-stroke grids with
small marginal digits) a ready-made instance of the fine-structure content class where §6 located
witness blind spots. Our puzzle-render corpus (tools/gen_puzzle_corpus.py) synthesizes that
class from seeds, license-clean, for exactly this probe.
The certificate discipline — sealed hashes, seed replay, deterministic-vs-model-in-loop trust labels, refusal as a result — is inherited from the environment-quality auditing line [16], where the same primitive audits reward functions instead of inputs. Perceptual coding's own calibration archive was famously unkept — the labs that tuned MP3's ear-models kept no record of who their expert listeners were [23], and the verdicts of those listening tests survive only as the format itself. The certificate is the record that lineage lacked: the meter named, the arrangement sealed, the verdict replayable.
9. Implications, limitations, reproducibility
Implications. (a) No single cheap witness earns the certificate across content classes — witness safety is content-class-relative (§4's reversal), and the measured answer is composition: certify only what independent witnesses with disjoint failure classes all accept. The production auto-routing now does exactly this (§6). (b) Calibrate tolerance per model; at tens of cents per model there is no excuse for a static threshold. (c) Certify for the weakest reader in the pipeline; transfer is one-directional. (d) Prefer dimension reduction over byte reduction; the billing axis is dimensions, and requantization buys nothing. (e) Prefer a refusal to a silent loss — the episode's two rounds are the whole argument in two tables. (f) Evaluate every policy on a held-out split; ours moved the headline from an in-sample 1.00 to an honest out-of-sample 0.96–0.97. (g) A coverage guarantee is only as good as its witness's eyes: the conformal dial transferred exactly where the witness has no blind spot on the content mix and failed through the blind spots everywhere else — fix the witness before reaching for the quantile. (h) Input certification and output verification couple: an output verifier (LM judge, trajectory reward model [22]) grades what the agent did with what it saw, and if the input path over-certifies, the output verdict is measured against a world the agent never received. In screenshot-per-turn agents, a certified input bound is a precondition for the output verdict to mean anything.
Limitations, stated not hidden.
Corpus and statistics. The public corpus is 110 calibration + 175 held-out items — better than
the 26-item start (cliff bases of 54–74 per model, CI widths less than half), still far from a
census; documents thinned to 15 usable items after filtering, and one photo item is
provider-rejected at full resolution. As with every compression standard's calibration corpus
[23], whose reading got measured becomes silently normative for everyone downstream — ours
measures agent-workload reading. All eleven tiers rows now carry public-corpus provenance (the breadth
refit landed every tolerance in a 0.5–0.75 band); Fable 5 was measured frugally on a stratified
stride of the corpus (24 cliff items — its CI is the wide one), and the full witness bake-off,
held-out evaluation, and conformal validation still cover only the three fully-swept models. The
transfer result is one corpus.
Measurement collection. k=3 repeats found temperature-0 answers perfectly stable (15 items ×
three models × two scales) — but both scales sit inside the safe band; stability at the cliff
edge remains unmeasured. Adaptive-cache models carry snap error in the frontier (accuracy read
at the nearest cached scale; mean snap ≤ 0.13); the adaptive band-finder's rare misses are large
(worst measured: adaptive x0.12 where the full sweep says x1.0 — adaptive rows are calibration
inputs, never per-item certificates).
Witnesses and policies. Tesseract is the only OCR leg (a pinned DeepSeek-OCR witness is the
planned mid-sandwich upgrade). The dual composite still over-certifies 2–3 chart items per
model — charts are the open class — and the conformal validation surfaced a second pocket: a
cluster of dense held-out code panels the composite accepts at or near the sweep floor while
every model fails, the concrete blind spot behind §6's coverage ceiling. detail:low cells for non-OpenAI-hosted models are reported
but untrusted (the field may be silently ignored). Auto-v3's false-refusal cost is now visible
(43–48% held-out refusals) but its price — savings a bolder policy would have safely captured —
is only bounded, not optimized. And everything here measures answer preservation on VQA-style
extraction, not open-ended generation quality. Finally, the constitution that §2 states cuts both
ways: for everyone downstream of a certificate, "what the model reads" is, operationally, what
the shipped witness passes — a calibrated filter's verdict tends to define fidelity rather than
track it. Collation, refusal, and the VLM calibration loop bound this inversion; nothing removes
it, and the seal binds the record, not the criteria.
Reproducibility. Every number traces to docs/research/data/numbers.json, generated from
sealed artifacts by the commands below; a checker fails the build on any numeric literal in this
document that does not trace (quantified claims — "every", "only", "zero" — are outside the
checker's reach and are re-audited manually at each regeneration, a discipline this draft earned
the hard way). The VLM answer cache is content-addressed; artifact release (cache, corpora, tools,
license) is pending, and until it lands "replayable" means replayable in this repository.
Certificates carry the acuity they were measured at; a provider resize change invalidates them,
and the weekly recalibration is the staleness policy. The entire analysis replays with zero API
calls:
PYTHONPATH=src python tools/ingest_public.py --all # rebuild the public corpora (deterministic)
PYTHONPATH=src python tools/stats.py # rows.json + stats.json (CIs, sign/permutation tests)
PYTHONPATH=src python tools/frontier.py # frontier.json (replay; --spend re-runs paid arms as cache hits)
PYTHONPATH=src python tools/heldout_eval.py --spend # heldout.json (replays from cache)
PYTHONPATH=src python tools/repeat_stability.py # stability.json plan (--spend replays)
PYTHONPATH=src python tools/transfer_matrix.py # transfer.json
PYTHONPATH=src python tools/acuity_curve.py # acuity.json (deterministic)
PYTHONPATH=src python tools/adaptive_stats.py # adaptive.json
PYTHONPATH=src python tools/conformal.py --sound # conformal_sound.json (local band sweep)
PYTHONPATH=src python tools/conformal.py --sound --validate # heldout validity (replays from cache)
PYTHONPATH=src python tools/update_tiers_curve.py # heldout_curve -> calibration/tiers.json
PYTHONPATH=src python tools/make_figures.py # figures/*.png + numbers.json
PYTHONPATH=src python tools/check_numbers.py # the traceability gate
References
[1] M. Kimhi, N. Shabtay, R. Giryes, et al. CARES: Context-Aware Resolution Selector for VLMs. arXiv:2510.19496, 2025.
[2] D. Cahyani, I. M. Suartana, N. Yudistira. Input-Adaptive Visual Preprocessing for Efficient Fast Vision-Language Model Inference. arXiv:2512.20839, 2025.
[3] C. Yang, et al. (OpenGVLab). PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models. arXiv:2412.09613, 2024 (CVPR 2025).
[4] H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, L. Qiu. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. arXiv:2310.05736, 2023 (EMNLP 2023).
[5] H. Jiang, Q. Wu, et al. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. arXiv:2310.06839, 2023 (ACL 2024).
[6] O. Barzelay, T. Azulai, N. Shapira, et al. VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents. arXiv:2603.15118, 2026.
[7] H. Wei, Y. Sun, Z. Li. DeepSeek-OCR: Contexts Optical Compression. arXiv:2510.18234, 2025.
[8] A. Saxena, A. Suglia, P. Minervini. VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models. arXiv:2603.06148, 2026.
[9] K. Feng, L. Lian, L. Dunlap, et al. Visually Prompted Benchmarks Are Surprisingly Fragile. arXiv:2512.17875, 2025.
[10] A. Della Fiore, A. Gnutti, M. Dalai, et al. End-to-End Semantic Preservation in Text-Aware Image Compression Systems. arXiv:2503.19495, 2025.
[11] C.-H. Kao, C. Chien, W.-C. Tseng, et al. Bridging Compressed Image Latents and Multimodal Large Language Models. arXiv:2407.19651, 2024.
[12] S. Ding, H. Fu, J. Liang, et al. ML-CLIPSim: Multi-Layer CLIP Similarity for Machine-Oriented Image Quality. arXiv:2605.09479, 2026.
[13] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Processing 13(4):600–612, 2004.
[14] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. arXiv:1801.03924, 2018 (CVPR 2018).
[15] S. Regmi, C. R. Pun. GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching. arXiv:2411.05276, 2024.
[16] C. L. Walker. Replayable Evidence for RL-Environment Quality: Deterministic Audits and Solver-Effort Difficulty Certification. stardata working preprint, 2026.
[17] K. Shao, K. Tao, K. Zhang, et al. A Survey of Token Compression for Efficient Multimodal Large Language Models. arXiv:2507.20198, 2025 (TMLR 2026). Companion repository: github.com/cokeshao/Awesome-Multimodal-Token-Compression.
[18] Y. Geifman, R. El-Yaniv. Selective Classification for Deep Neural Networks. arXiv:1705.08500, 2017 (NeurIPS 2017).
[19] A. N. Angelopoulos, S. Bates. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv:2107.07511, 2021.
[20] Provider image-token documentation (accessed 2026-07): Anthropic platform.claude.com/docs/en/docs/build-with-claude/vision; OpenAI developers.openai.com/api/docs/guides/images-vision; Google ai.google.dev/gemini-api/docs/tokens.
[21] I. E. Sutherland, R. F. Sproull, R. A. Schumacker. A Characterization of Ten Hidden-Surface Algorithms. ACM Computing Surveys 6(1):1–55, 1974.
[22] J. Kwok, S. Li, P. Atreya, Y. Liu, Y. Jiang, C. Finn, M. Pavone, I. Stoica, A. Mirhoseini. LLM-as-a-Verifier: A General-Purpose Verification Framework. arXiv:2607.05391, 2026.
[23] J. Sterne. MP3: The Meaning of a Format. Duke University Press, 2012.
[24] Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes. arXiv:2607.13188, 2026.
Internal technical reports (this work): docs/validation.md (the recon/ocr divergence on real
screenshots); docs/vlm-calibration.md + -round2/-round3/-fable (per-model text calibrations);
docs/vlm-calibration-starlab.md, -gesture.md (the render bake-offs); docs/design.md (the
certificate schema); docs/market-landscape.md (the verified competitive map);
calibration/tiers.json (the measured tolerance table of record). The live service:
api.s2ar.dev; the interactive degradation demo: s2ar.dev/.