← Blog

The Average Is Not a Guarantee

Cody Lee Walker · codylee.ca · 2026-07

Every compression method that claims to be "safe for LLMs" backs the claim the same way: an aggregate. Retains 95% of benchmark accuracy on average. If you ship real workloads, you should hear that sentence as: we don't know which of your images it breaks.

We measured how bad the gap gets. The natural per-image safety check — the one everyone reaches for first — is pixel error (RMSE): compress, diff the pixels, accept if the numbers are close. It feels rigorous. It is confidently, silently wrong on exactly the content you care about most.

The worst item on the board

On our calibration corpus, RMSE calls a document downscaled to x0.051 — 32 pixels wide — "identical" to the original. Every model we tested stops extracting the right value around x0.2, six times larger. The metric isn't just miscalibrated at the tail; it over-certifies small, dense text for every model we measured, because averaging pixel differences across an image dilutes the catastrophic local damage — a blurred "7" becoming a "1" — that changes the model's answer.

Run the naive RMSE policy on real held-out content and it silently loses 42–50% of answers — while shipping the biggest savings number on the board. That's the trap in one sentence: the metric that certifies the most savings is the one lying to you.

We fell for it ourselves

Our own physics-sim agent (starlab) reads detail from screenshots every turn. RMSE certified compressing its frames to x0.05 — a 16× "saving" that would have been pure hallucination fuel: on those frames the model's answers were already coin-flips. A gradient-energy witness — an instrument that attests whether fine structure survived, with a measured 0.99–1.0 safety record on renders where RMSE scored 0.18 — capped the safe compression at x0.8. The honest number was 20–35% savings, not 16×. Smaller, and real.

Witnesses are collated, not counted

The fix isn't a better single metric — it's the philologist's move. A witness is an instrument that attests what an image still says to a reader: OCR for text, gradient energy for structure, SSIM for general content. Each fails in a known, measured way on known content classes — we ran the bake-off across five corpora and 12,123 sealed model calls to map exactly where. They are collated and weighed, not counted, and the composite scores 0.98–1.0 (geomean) where RMSE over-certifies 5×.

Two properties fall out that no aggregate can offer:

• A per-request verdict. The certificate names the witness, the measured drift, the tolerance, and the target model — for this image.
• An honest refusal. When no witness will vouch for a compression, you get the original back with a certificate saying so — free. A system that can't say "no" per-image can only be wrong quietly.

On the 175-item held-out split the calibrated policy holds 96–97% accuracy at 12–22% billed savings — with the refusal rate published, not hidden inside an average. That's the difference between a benchmark number and a guarantee you can price.