A benchmark should be able to fail its own claim

When the benchmark fails before the model does

FragileVision does not just ask whether a model was right. It asks whether the question, the sample and the people who labelled it earned the right to say so.

I went looking for the best model. I found a problem with the question.

The first version of this work was a benchmark on my own photographs. I expected model size to explain the mistakes. It did not.

A four-billion parameter model with a short, plain question beat an eight-billion parameter model with the original wording. Negation could turn a strong result into an almost constant answer. A single whitespace change in a prefill could decide whether a model returned one word, a paragraph, or nothing at all.

I wrote that story in How you word the question matters more than model size. The article ended with a methodological problem: if a score can move that much while the photographs and weights stay fixed, then holding one prompt fixed is not a neutral choice. The prompt is part of the instrument.

The scripts behind that benchmark could prove the point once. They could not make the discipline repeatable. FragileVision is the tool I wanted after the result made me distrust my own setup.

The rule

A result is evidence only after the ways it could be an artefact have been made visible.

That sounds severe. It is meant to be. Models already produce answers with an air of certainty. Evaluation software should not add another layer of false confidence.

The model is only one of the moving parts

Most leaderboards vary the model and freeze everything else. FragileVision also freezes the model and puts pressure on the evaluation around it.

01

The wording

The same semantic question is asked through controlled changes of language, negation, examples, order, format and length. Prompt Fragility measures when the verdict moves.

02

The sample

Near duplicates, dominant scenes, class imbalance and train/test leaks can make repetition look like evidence. The audit treats the scene, not the file, as the unit of independence.

03

The ground truth

A label written by one person is an opinion with a database row. Blind review, named judgements, agreement coefficients and adjudication show how much of it the panel actually shares.

These failures should not be folded into one “robustness” number. A model that changes between identical calls is not the same problem as a prompt that changes its answer. A panel that cannot agree is not model error. A burst of twenty frames is not twenty independent observations.

FragileVision keeps those surfaces separate even when that makes the final report less tidy.

A claim leaves a paper trail

The application is a ledger before it is a dashboard. It records what happened closely enough that a later result can be challenged instead of merely admired.

01

Import locally

Images are copied into managed storage, validated and deduplicated by SHA-256. Nothing is uploaded.

02

Audit the sample

File integrity, resolution, balance, source groups, perceptual similarity and split leakage are checked before a run.

03

Build ground truth

Reviewers work blind. Unanimity and strict majority can settle a case; ties and pluralities cannot. Adjudication is added, never used to erase disagreement.

04

Stress the question

Prompt variants preserve the intended meaning while changing the linguistic surface. Repetitions measure drift separately.

05

Run local models

Ollama and private OpenAI-compatible endpoints are allowed. Public inference endpoints are refused by design.

06

Publish the limits

The Claim Card carries intervals, baselines, coverage, parser failures, paired tests, evidence checks and the fingerprint of the evaluation.

Every model response keeps the exact final prompt, adapter, seed, temperature, raw output, parser decision, latency and token counts. The replay bundle is intentionally boring: CSV, JSON, Markdown, local paths and hashes. Reproducibility tends to look like paperwork.

There were three people in the painting. There was nobody in the room.

One small run found a distinction that the ground truth contained and the question did not.

The test used 98 photographs and Qwen3‑VL 4B. The canonical prompt was «Nella fotografia c’è più di una persona?» — “is there more than one person in the photograph?” The labels counted people physically present in the photographed scene. Painted and sculpted figures counted as no.

Nella fotografia c’è più di una persona?
98photographs
8readable false positives
8/8only depicted figures

All eight readable false positives showed figures in paintings, frescoes or sculpture, but no real person. The model returned valid JSON and no explanation, so I cannot claim to know its reasoning. I can only say that every readable false positive landed on the boundary the prompt forgot to define.

Under an ordinary reading, the model’s answer is defensible. A photograph of a painting with three people does contain three depicted people. The intended category was narrower: real people physically present when the photograph was made. That qualifier existed in my head and in my labels, not in the question.

This is an exploratory observation, not a model claim. It comes from one model, one source group, one wording and 98 photographs; nine other responses were unreadable. The next test is to compare the original against an explicit version that excludes paintings, photographs, statues and screens.

This is why prompt fragility is not just about synonyms. Sometimes a variant reveals that two perfectly reasonable readers were answering different questions. Before blaming the model, the benchmark has to say which world it means: the physical scene, or everything represented inside the image.

The audit once reported a clean dataset it had never looked at

The perceptual hash originally depended on sips, a macOS command. On Linux and Windows the decoder returned nothing. No hashes meant no visual comparisons; no comparisons meant zero near-duplicate pairs.

The interface showed the zero without explaining where it came from.

This was the worst possible bug for a project built around methodological honesty. The same demo folder produced 34 near-duplicate pairs on macOS and none on another operating system. The second answer looked reassuring and was false.

Zero can mean “nothing found”. It can also mean “nothing measured”.

The fix has two layers: Pillow now provides a decoder outside macOS, and the audit raises a high-severity warning whenever an image has no perceptual fingerprint. The result tile shows a dash, not a clean zero, when no comparison was possible.

High · visual analysis None of the 12 images has a perceptual fingerprint. The near-duplicate check compared nothing; zero pairs does not mean a clean dataset.
Same 12 intentionally repeated images. Decoder-specific hashes are never mixed in one cache.
demo datasetpairsunhashed
macOS · sips340
Linux · Pillow310
without decoder—12

The two decoders do not produce identical hashes. I measured that too: over 200 photographs they differed by 2.1 bits on average and by at most 7 bits out of 64. Identical-pair detection stayed close; the wider “same scene” boundary moved enough that pretending the engines were interchangeable would be dishonest.

So the engine is named in the audit and included in the feature-cache version. Comparisons across machines have to compare like with like. The near-duplicate threshold was not moved to make the fallback look better.

This section is here because the mistake says more about the software than another feature list would. FragileVision did not begin perfectly honest. It had to be audited by its own rule.

Some outputs are better left blank

A useful evaluation tool needs a vocabulary for “I do not know”, “I could not compare this” and “these samples are not independent”.

×

No verdict hidden inside ambiguous prose.If an answer contains more than one possible label, the parser records it as unreadable instead of choosing the convenient word.

×

No 100% score from one easy answer.Models in the Arena are ranked only on units every selected model answered readably. Coverage remains beside the score.

×

No random split through one scene.Source groups and near-duplicate clusters move together. If only one independent scene exists, the split is refused rather than fabricated.

×

No forced consensus.A tie remains a conflict. An adjudicator signs a new decision; the original reviewers are not overwritten.

×

No global model ranking.The Claim Card describes one model–dataset–prompt system. It does not turn a local observation into a universal league table.

This restraint has a cost. There are more dashes, warnings and insufficient-evidence grades than users expect from a dashboard. I think that is a feature. A blank cell with a reason is more informative than a polished number with no denominator.

Local first is an architectural constraint

The core has no required runtime dependencies. It is Python 3.11, SQLite and a static interface served on loopback. Pillow is optional on macOS and required elsewhere for the visual audit. The application rejects public model endpoints and does not send telemetry.

That choice is not branding. Evaluation datasets are often the very material that cannot be uploaded: unpublished photographs, internal screenshots, medical images, prototypes, client work. “Private by policy” is weaker than “there is no code path to the public API”.

Local installation
$ git clone https://github.com/VellBlue/fragilevision.git
$ cd fragilevision
$ python -m pip install -e ".[images]"  # Linux / Windows visual audit
$ python -m fragilevision

# then open http://127.0.0.1:7331

There is also a macOS bundle built from the same package. Windows and Linux use the Python entry point. The visual fallback has been exercised under a simulated Linux platform and in the test suite; it has not yet been run on a physical Windows or Linux machine. That distinction belongs in the article, not in a future bug report.

The tool can expose weak evidence. It cannot turn it into strong evidence.

FragileVision can tell you that your classes are imbalanced. It cannot choose a defensible population for you. It can find two frames that look almost identical. It cannot know whether the distinction matters to your domain. It can measure reviewer disagreement. It cannot repair a vague concept by statistics.

Heuristic

The Evidence Gate is a transparent checklist, not peer review and not a proof of validity.

Calibration

The 12-bit near-duplicate threshold was calibrated on one real photographic archive. Other domains should challenge it.

Scale

The current pairwise visual comparison is deliberately capped. Large datasets need a different indexing strategy.

Maturity

Version 0.15 is a research preview. The storage format migrates forward, but it may still change before 1.0.

There is a synthetic stressor and a rights-safe geometric dataset so the whole workflow can be explored without a model or private images. The demo intentionally triggers its own near-duplicate warning. It is there to show that the audit works, not to provide a benchmark result.

The larger ambition is modest: make it harder to publish a number without also publishing what the number could not see.

Every number should confess its limits.

FragileVision is usable now, inspectable line by line, and still early enough that disagreement is useful.

Read the code on GitHub ↗