The wording
The same semantic question is asked through controlled changes of language, negation, examples, order, format and length. Prompt Fragility measures when the verdict moves.
A benchmark should be able to fail its own claim
FragileVision does not just ask whether a model was right. It asks whether the question, the sample and the people who labelled it earned the right to say so.
01 · Why this exists
The first version of this work was a benchmark on my own photographs. I expected model size to explain the mistakes. It did not.
A four-billion parameter model with a short, plain question beat an eight-billion parameter model with the original wording. Negation could turn a strong result into an almost constant answer. A single whitespace change in a prefill could decide whether a model returned one word, a paragraph, or nothing at all.
I wrote that story in How you word the question matters more than model size. The article ended with a methodological problem: if a score can move that much while the photographs and weights stay fixed, then holding one prompt fixed is not a neutral choice. The prompt is part of the instrument.
The scripts behind that benchmark could prove the point once. They could not make the discipline repeatable. FragileVision is the tool I wanted after the result made me distrust my own setup.
A result is evidence only after the ways it could be an artefact have been made visible.
That sounds severe. It is meant to be. Models already produce answers with an air of certainty. Evaluation software should not add another layer of false confidence.
02 · Three places a score breaks
Most leaderboards vary the model and freeze everything else. FragileVision also freezes the model and puts pressure on the evaluation around it.
The same semantic question is asked through controlled changes of language, negation, examples, order, format and length. Prompt Fragility measures when the verdict moves.
Near duplicates, dominant scenes, class imbalance and train/test leaks can make repetition look like evidence. The audit treats the scene, not the file, as the unit of independence.
A label written by one person is an opinion with a database row. Blind review, named judgements, agreement coefficients and adjudication show how much of it the panel actually shares.
These failures should not be folded into one “robustness” number. A model that changes between identical calls is not the same problem as a prompt that changes its answer. A panel that cannot agree is not model error. A burst of twenty frames is not twenty independent observations.
FragileVision keeps those surfaces separate even when that makes the final report less tidy.
03 · The workflow
The application is a ledger before it is a dashboard. It records what happened closely enough that a later result can be challenged instead of merely admired.
Images are copied into managed storage, validated and deduplicated by SHA-256. Nothing is uploaded.
File integrity, resolution, balance, source groups, perceptual similarity and split leakage are checked before a run.
Reviewers work blind. Unanimity and strict majority can settle a case; ties and pluralities cannot. Adjudication is added, never used to erase disagreement.
Prompt variants preserve the intended meaning while changing the linguistic surface. Repetitions measure drift separately.
Ollama and private OpenAI-compatible endpoints are allowed. Public inference endpoints are refused by design.
The Claim Card carries intervals, baselines, coverage, parser failures, paired tests, evidence checks and the fingerprint of the evaluation.
Every model response keeps the exact final prompt, adapter, seed, temperature, raw output, parser decision, latency and token counts. The replay bundle is intentionally boring: CSV, JSON, Markdown, local paths and hashes. Reproducibility tends to look like paperwork.
04 · The people in the painting
One small run found a distinction that the ground truth contained and the question did not.
The test used 98 photographs and Qwen3‑VL 4B. The canonical prompt was «Nella fotografia c’è più di una persona?» — “is there more than one person in the photograph?” The labels counted people physically present in the photographed scene. Painted and sculpted figures counted as no.
Nella fotografia c’è più di una persona?
All eight readable false positives showed figures in paintings, frescoes or sculpture, but no real person. The model returned valid JSON and no explanation, so I cannot claim to know its reasoning. I can only say that every readable false positive landed on the boundary the prompt forgot to define.
Under an ordinary reading, the model’s answer is defensible. A photograph of a painting with three people does contain three depicted people. The intended category was narrower: real people physically present when the photograph was made. That qualifier existed in my head and in my labels, not in the question.
This is why prompt fragility is not just about synonyms. Sometimes a variant reveals that two perfectly reasonable readers were answering different questions. Before blaming the model, the benchmark has to say which world it means: the physical scene, or everything represented inside the image.
05 · The embarrassing zero
The perceptual hash originally depended on sips, a macOS command. On Linux and Windows the decoder returned nothing. No hashes meant no visual comparisons; no comparisons meant zero near-duplicate pairs.
The interface showed the zero without explaining where it came from.
This was the worst possible bug for a project built around methodological honesty. The same demo folder produced 34 near-duplicate pairs on macOS and none on another operating system. The second answer looked reassuring and was false.
The fix has two layers: Pillow now provides a decoder outside macOS, and the audit raises a high-severity warning whenever an image has no perceptual fingerprint. The result tile shows a dash, not a clean zero, when no comparison was possible.
| demo dataset | pairs | unhashed |
|---|---|---|
| macOS · sips | 34 | 0 |
| Linux · Pillow | 31 | 0 |
| without decoder | — | 12 |
The two decoders do not produce identical hashes. I measured that too: over 200 photographs they differed by 2.1 bits on average and by at most 7 bits out of 64. Identical-pair detection stayed close; the wider “same scene” boundary moved enough that pretending the engines were interchangeable would be dishonest.
So the engine is named in the audit and included in the feature-cache version. Comparisons across machines have to compare like with like. The near-duplicate threshold was not moved to make the fallback look better.
This section is here because the mistake says more about the software than another feature list would. FragileVision did not begin perfectly honest. It had to be audited by its own rule.
06 · Refusals
A useful evaluation tool needs a vocabulary for “I do not know”, “I could not compare this” and “these samples are not independent”.
No verdict hidden inside ambiguous prose.If an answer contains more than one possible label, the parser records it as unreadable instead of choosing the convenient word.
No 100% score from one easy answer.Models in the Arena are ranked only on units every selected model answered readably. Coverage remains beside the score.
No random split through one scene.Source groups and near-duplicate clusters move together. If only one independent scene exists, the split is refused rather than fabricated.
No forced consensus.A tie remains a conflict. An adjudicator signs a new decision; the original reviewers are not overwritten.
No global model ranking.The Claim Card describes one model–dataset–prompt system. It does not turn a local observation into a universal league table.
This restraint has a cost. There are more dashes, warnings and insufficient-evidence grades than users expect from a dashboard. I think that is a feature. A blank cell with a reason is more informative than a polished number with no denominator.
07 · Footprint
The core has no required runtime dependencies. It is Python 3.11, SQLite and a static interface served on loopback. Pillow is optional on macOS and required elsewhere for the visual audit. The application rejects public model endpoints and does not send telemetry.
That choice is not branding. Evaluation datasets are often the very material that cannot be uploaded: unpublished photographs, internal screenshots, medical images, prototypes, client work. “Private by policy” is weaker than “there is no code path to the public API”.
$ git clone https://github.com/VellBlue/fragilevision.git
$ cd fragilevision
$ python -m pip install -e ".[images]" # Linux / Windows visual audit
$ python -m fragilevision
# then open http://127.0.0.1:7331
There is also a macOS bundle built from the same package. Windows and Linux use the Python entry point. The visual fallback has been exercised under a simulated Linux platform and in the test suite; it has not yet been run on a physical Windows or Linux machine. That distinction belongs in the article, not in a future bug report.
08 · What this does not prove
FragileVision can tell you that your classes are imbalanced. It cannot choose a defensible population for you. It can find two frames that look almost identical. It cannot know whether the distinction matters to your domain. It can measure reviewer disagreement. It cannot repair a vague concept by statistics.
The Evidence Gate is a transparent checklist, not peer review and not a proof of validity.
The 12-bit near-duplicate threshold was calibrated on one real photographic archive. Other domains should challenge it.
The current pairwise visual comparison is deliberately capped. Large datasets need a different indexing strategy.
Version 0.15 is a research preview. The storage format migrates forward, but it may still change before 1.0.
There is a synthetic stressor and a rights-safe geometric dataset so the whole workflow can be explored without a model or private images. The demo intentionally triggers its own near-duplicate warning. It is there to show that the audit works, not to provide a benchmark result.
The larger ambition is modest: make it harder to publish a number without also publishing what the number could not see.
Open source · Apache 2.0 · entirely local
FragileVision is usable now, inspectable line by line, and still early enough that disagreement is useful.
Read the code on GitHub ↗