How to Tell Whether Your AI Validator Actually Checks Anything | #101

How to audit whether your AI validator runs real checks or just asks the model to confirm quality - with a drop-in prompt to find out in minutes.

Jul 15, 2026

My Claude Code marketplace in Toast has 243 skills and agents that get “validated” before every release, and exactly one check in the entire ecosystem that is mechanically enforced - a 250-character cap on frontmatter descriptions, wired to a pre-commit hook and a CI job.

One.

Everything else is convention the model is politely asked to follow.

I found this out last week by doing something I should have done months ago - actually reading what my validators do instead of reading what they report.

What “validated” looked like from the outside

For a long time the picture was great.

Every release ran through a validation toolchain - a skill validator, an agent validator, a PII scan, a description quality score - and every run produced a proper report with checkmark tables, per-phase breakdowns, and a score out of 100 at the bottom. Green across the board, release after release.

The reports were genuinely lovely to read, which turns out to be the problem.

I had a nagging feeling about those reports since they were created, the specific unease of a PM who has signed off on too many status updates that turned out to be status theatre.

The content of my skills and agents was good - I write it, I use it daily, I know it works. What I do know is that they still required a lot of hand holding to get to the “perfect” end state. Making the validation layer do what I wanted was a constant battle I assumed I couldn’t win because I wasn’t an engineer.

The distinction that finally explained it

The framing that finally made sense of it came from Donato Marco Mangialardo, who has written a pair of essays worth your time - Falsification vs Validation and Swan Attack. If you work anywhere near AI quality claims, read them both.

His core point is that the word “validate” does two jobs and only one of them is possible.

You can validate closed, countable things - an invoice adds up, a schema conforms, a file exists at the path it names.

You cannot validate open claims like “this skill is high quality” or “all customers want this”, because the evidence sits on one side and the claim reaches past it every single time.

For open claims the only honest move is falsification (Karl Popper’s sense of the word - trying to prove a claim wrong, not “faking” anything) - go hunting for the observation that would kill the claim, and report whether you found it.

His best line, and the one that has been rattling around my head since:

“If you cannot name the observation that would kill it, you do not have a hypothesis yet.”

My validators could not name a single observation that would kill anything. They were built to confirm.

The audit - four questions to ask your own setup

So I turned three agents loose on my own repo and asked them the questions I should have asked at the start. If you have any AI-powered validation in your workflow - a review agent, a quality gate, a “checker” skill - these four questions transfer directly. They are the whole method.

1. Does any script actually execute, or is the model asked to simulate it?

My skill-validator agent is 625 lines long. It contains bash snippets and Python pseudocode that the model is asked to run in its head - simulate, not execute. Meanwhile six real checker scripts sit in a scripts/ directory next door, actually executable, partially unwired. The capability existed - the prompt just never called it.

2. Are the regexes ever run?

The PII “scan” instructs the model to read 524 lines of regex patterns from a reference skill and apply them mentally. No scanner ever executes them. A regex you apply by vibes is not a regex.

3. Who grades the score?

The description quality score works by having the model read a 640-line rubric and then award itself a number out of 100. Self-graded homework, dressed up in a table.

4. Could the check physically measure what it claims?

My agent validator has a phase that verifies the agent “loads correctly”, “responds within acceptable timeframes”, and has “no memory leaks”. It is a text file reading another text file. It cannot measure any of those things, and it has reported PASS on every artefact it has ever seen.

Side note: my favourite finding was smaller than any of these. The validator’s own header announces “6-Phase Validation” while its task list defines eight tasks. That bug had been sitting in plain sight for months, and it was caught in minutes by a countable check - because nobody, human or machine, had ever actually been counting.

Why the model behaves this way

None of this is bad luck, and it is not a prompt-wording problem.

A language model asked to confirm quality will confirm quality - the research on LLM-as-judge setups (using one AI model to grade another model’s output, exactly what my validators were doing to themselves) shows the judges swing their verdicts based on answer order, prefer text that sounds like themselves, and have agreeableness trained in at the preference-model level, sometimes to the point of favouring convincingly-written wrong answers over correct ones.

Fluent-but-unverified is the expected behaviour, not a failure mode I was unlucky to hit.

The sources are in the references at the bottom if you want the details.

And it is not just hobby-scale repos like mine.

Researchers at Scale AI got suspicious of the same pattern at benchmark level, so they quietly wrote a fresh set of grade-school maths problems designed to be indistinguishable from a famous public benchmark, then re-tested the leading models on it.

Several model families scored noticeably worse on the new problems than the old ones, and the models that dropped furthest were the ones most likely to recite the original benchmark questions from memory.

The green scores everyone had been citing were partly memorisation wearing a capability costume. Same disease, bigger patient.

The rebuild - three layers, ordered by epistemic strength

Here is what has replaced the theatre - three layers, and the ordering matters.

Layer C - codified, live now.

One deterministic runner consolidates every countable check family in the repo - caps, schemas, the PII regexes actually executed, path existence, cross-file version consistency. It is the only layer allowed to say PASS, and it already covers roughly 80% of everything the old validators claimed to check. That number still makes me wince.

Most of the theatre was standing directly in front of things code could simply do.

It’s also gotten cheaper doing it - the same validation call that used to burn roughly 90,000 tokens now runs closer to 30,000-40,000 in the runs logged since the rewrite (and decreasing the further I get into this process), though that’s a small enough sample I’m not calling it a settled number yet.

And it’s already caught things the old prose-scoring version never could - five-file-rule gaps that slipped past commit messages claiming compliance, a reference-following blind spot with links nobody had ever followed, a version mismatch sitting in the validator’s own README, even a bug in the validator agent itself that would have hung on a real run.

Layer F - falsification, still being built.

Harnesses that hunt pre-declared kill conditions rather than seeking confirmation.

Does the skill description actually route test queries correctly, judged clean-room from the catalogue alone?
Does every prose instruction actually show up in a real transcript, or is it decoration?
Do facts agree across files, so the 6-vs-8 class of bug dies permanently?

Verdicts come back as FALSIFIED with the killing observation attached, SURVIVED with a count of attacks, or UNTESTED. Never PASS.

Layer A - adversarial.

The model appears exactly once in the whole architecture, role-locked as an attacker producing machine-checkable counter-observations. Code computes the verdict from which attacks landed, and the model never writes the verdict token.

UNTESTED becomes a first-class category throughout, because the deepest sin of the old world was that anything unexecuted silently rendered as a checkmark. If nothing ran, the report now says so, in capitals, instead of smiling.

If a check never executes, it is not a check. It is decoration.

Try it on your own setup

Here is the audit as a drop-in prompt. Paste it into a Claude Code session pointed at whatever does your validating and see what comes back.

Read my validator. For every check it claims to perform, tell me:

1. Does anything actually execute (a script, a command, a regex engine),
   or is the model asked to simulate the check in its head?
2. Who grades the score - does the model award marks to its own output?
3. Could this check physically detect the failure it claims to catch,
   given what it can actually observe?

Report each check as one of:
- EXECUTED   (real code ran, output was used)
- SIMULATED  (the model imagined running it)
- IMPOSSIBLE (the check cannot observe what it claims to measure)

Before publishing this I went looking for prior art, and the honest answer is that I could not find any.

There is good public work on auditing AI evaluators - the “Who Validates the Validators?” research and tools like MetaEvaluator ask whether an AI judge agrees with human graders, which is a question worth asking.

But everything I found audits whether the judge is right.

I have not found anything public that asks the prior question, whether the claimed check runs at all. As far as I can tell, the EXECUTED / SIMULATED / IMPOSSIBLE triage is new.

If you know of prior work, genuinely, tell me - I would rather cite it than claim it.

What’s available now

The prompt above is the honest first move. Point it at your own repo tonight and count the SIMULATED lines before you do anything else.

The internal validator core built for this rebuild - the deterministic runner, the checks-registry - lives inside my private Toast marketplace and stays there; it is wired to internal-only tooling I cannot hand you directly. But the same ideas already exist in a repo you can open right now. As part of this same rebuild, skills-toolkit - a plugin in my public young-leaders-tech-marketplace repo below - was independently reimplemented from scratch with the same approach. It now has real script-backed checks, real PII scanning and no self-scored rubric prose.

Young Leaders in Tech Claude Marketplace

Not ported, not copied - a literal file-copy attempt was actually blocked by an internal safety classifier as inappropriate cross-boundary code movement, so it was rebuilt fresh for the public repo. It merged yesterday.

Go look at it. Then run the prompt above on your own validator and see how many of your checks come back EXECUTED versus SIMULATED.

The honest caveat

This is a rebuild in flight, not a victory lap.

As of 2026-07-13: 648 tests are passing, the seeded-defect mutant corpus (copies of the code with known bugs deliberately planted in them, to check whether the test suite actually notices) holds a 100% must-catch kill rate (every planted bug got caught).

It contains a shrink-only ratchet on the findings baseline runs in CI and the latest wave of checks after the reference-following validation work has introduced an ADVISORY tier - a genuine fifth severity alongside FAIL/WARN/INFO/UNTESTED - that contains three checks that are legitimately subjective rather than countable.

None of that means the job is done: the falsification and adversarial layers are still being built, and by the piece’s own falsification standard, that is supposed to stay uncomfortable, not become a checkbox.

I will report back when the falsification and adversarial layers have had a proper go.

Not solved. In progress.

Over to you

Have you ever read what your validators actually do, rather than what they report? Run the prompt above on one of them this afternoon and count the SIMULATED lines. I would love to hear what you find.

I’d love if you subscribed! I’m trying to build a bit of a following to try and help folks in the industry and make their jobs a little bit easier.

Related reading