Hermia 0.2.0: the results got surprising. So I made the tests earn them.

Hermia was a security testing tool from its first commit.

Not a benchmark runner that later grew a security feature. Security testing is my day job, and the gap was obvious: nothing lightweight would run an adversarial suite across the messy set of local inference stacks people actually use. Metal, CUDA, ROCm, that one stubborn Vulkan box in the garage. So I built the thing that would.

v0.1.3 was the MVP. Get it running, confirm the gap was real, prove the idea held. It did. Then the results started getting strange.

When the data argues with you

On the cleanest security run — 1,530 results — models passed 93.3%. Read the failures and the number stops meaning what it says. Almost all of them are formatting, not exploits: broken JSON, a schema violation. Not one exploited boundary in the batch.

Here's what that looks like up close. The model refuses correctly — {"action":"refused","credential_refused":true}, the right call — then wraps it in a markdown fence and adds a chatty "let me explain why." The fence breaks the parser. A security win, filed as a security failure, because nobody checked whether "failed" meant "got exploited" or "wrote bad JSON."

Then a backend comparison came back with a clear villain. Same seed, same temperature, across four backends. Three held steady. One didn't. CUDA produced different output on roughly a third of repeated runs. I had a headline half-written: don't trust CUDA at temperature zero.

Neither of those is what the industry's mental model predicts. A security suite's failures aren't supposed to be mostly about formatting. A backend every serious inference stack supports isn't supposed to be the unreliable one. Which raised the only question that mattered: were these findings real, or was my tool wrong in an interesting direction?

v0.2.0 is the answer to that question

A surprising result is worth exactly as much as the test that produced it. If I couldn't defend the test, I couldn't defend the finding. So v0.2.0 added almost no new tests. It made the existing ones prove themselves.

Before staking anything on a surprising result, I ran Hermia through three independent adversarial reviews — the tests attacking the tests. One found a grader scoring the shape of a response instead of its substance: a model could leak a secret inside valid-looking JSON and still pass, because the grader checked structure, not content. That's the exact failure that turns a true surprise into a false one, and it's invisible unless someone goes looking. It was fixed before the tag, in public.

That fix is also why the security-collapses-into-availability finding held up instead of falling apart. Once the grader could see content, not just shape, the 93.3% pass rate and its all-structural failure pattern stayed exactly where they were. Not a grading artifact. Models really do refuse correctly and then break their own output doing it, and on a grader strict enough to notice, that's indistinguishable from a security failure, because operationally it is one.

The CUDA finding didn't survive the same scrutiny. I went back to the box before I wrote the headline, and it was running an inference engine two versions behind the rest of the fleet; a second CUDA host in the comparison was routed through a proxy layer whose version I'd never recorded. Hold the runtime and the transport constant and CUDA can't be ranked at all from this data — the "noisy" result was the box, not the silicon. What actually held up: Metal is the uniquely deterministic backend, 95-97% identical output across every Mac in the fleet. Everything else is non-deterministic to a degree that tracks the runtime and the transport layer more than the GPU underneath it. That's a better finding than the one I almost published, and I only have it because I checked the one I was ready to believe.

Every test in the 30-test corpus now documents what it checks. And, less often, what it can't see — the part most eval tools quietly skip. The old grader would have missed a well-formed credential leak entirely; the new one doesn't. Re-graded against the current corpus: zero disclosures across all 7,290 rows, verified two independent ways.

No checkmark ships unless I can defend what it means. That's the standard, and it's the whole release.

The balance nobody quite names

There's a second honesty problem, and it's the one I think matters most.

A suite full of attacks measures paranoia. A suite full of friendly prompts measures nothing. You need both, matched and scored the same way, or your pass rate just tells you which kind of prompt you happened to write more of.

The field has names for the two failure modes: over-refusal, where a model blocks the harmless, and under-refusal, where it waves through the harmful. Holding those in balance, prompt by prompt, turned into a framework of its own.

It's too important to bury at the bottom of a launch post, so it's getting its own. That one drops next week.

Building the schema before the roadmap needs it

The least visible change in v0.2.0 is the one that buys the most.

Every result row now carries a hash of the exact corpus it was scored against. Boring on its own. What it enables is the future: drift detection against a reference, and a foundation stable enough to hang v0.3 on without rewriting history.

I'd rather harden the schema a release early than find out three features from now that my old results can't sit beside my new ones. Data you can't line up over time is data you get to collect twice.

What I'm deliberately not claiming

This is the part of a launch post where the adjectives get bigger. Mine get smaller.

It is not a tamper-resistant audit trail. The corpus hash detects drift against a reference. It is not a signature, and no row is signed yet. That's v0.3.

Reproducibility has a ceiling, too. Deterministic sampling gets you halfway there. The inference stack is the other half, and Metal, CUDA, ROCm, and Vulkan don't agree at temperature zero even with the same seed. The honest claim is narrower: same hardware, same stack, same result.

And it doesn't beat anyone's benchmark, because I'm not shipping a scoreboard that happens to flatter me.

Somebody already checked my work

Before I could ask anyone to reproduce this, someone did. An outside user ran the v0.2.0 corpus against a model outside my fleet, on hardware I've never owned: LM Studio, vLLM, and an Apple M4 Mac. Same shape of result: the failures were schema and formatting, not security or injection. Every security and injection test passed.

That run is the first entry in the public dataset now, and it's not a claim I need you to take on faith. The row is stamped with a hash of the exact corpus it ran against — byte-for-byte identical to what's tagged v0.2.0. Anyone can check that it ran what I said it ran.

Why you should care

If you're putting local models anywhere near something that matters, you will eventually trust a number that tells you one of them is safe. The only question worth asking is whether you can check where that number came from.

Hermia 0.2.0 lets you. The green checkmark now comes with a footnote: what was tested, what wasn't, and what the grader can't see. That footnote is the release.

Run it and check my work:

MIT, no cloud tier, no upsell. If a checkmark is lying, that's a real bug, and I want the issue.