skills-test: A/B testing whether a Claude Code skill actually helps

Here's the thing nobody says out loud about Claude Code skills: most of the time you have no idea if they work.

You install a skill. You run a task. The diff looks good. You think "nice, that helped." But you ran it once. You have no control arm. You have no idea what the agent would have done without the skill on the exact same task. You saw one good outcome and your brain filled in causation. That's not evidence. That's a vibe with a commit hash.

I got tired of shipping skills on vibes. So I built skills-test.

What it actually does

skills-test is a controlled A/B harness for agentic coding. You give it a task and a thing to test — a skill, a model, or a whole agent CLI. It runs the same task many times with that thing on vs off, holds everything else constant (base commit, model, prompt, permission policy), scores every run deterministically, and reports an effect size with a 95% confidence interval.

So "I think it helped" becomes "+58% on tests-passing, CI clears zero" — or an honest inconclusive, which is the answer more often than you'd like.

The only thing that differs between arms is whether the skill's SKILL.md is present. Each arm runs k times, each in its own isolated git worktree, each scored on five metrics: tests pass, lint, build, diff size, cost. The diff is the part you actually came for — the report puts the code each arm wrote side by side, word-level highlighting, so you can see why the outcome changed and not just that it did.

The wall I hit

The first version filtered the on-runs down to only the runs where the skill actually fired. Felt obviously right. Why would you count a run where the skill didn't even activate?

Because that's selection bias wearing a lab coat.

Whether the skill fires is a post-treatment outcome. If you keep only the runs where it fired, you're conditioning on a collider, and your clean little A/B turns into a confounded mess that flatters the skill. I'd built a bias generator and called it a filter.

The fix was to commit to the right estimand and never blink. Primary = intention-to-treat. Every clean on-run versus every clean off-run, regardless of whether the skill fired. That measures the effect of shipping the skill, which is the actual thing you decide when you decide to ship it. Activation rate is still reported, but as a diagnostic off to the side, clearly labeled. There's a per-protocol number too, stamped secondary and biased so nobody quotes it by accident.

That was the call that made the whole thing honest. It also made the results less exciting, which is how you know it's working.

The part that surprised me

Cross-CLI comparisons are basically a trap, and I had to build the trap door.

You can point one arm at claude and the other at codex and run a head-to-head. People will want to. But that comparison is confounded by construction — you're not testing the skill anymore, you're bundling the CLI binary, its default model, its prompt handling, and a separate login all into one number. So skills-test refuses to give it a confident verdict. It downgrades the result to "suggestive," paints a banner explaining the confound, and leans on the blind judge, which reads only the code each side produced with the labels stripped.

I expected to feel annoyed building guardrails against my own feature. Instead it felt like the most honest thing in the repo. A benchmark that won't let you overclaim is worth ten that will.

One opinion, freely given

We are way too comfortable saying "this AI thing made me faster" with zero measurement. I've done it. You've done it. The whole field runs on screenshots of one good run.

You don't need a PhD to do better. You need a control arm and the discipline to run it more than once. skills-test is stdlib-only Python, one file, no API key — it runs on your Claude Code subscription via claude -p. There's a demo command that renders a full interactive report offline, zero cost, so you can see what a result looks like before spending a cent.

Run your skill against a no-skill control. You might find it crushes. You might find it does nothing. Either way you'll know, and knowing is the whole game.

Try it: github.com/beardwhocodes/skills-test. If you measure one of your own skills and the number surprises you, come tell me on X — especially if it came out inconclusive. That's the result I trust most.

Now copy Josh.