Is that LLM failure real, or did you just get unlucky?

Your eval says 8 / 10 passed. That single number cannot tell you whether the prompt is broken or merely nondeterministic — and those need opposite fixes.

Get it on GitHub — MIT Have us do it — $150

The problem

Run an eval once and a failure is ambiguous. It might be a prompt that fails every time — a logic bug. Or one that works 85% of the time and happened to fail the once you looked — temperature, prompt sensitivity, retrieval nondeterminism.

Most tools run each case once. flakelens runs it N times and splits the two apart.

Why the interval matters

If a case passes 10/10, the tempting read is “100% reliable.” It isn’t. The usual normal-approximation interval agrees with that misreading — it collapses to zero width at 0/n and n/n, precisely where you’re asking the question.

flakelens uses the Wilson score interval, which doesn’t collapse. At 10/10 it reports a true pass rate of ≥ 72%. To honestly claim 95%+ you need 73 consecutive passes. The tool says so rather than letting you ship on ten green checkmarks.

Use it

pip install flakelens
flakelens cases.json --provider mymod:call --runs 10

Zero dependencies, Python 3.10+, 24 tests. Exits non-zero on broken cases so it drops straight into CI; --fail-on-flaky gates on flakiness too.

What it deliberately isn’t

• Not a semantic judge. Checks are deterministic — format, refusal, injection, hallucination markers. An LLM-as-judge is itself flaky, which is the thing being measured.
• Not a benchmark. Bring your own cases; 25 from real traffic beat 500 synthetic.
• Costs N× tokens. Ten runs means ten calls. Start with your 20 most important cases.

Done-for-you audit — $150