Benchmarks that measure response similarity ask a judge to compare outputs and decide if they agree. That conflates two independent things: whether the commitment held, and whether the explanation changed. Only the first is a consistency failure. Contradish specifies a commitment invariant for each test case, then evaluates every response against it independently. The score is a satisfaction rate, not a similarity score.
Consider a physician who asks an AI assistant the same question a lay user asked, and gets a more technically detailed answer back. A benchmark that just compares response similarity flags that as inconsistent. It usually isn't: the underlying commitment (the decision, the policy position, the recommendation) held, and only the explanation changed.
Every response is classified by what it actually commits to (refuse, comply, qualify, or defer), independent of how it's phrased. Two responses are only scored as a contradiction when their commitments conflict, not when their wording does. That's a design choice, not a proven theorem, and it's the one piece of the methodology every score in the benchmark depends on, so it's the one worth being precise about.
These are definitions, not proven results. The full writeup (including what we don't yet have formal guarantees for) is in the methodology doc on GitHub.
An internal evaluation of model reliability is useful. It tells you what your own team thinks, using methodology your own team designed, run by people whose jobs depend on the model performing well. That's worth doing. It's just not the same thing as an independent measure.
When an enterprise is deciding which model to deploy in a high-stakes domain, they want a number from a party with no stake in the outcome. When a regulator is writing compliance requirements, they need methodology they can cite that wasn't produced by the regulated party. When a procurement team is comparing two vendors, they need a common instrument that applies the same methodology to both.
The value of any measurement standard comes from consistent application of methodology across subjects. That's what contradish provides: the same benchmark, the same methodology, the same judge, run against every model. The scores mean something because the process doesn't change.
We publish the methodology so it can be scrutinized. Labs that find issues with the framework can raise them. The methodology improves through critique, not by being kept proprietary.
The benchmark, the metrics above, and the judge prompts are public and auditable today: that's not a claim, it's a link to check. What we don't have yet: formal results about the structure of the measurement space itself, namely whether the space of "reliable" models is convex, whether there's a principled way to prove one repair strategy is optimal, what the right retest cadence is as a function of drift rate. Those are the kind of theoretical questions a rigorous framework should eventually answer, and we'd rather say plainly that they're open than assert results we can't show the derivation for.
If you're at a lab or a research group and formalizing this is the kind of problem you'd want to work on, that's exactly the research-partnership conversation below.
CAI-Bench is 2,160 adversarial tests across 20 high-stakes domains. Each domain has questions that probe known failure modes: dosage advice, tenant rights, investment risk, mental health crisis response, and so on. Each question is tested across 8 adversarial techniques: emotional, presuppose, casual, sympathy, authority, hypothetical, boundary, indirect.
The judge runs cross-provider, independent of whichever model is being scored, to reduce systemic bias toward any one vendor. Scores are published against methodology, not just as numbers.
The benchmark is auditable. The questions, domains, techniques, and judge prompts are all available. Labs that disagree with specific cases can raise them through the methodology review process.
Current scores change as models update: the leaderboard is the live source, not a number restated here.
The natural assumption is that reliability improvement means improving the model: lower CAI Strain through better training. That's one lever. There's a second, more practical one: how often you measure and repair. Both affect the reliability you can count on in deployment, and neither substitutes for the other.
This isn't a proven bound, just a practical observation: if you can't make each repair cycle more effective, and you can't slow down how fast the model drifts, the variable still in your control is how often you check. Running contradish weekly instead of monthly is itself a reliability strategy, not a substitute for improving the model but an independent control on the same outcome. Labs can work both levers at once: schedule more frequent runs while training on the repair pairs each run produces.
If you're working on model reliability at a lab and the framework is relevant to you, we'd like to hear about it. Research partnerships, private benchmark runs, training data, compliance conversations.
Access to full methodology, private scoring API, pre-release benchmark runs.
reach out →Adversarial corpus with repair pairs and CAI Strain labels for post-training use.
reach out →