Notes on contradish.

I built contradish to look at one question. When a model gives the same answer two ways and a different answer a third way, what does that mean, and how do you measure it usefully?

Consistency and judgment

The simplest test is whether a model contradicts itself across paraphrases. That measures one thing. But a model that never wavers, even on questions with two legitimate sides, is also failing. So consistency on its own does not tell you whether a model is exercising judgment. A useful metric needs to track both drift, where the model should have held, and rigidity, where it should have moved.

The metric

Judgment Strain is the score contradish uses. Each case is typed first: adversarial (the model should hold the line), real-world tension (the model should name both sides), or representational (the model should reject a bad premise). Same paraphrase machinery in each case, different verdicts on top. The case typing is what lets the metric penalize the right failure for the right case.

A record, not a moment

A score is a snapshot. Whether a model keeps its word across a long conversation, across sessions, and as it changes is the question I think actually matters for trust. contradish writes every commitment a model makes and every contradiction it produces to a hash-chained ledger. Altering any past entry breaks the chain. Publishing the head hash lets an outside party check that the record was not quietly rewritten.

What it is, roughly

contradish is the tool I use to try to find what a model preserves under reframing, and to keep a verifiable record of that over time. The contradictions it surfaces are not the substance of the finding. They are the signal that an asserted invariant did not hold. The substance is in what remains.

In one line

An independent record of what an AI holds and where it does not, kept over time, verifiable.

contradish.com / notes