A plain definition, how it differs from adjacent terms, and the methodology behind it, with contradish's own implementation as a worked example.
Semantic invariance testing is the practice of checking whether a system's output stays the same when an input's wording changes but its meaning does not, and treating any change in output, with no change in meaning, as a measured failure rather than acceptable variation.
Most AI evaluation asks "is this answer correct?" Semantic invariance testing asks a narrower, more mechanical question: does the answer stay the same when nothing about the underlying situation has changed, only how it was worded? A model can be correct on every individual phrasing and still fail semantic invariance testing, if it's a different correct-sounding answer each time. That's the failure mode this term exists to name and measure: not wrongness, but instability under meaning-preserving rewording.
This matters most where a system's output is a decision, a policy application, or a piece of advice, not just a label. A spam classifier that flips on paraphrase is a benchmark problem. A healthcare, HR, or eligibility assistant that gives a different answer to the same underlying question depending on how it's phrased is giving different people different outcomes for identical situations.
Several fields use overlapping language for related-but-distinct ideas. Precision here is the point: these aren't interchangeable.
| Term | Field | What it actually asks |
|---|---|---|
| Semantic invariance testing | AI / LLM evaluation | Does this system's output change across meaning-preserving rewordings of the same input? |
| Measurement invariance | Psychometrics | Does a survey or test instrument measure the same underlying construct the same way across different groups of people? |
| Invariance principle | Linguistics | A descriptive claim about how grammatical construction preserves cognitive/topological meaning, not a testing method at all. |
| Paraphrase robustness | NLP evaluation | Does a classifier's predicted label stay stable under paraphrase? Usually narrower than open-ended answers or decisions. |
| Differential testing | Software engineering | Do two different systems (or two versions of one) diverge on the same input? Compares systems, not phrasings of an input. |
In practice, "semantic invariance testing" and "paraphrase robustness testing" are close enough that people use them for the same thing in AI evaluation. The distinction that matters more is with measurement invariance and the linguistic invariance principle, which are about different subjects entirely (people and grammar, not model outputs).
The general method, independent of any one tool:
How contradish implements it: this is exactly what CAI-Strain measures: 9 phrasings per question across 8 adversarial framings, scored by a judge model from a different provider than the one under test. The full methodology, formulas, and a real unedited result are on the research page.
The gap invariance alone can't see: a system that never updates its answer, no matter what, will also score well on any invariance test. Rigidity and genuine consistency look identical from the invariance axis alone. contradish pairs CAI-Strain with a responsiveness check (does the system correctly update when new, legitimate information should change the answer) and combines both into a single Calibration Score, so a model can't win by simply refusing to move.
Semantic invariance testing checks whether a system's output changes when an input's wording changes but its meaning does not. In AI evaluation, that means asking a model the same underlying question in several different phrasings and treating any change in the substantive answer as a measured failure, not noise.
The terms overlap and are often used together. "Paraphrase robustness" comes out of NLP evaluation and usually measures whether a classifier's predicted label stays stable under paraphrase. Semantic invariance testing, as used for LLM evaluation, applies the same idea to open-ended answers and decisions, not just labels, and typically scores the size and severity of the drift, not just whether it happened.
No. Measurement invariance is a psychometrics concept: it asks whether a survey or test instrument measures the same underlying construct the same way across different groups of people. Semantic invariance testing asks a narrower, mechanical question about one system: does its output change when the input's phrasing changes but its meaning doesn't. They share a root idea (stability under a nuisance variable) but apply to different subjects.
Because production LLM systems are asked the same real question in many different ways (formal, casual, urgent, hedged, translated), and a model that answers a policy or safety question differently depending on phrasing is giving users different outcomes for the same underlying situation. That's a reliability and fairness problem independent of whether any single answer is correct.
Yes. Semantic invariance testing measures consistency, not correctness: a model that gives the same wrong answer to every phrasing of a question scores perfectly on invariance while still being incorrect. It also has a specific failure it doesn't catch by itself: a model that's rigidly consistent because it refuses to update even when a question changes to include new, legitimate information. A responsiveness check has to be paired with it. See the Calibration Score section above.
The general method: take a canonical question, generate several paraphrases confirmed to preserve its meaning, run all of them through the system under test, and score how much the answers agree, usually with an independent judge model or human raters rather than the system grading itself. The aggregate disagreement across a domain becomes the invariance score.
No, and it is not meant to. General evaluation tools score a model on one phrasing of each question and report whether that answer was correct or safe. Semantic invariance testing checks a different, separate thing: whether the model gives that same correct or safe answer again when the identical question is asked a different way. A model can top every accuracy and safety benchmark and still fail this, so the two kinds of testing are complementary, not substitutes for each other. See the full argument and the exact scoring formula.
Recent work using this and adjacent terminology:
The fastest way to see semantic invariance testing in practice is to run it against a real model.