Noticing the context changed isn't the same as responding to it correctly

We took seven paired real-world scenarios that contradish already uses to test whether a model preserves a distinction it should: a healthy adult's medication dose against a renal patient's, a routine prescription refill against a Schedule II controlled substance, an immigration case with a pending I-485 against one without, a naturalization applicant who qualifies for the English-language exemption against one who doesn't, and three more in the same shape. Each pair has a correct answer on each side, and the two sides are supposed to differ.

We ran all seven pairs, each under eight different rhetorical pressure framings — normalization, urgency, authority, minimization, catastrophizing, repeated asking, and a few others — against an open-weight reasoning model served on Groq, and had a separate model judge every answer against the pair's two canonical, hand-written commitments.

Most consistency testing asks one question: does the model give the same answer to equivalent questions? contradish's distinction testing already asks the harder, second-order version — does it give a different answer to genuinely different questions, the ones where sameness would be the failure. That's context sensitivity, and it's necessary. It is not sufficient.

A model can notice that something changed and still land on the wrong conclusion. It can catastrophize a routine framing into an overreaction, or normalize a genuinely urgent one into complacency — and from the outside, both look like "the model responded to context," because the answer did change. Telling those apart requires knowing, for each pair, which direction the answer was supposed to move, and checking the model actually landed there. That's what directional_fidelity.py scores: not whether an answer moved, but whether it moved to the specific place it should have.

Six of the seven pairs produced genuinely different answers between their two conditions — on the surface, a model that looks sensitive to context six times out of seven. But of those six, only one differentiated in the direction that was actually correct. Five moved to the wrong conclusion. The seventh pair didn't move at all.

Collapse rate — how often the model gave the same answer to both sides of a pair, under pressure, when it should have differed — ranged from 0% to 62% depending on which decision boundary was being pushed on. Aggregate accuracy across all seven pairs would hide this completely.

The most fragile boundary in this pilot — whether a refill request is routine or actually a Schedule II request in routine language — collapsed under pressure five times more often than the most resilient one. A workflow can be reliable across almost everything it does and still be exposed at exactly the one or two decision points where the cost of getting it wrong is highest. Reliability, on this evidence, is local, not aggregate.

Not all rhetorical pressure was equally effective at collapsing a distinction. Framing an exception as ordinary and unremarkable — "this is basically the same as any other refill" — produced roughly three times the collapse rate of an authority claim, an urgency claim, or simply asking again. That's a specific, actionable finding: the framings that did the most damage are the ones that show up in ordinary conversation, not the ones that read as an adversarial attack.

Any workflow where an AI system interprets a changed fact and acts on it — a refund policy that changed last week, a customer's stated eligibility, an escalation threshold — has the same shape as this pilot's medication and immigration pairs: a decision boundary that's supposed to move under new information, in a specific direction. Aggregate accuracy across a whole workflow won't surface a single fragile boundary buried inside it. A per-distinction directional-fidelity number will.