When the AI is wrong, catch it by asking

AI does not say "I do not know" when it does not know. It says something plausible. Every time I write about learning through Socratic dialogue the same objection comes back: what if you learn something wrong? Fair. And it is exactly why the method matters more, not less. A wrong answer does not survive a few questions.

Two parts here. Four questions that expose a wrong answer inside the conversation, and three cases my company actually caught while putting AI to work.

Four questions a wrong answer cannot survive

One. What is the basis for that? A right answer cites something; a wrong one invents something. An invented source collapses on the next question. Which section of which document? Where did that number come from?

Two. What would someone arguing the opposite say? Few questions have only one side. Make the AI argue the other side and you see how solid the first answer was. If the opposite is more convincing, doubt the first.

Three. You said something different earlier. Same as the fourth of the seven questions. Long conversations drift into contradiction. Point at the gap and one of the two has to be wrong.

Four. How can I check this myself? The most important one. An answer that comes with a way to verify it can be verified. An answer that cannot name one is itself a signal.

Three we actually caught

My company runs several AI agents and keeps a record of everything they do. These three are in that record.

First, an invented permission. Attaching a feature to a service required a permission, and the AI named one. It sounded right. The admin screen did not have it. Asked again, two similarly named permissions came up, and the one we needed was one of those. Had we asked "what is the basis" first, it would have been caught before opening the screen. What caught it was "how do I check this". The screen not having it was the check.

Second, a claim without measurement. The AI said that adding a certain word to a company name would make it read like a different industry. Plausible, and I thought so too. Asked to argue the opposite, it suggested counting companies with that word in their name. We counted. More than a hundred of them, all in the same industry as ours. The first answer was an impression; the record said the opposite. What caught it was "argue the other side".

Third, a wrong measurement reported as a failure. Asked to confirm that visitor statistics were being collected, the AI measured with a one-line command and reported that collection had stopped. In fact that service only injects its tracking snippet for real browsers, so the command line always sees zero. The statistics had been accumulating the whole time. What caught it was "you said something different": a few days earlier it had said collection worked, now it said it did not, so what changed? What had changed was the way of measuring.

The three share something. The answer was not what went wrong; the way the answer was produced was. Invented, unmeasured, or measured too narrowly. All four questions are questions about that method.

Verification happens outside

Catching inside the conversation and measuring outside it are different things. The four questions create doubt; they do not confirm. Confirmation comes from the original text, the official documentation, an actual run, the admin screen.

So we keep one rule. When an AI says "this works", that is a claim. Only what has actually been opened or run counts as a result. All three cases above were caught by that rule.

Why I still use the method

Talking with a fallible partner beats having no one to ask. People are wrong too. But it is hard to ask a person for their basis, to make them argue the other side, to tell them they contradicted themselves. With an AI those questions cost nothing.

So the questions are the verification. Someone who asks once and believes is in danger. Someone who keeps asking is fine even in the presence of wrong answers. A wrong answer does not survive a few questions.