Before you blame the AI

You gave the AI your files, asked a question and got a wrong answer stated with total confidence. It's tempting to decide the tool is useless or to switch to another one. Before you do, run these five questions. In everyday use, a lot of "it made it up" turns out to start with what it was given: the answer wasn't there, an old version was, or the key part was buried.

Think of an office with three versions of the travel policy in one shared folder, none marked as current. An assistant that confidently summarises the 2022 version isn't broken. It found a policy and read it correctly. The folder was the problem.

The five questions

Was the answer in what I gave it? Search your own material for the fact. If it isn't there, the AI filled the gap from general knowledge or a guess.
Did I give it the current version, and only that version? Old drafts, superseded policies and duplicate files compete with the right one. Remove them or label them clearly.
Was the key part buried? In a long paste, put the part that matters first or last, and say where it is: "The rule I'm asking about is in section 4."
Did it actually use it? Ask it to quote the exact line its answer rests on, then find that line yourself.
Did it mix in outside knowledge? Ask it to mark which parts came from your material and which from general knowledge.

Ticks aren't saved anywhere.

Only after all five, consider that you need a better brief, a fresh chat or a different tool. If the brief was the problem, The complete brief helps. If the chat has run long and drifted, Start fresh.

Diagnose a wrong answer

Your last answer said: [the claim that looks wrong].
1. Quote the exact line or lines from the material I gave you that this rests on, and name the file or section.
2. If no line supports it, say so plainly.
3. List any other places in my material that say something different on this point.
4. Mark which parts of your answer came from my material and which from general knowledge.
Don't defend the answer. I'm trying to find where it came from.

Worked example: the wrong refund window

Jess runs a small online shop and uploads her policies to an assistant's project so it can draft customer replies. A draft tells a customer they have 14 days to return an item. Her policy is 30 days. She runs the diagnose prompt. The assistant quotes a line from "returns-policy-OLD.pdf", which she'd forgotten was in the project. She deletes the old file, asks again, and gets the 30-day answer quoting the current policy. Then she adds "one current version of each policy in the project" to her [never-again list](/library/never-again-list).

Common mistakes

Switching tools before checking the material. The new tool will read the same old file.
Pasting everything "just in case". More material means more places for the wrong answer to come from.
Asking "why did you say that?" without asking for the quote. You'll get a plausible story, not the source.
Fixing the answer and not the folder, so it happens again next week.

No sign-up, and this page sends nothing you do anywhere.

Want this live, with your questions? AI in an hour is $25. Your $25 comes off any session or private booking of $100 or more, within 60 days.

Before you blame the AI

The five questions

Common mistakes

Related