What should we ask before approving an AI project?

Short answer: six questions — where each answer comes from, what happens when the system doesn't know, how you'll prove it improved, which guardrails specifically, whether a decision can be reconstructed months later, and why the build/buy split is what it is. A vendor who gets vaguer as you go down the list has told you what you need to know.

Most AI projects that fail could have been stopped by a few questions asked at the start. These are the ones that separate a real system from an impressive demo. Use them with your own team, or with any vendor you're about to sign.

1. Where does each answer come from?

Ask to see the exact source and citation behind an answer. If nobody can show you, the system is guessing — and in any setting where answers have consequences, guessing is disqualifying.

2. What happens when it doesn't know?

The right answer is "it refuses, or hands over to a person." The wrong answer is "it always gives you something." A system that always answers is a system that sometimes answers wrongly, with the same confidence.

3. How will you prove it's working — and that it improved?

Look for a reference set of known-good answers that every change is tested against. "It feels better" is not measurement. (More on this in how to know your AI is getting better.)

4. Which guardrails, specifically?

"We have guardrails" is not an answer. Ask them to name, separately, what protects against unsafe content, prompt manipulation, leaks of personal data, and answers that aren't grounded in a source.

5. Can you reconstruct a decision six months later?

If someone asks why the system answered a certain way last quarter, can it be replayed? Without an audit trail, a regulated business can't put the system in front of customers — and nobody can debug it either.

6. Build, buy, or both — and why this split?

A thoughtful answer has reasons on both sides. "We build everything" or "we just use the product" usually means nobody thought about it. (See buy or build.)

What the answers tell you

You don't need to be technical to use this list. Watch how the answers change as you go down it. If they get more specific, you're talking to people who have built this before. If they get vaguer, you've learned the most important thing before spending anything.