How do we know our AI is actually getting better?

Short answer: measure it. Keep a fixed golden set of real questions with their expected answers and sources, score every change against it, block any answer that cites no source, and watch how often the system says it can't answer in production, as an early sign that something drifted.

Someone tweaks the system, tries a few questions, and declares "it feels smarter now." That isn't measurement — it's a feeling, and in regulated work a feeling doesn't survive an audit. Here's what measurement looks like instead.

1. Build a golden set

A fixed list of real questions, each with its expected answer and the document it should cite. Fifty is enough to start. This becomes your ruler: the thing every change is measured against.

2. Score every version against it

Whenever anything changes — the model, the settings, the documents — re-run the whole golden set and compare. Did accuracy go up or down? Did it start citing the wrong source? You see it immediately, before a customer does.

3. Make hallucinations structurally impossible to ship

The simplest and most powerful rule: every answer must cite a source. No citation means the model is answering from its own memory, and that answer gets blocked. You don't review for hallucinations by hand; you stop them at the door.

4. Watch production, not just the lab

Track how often the system says "I can't answer this." A rising rate is an early warning that something has drifted — and it doubles as a queue of questions your experts should look at.

Evidence, not vibes

"It feels smarter" is a feeling. A golden set, a citation rule and a drift alert are evidence. If your AI system doesn't have a ruler yet, that's the first thing worth building — before the next change, not after it.