PIPELINEAI Knowledge
Why AI invents things — and how to stop that in your business
Invented answers aren't a flaw in the model, but a consequence of how it works. That's why only a setup that makes them impossible will help.
A language model generates probable continuations of text. It has no concept of “I don't know” — it only has probabilities. A plausible-sounding false answer is indistinguishable from a correct one to the model.
01
Why it happens
A language model generates probable continuations of text. It has no concept of “I don't know” — it only has probabilities. A plausible-sounding false answer is indistinguishable from a correct one to the model.
02
What doesn't help
Asking the model not to invent things. That reduces the frequency a little and doesn't eliminate the problem.
03
What helps
Three things together: only allow answers from a verified source, cite the source in the answer, and provide a defined fallback for when nothing is found. The third point is the most important and the one most often left out.
04
The test before going live
Ask questions whose answer isn't in the documents. A well-built bot says it doesn't know and redirects. A poorly built one invents something — and you notice that with the customer.
05
An example from practice
A chatbot without source binding answers the question about a specific warranty period with a plausible-sounding but false number — nobody notices until a customer relies on it. With source binding, the bot would have either quoted the correct period from the warranty terms or honestly said it doesn't know.
Questions
Frequently asked questions
Aren't newer models simply better at not inventing things?
They're better, but the problem fundamentally remains — it's not a question of model generation, but of how they work. A system without source binding will always occasionally invent something, no matter how new it is.
How often does this happen in practice?
Without countermeasures, noticeably often with tricky or rare questions. With clean source binding and a defined fallback, the rate drops to a level where what remains is almost always caught as I don't know instead of as a false claim.
Can you retrofit this into an existing system?
Yes, usually without rebuilding everything. Source binding and the defined fallback can be added to an existing configuration — the effort lies in testing, not in rebuilding.
How do you test this systematically instead of just trying a few questions?
With a fixed list of real queries, including some whose answer deliberately isn't in the documents. This list is run through again after every change — that's the difference between a random sample and a real test.
Next step
Let's talk briefly about whether this makes sense for you.
A twenty-minute conversation is usually enough to clarify whether a project makes sense. If not, we'll say so — that's cheaper for both sides than an offer nobody needs.