How to fix a RAG system that gives wrong answers

Why the usual way fails

Wrong answers in a retrieval system look like model mistakes, so teams change the model first. Quinn Favo's article on enterprise RAG puts it plainly: most failures come from retrieval, permissions, stale content or a pipeline nobody can inspect. If the right document never reaches the model, no model can answer well. And when there is no way to trace which passages were fetched for a given answer, every fix is a guess. Permissions add a second trap: a document a user should not see can still end up in the context, or a document they need can be filtered out without anyone noticing. Old copies of a policy often sit next to the new one, and nothing tells the retriever which is current.

How AI & Automation Consulting helps

The LLM Private Cloud and audit packages treat RAG as infrastructure, with retrieval, permissions and observability built in from the start. An AI Automation Audit looks at your existing flows, prompts, tool integrations and data boundaries, which is where stale content and permission leaks usually hide. In the financial-services case, the work included a scoped RAG layer over internal documents on client-owned infrastructure. You end up with a pipeline your engineers can inspect and keep fixing themselves, with documentation that explains how retrieval, permissions and indexing fit together.

Step by step

1. Collect a handful of real questions the assistant got wrong, with the answer you expected and the document that holds it.
2. Start with an AI Automation Audit of the flows, prompts, integrations and data boundaries behind the assistant.
3. Check retrieval first: did the right document come back, and was the user allowed to see it?
4. Remove or re-index stale content so old versions stop outranking current ones.
5. Add observability so every answer can be traced back to the documents it used, and keep those traces long enough to compare.
6. Hand the runbooks to your team so the next bad answer does not need outside help.

Tips

• Keep a small test set of questions with known answers and rerun it after every change. It tells you quickly whether a fix helped or quietly broke something else.
• Treat permissions as part of retrieval. If a document should not reach a user, it should not reach the model on that user's behalf either. Check this early, because a permissions mistake is worse than a wrong answer.
• Make the pipeline readable. Fewer black boxes and clear operating boundaries make it much easier to see where an answer went wrong.
• Do not swap models until retrieval is checked. A stronger model on top of bad retrieval mostly gives you more confident wrong answers, and a bigger bill for the same mistakes. Fix what reaches the model first, then judge the model on clean input.

Questions