When AI Sounds Right But Isn’t Enough

by Ruchika Chourey

‍

What I Learned Building Trust, Reducing Hallucination, and Scaling QA in a RAG System

‍

Introduction — When AI Sounds Right but Isn’t Enough

‍

AI systems don’t always fail loudly. Sometimes, they sound fluent, confident — even helpful — but something still feels off. In Retrieval-Augmented Generation (RAG) systems, that gap between “sounding right” and “being trustworthy” is where hallucinations hide.

In this post, I share what we learned while building AskNeedl — a RAG-based intelligence layer for enterprise search — and how we identified subtle failure patterns, built user trust through citations, and introduced structured QA and feedback loops.

From hands-on collaboration with ML teams to aligning with real user expectations, this is a product-led view of making AI feel not just intelligent, but credible.

‍

Why Hallucination Matters in RAG

‍

Hallucination — the generation of incorrect or fabricated information — is often framed as a model issue. But in Retrieval-Augmented Generation (RAG) systems, it’s more nuanced. It’s not just what the system says that matters — it’s how confidently, how completely, and how credibly it says it.

For end-users, especially in enterprise settings, even minor inaccuracies can undermine the system’s usefulness. They don’t want something “probably correct” — they want something verifiably true.

‍

In search-driven workflows like compliance monitoring, competitive intelligence, or financial disclosure tracking, AI responses without grounding are seen as speculation. Citations aren’t a bonus — they’re the baseline for trust.

‍

And this is where the difference between internal AI confidence and external user perception becomes stark.

‍

🎯 The User’s Mental Model: “Known-Knowns”

‍

Most enterprise users aren’t asking open-ended questions. They’re not saying “Tell me something new.”

They’re saying:

‍

“I know this clause exists. I just don’t remember where.”‍
‍“I read this yesterday, show me the source.”‍
‍“Has this event been disclosed recently?”

‍

In other words, they’re navigating known-knowns — factual information they believe exists in the system. So if the system’s response lacks a citation or cites something that doesn’t exactly match, the user immediately assumes one of two things:

• The system missed something important
• Or worse, it made something up

This mismatch between what the user expects (fact-grounded answers) and what the AI generates (inference-driven summaries) is the root of the hallucination challenge in RAG.

‍

🧠 Why RAG Hallucination Isn’t Just a Model Problem

‍

RAG systems rely on a two-stage process:

1. Retrieve relevant documents or passages
2. Generate a natural language answer using those snippets

‍

If step 1 misses a document, even slightly, step 2 is compromised.
If the retrieved documents aren’t displayed or cited, the user can’t verify the result.

‍

So even when the generation isn’t technically hallucinated, the absence of visible grounding creates the perception of hallucination.

This perception gap can be just as damaging as actual model inaccuracies, especially when the end user is from legal, regulatory, or finance teams.

‍

The Product Imperative: Trust is a Feature

‍

As product managers, we often talk about accuracy, latency, and coverage. But for enterprise AI, trust is the true UX metric.

That’s why citation design — where the source appears, how confidently it’s linked, how much of the document is exposed — isn’t just UI polish. It directly impacts user adoption.

In our case, we saw that:

• Users were more forgiving of incomplete answers with citations than of fluent answers without them
• When citations were missing due to UI rendering delays, users raised concerns about hallucination, even when the answer was valid
• Explicitly anchoring answers to document excerpts built credibility

‍

The Patterns We Saw — Where RAG Fails Quietly

‍

Not all hallucinations are loud. In enterprise RAG systems, the most dangerous failures are the ones that sound perfectly correct but aren’t fully grounded.

As a product team, we didn’t wait for users to report issues — we proactively analyzed query logs, response sessions, and internal QA checks to understand when and why answers broke down. We noticed a consistent set of patterns across query types — not random errors, but recurring failure themes that pointed to deeper architectural or retrieval limitations.

Here are some of the most common ones we encountered:

‍

🧩 1. Temporal Drift in Responses

Example:
Query: “Has the promoter increased stake in the last 6 months?”
Response: “Yes, the promoter increased stake as per SAST filing dated March 2023.”
Issue: The answer referenced a filing outside the 6-month window.

We traced these errors back to retrieval ranking — older filings were being prioritized over recent ones, especially when date filters weren’t enforced.” But to the user, the result felt wrong. And without a citation showing the date clearly, it felt made up.

‍

🧠 2. Missing Multi-Document Reasoning

Example:
Query: “What are the top three reasons for the revenue drop in Q1?”
Response: System pulls from management commentary, earnings call transcript, and a presentation slide.
Issue: Only one data source retrieved, with a generic summary.

The answer might be fluent, but it’s incomplete — not due to poor generation, but because RAG retrieved too little context. This pattern emerged mostly in synthesis-style questions where multiple documents needed to be stitched together.

‍

📌 3. Weak Anchoring in List-Type Responses

Example:
Query: “List key risk factors from the latest annual report.”
Response: “The key risks include geopolitical uncertainty, regulatory changes, and supply chain disruptions.”
Issue: No citation, no paragraph reference, and unclear sourcing.

List-type answers tend to sound complete. During internal reviews, we saw that users quickly flagged these responses, especially when the system skipped over recognizable sections like “Risk Factors,” — as unverifiable, even when they were accurate.

‍

❓ 4. Ambiguous Queries Leading to Overgeneralization

Example:
Query: “Is there any update on the merger?”
Response: The system returned a response about a different merger involving a similar-sounding company or division.
Issue: Retrieved tangential documents due to vague entity match.

When queries were short, vague, or context-dependent, the system sometimes retrieved tangential documents. The generator then created a plausible-sounding answer, often with incorrect entity alignment.

This category taught us that query intent disambiguation is critical.

‍

🔁 5. Citation Not Rendering / Missing in Output

Example:
The backend retrieved the correct document and passed it to the model, but the frontend failed to show the citation tag.

‍

One user looked at a response and asked, “Is this from our documents, or is it pulling from the internet?” That moment made it clear: in the absence of a visible citation, even grounded answers felt fabricated.

‍

This was a pure UX gap, but from the user’s perspective, it looked like a hallucination.
“If I can’t see the source, I can’t trust the answer.”

‍

What These Patterns Taught Us

• Hallucination isn’t just about wrong answers — it’s also about partially right answers with missing context
• The more vague or synthesis-based the query, the higher the likelihood of breakdown
• We learned that users were far more forgiving of incomplete answers — as long as they could trace the reasoning or verify the source.
• Citation presence is sometimes more important than perfect phrasing

‍

“These patterns taught us that hallucinations aren’t always about factual errors — sometimes, it’s about what the user believes the system should know and show”

‍
You can also check this post on Medium.