What to Check Before Trusting a RAG Answer: Retrieval Evaluation, Faithfulness, and Sources

Even if the answer seems correct, you need to look at the search results separately.

Once you've developed a strategy to improve search quality, you'll naturally want to move on to answer generation. In the previous flow, “what unit to search for and what unit to read” was separated into hybrid search, parent document retrieval, and multi-vector retrieval. If you first came to this article through a search, you can start reading here after the retriever has already returned the candidate document. This is because it is difficult to judge the quality of RAG just by looking at the final answer.

This time, we will use the virtual learning documentsample-office-guide.mdfrom Part 1. This is not an actual internal document, but an example created to explain the search flow. Let’s say the user asks:

Copy1# This question checks both the equipment rental location and conference room reservation conditions.2If I want to use a monitor for conference room presentations, where can I rent it and when should I make a reservation?

A good answer doesn't end with a plausible sentence. You must also leave information about whether the answer comes from the actual search results, which section each of the two conditions come from, and whether you can check by reopening the source. In RAG, the “correct answer” is not a sentence well written by the model, but is closer to the state in which the search candidate and the answer sentence are connected to each other.

So, the flow of Part 5 is divided into before and after creating the answer. Before answering, retriever candidates are evaluated, and after answering, it is checked whether the sentence went out of context. At the end, citation and source attribution are left for the user to check.

Search evaluation involves reading candidate documents before responding.

Retrieval evaluation is not a step to check whether the answer has been beautifully generated. Before calling the model, this is the step to check whether the candidate document that solves the question has been properly uploaded. Skipping this evaluation makes it difficult to isolate the cause when an answer is incorrect. Whether the search was wrong, the prompt wasn't enough, or the model misread the context, it all gets mixed up at once.

For example, let's say retriever returns the candidates below.

This result is not bad. Since the question asks about two work conditions, it is a good sign thatequipmentandreservationare posted together. Conversely,securitycan be excluded before inserting it into the prompt. The key to evaluation is not “all of the top k items must be relevant,” but whether sufficient evidence for the answer has been provided and whether unnecessary candidates can be identified.

A small evaluation record can be left as shown below.

Copy1# Search candidates are read in the pre-answer stage, and documents to be included in the prompt and documents to be excluded are divided.2retrieved_docs = retriever.invoke(question)3 4selected_docs = [5    doc for doc in retrieved_docs6    if doc.metadata["section"] in {"equipment", "reservation"}7]8 9rejected_docs = [10    {11        "source": doc.metadata["source"],12        "section": doc.metadata["section"],13        "reason": "The question is not directly related to the equipment rental or booking terms being asked.",14    }15    for doc in retrieved_docs16    if doc.metadata["section"] not in {"equipment", "reservation"}17]

This code is not an example intended to replace an actual evaluation framework. The important thing is to create a space to read the candidate before answering. Only when there is that position can you decide whether to increasek, apply a metadata filter first, use multi-query, or include reranking.

self-query separates metadata conditions before searching

Questions often contain a mixture of text meaning and metadata conditions. For example, a user asks:

Copy1# The question contains both section conditions and actual meaning search conditions.2Among the conference room information, only monitor rental and reservation deadlines are provided.

Here, ‘Meeting room information’ is a condition to narrow the scope of the document, and ‘Monitor rental and reservation deadline’ is what you actually want to find. The self-query approach is closer to the idea of ​​separating these conditions before searching. Instead of throwing the entire question into an embedding search, first narrow the scope with a metadata filter and make the text search focus on the remaining semantic conditions.

For the examples in this article, thinking simply is enough.

One thing to be careful of when applying self-query is that the filter may not become too strong. Although the user said 'room information', the actual answer may also require theequipmentsection. So, rather than using the filter results as if they were closed answers, you should compare them again with the original question and the separate conditions.

Answer fidelity contrasts the sentence with the context

Even if the search candidate is good, the answer does not always faithfully follow the evidence. Answer faithfulness is the perspective of seeing whether the answer is spoken only within the context provided.

Let's look at the example answer.

Copy1# This answer must check whether the last sentence adds out-of-context information.2Presentation monitors can be used after recording rental at the information desk.3Meeting room reservations can be made up to 1 hour before the start of use.4However, all equipment rentals require team leader approval.

The first two sentences can be seen as coming from candidatesequipmentandreservation. The problem is the last sentence. If the search candidate forsample-office-guide.mddid not have the team leader approval condition, this sentence should be omitted. Even if the sentence seems plausible in business terms, it is additional information without basis in the RAG response.

When checking your answers, it is better to break them down into sentences.

After going through this process, it becomes easier to decide whether to fix the prompt, post-process the answer, or force citation output. Especially in business document RAGs, the ability to say “I don’t know” or “I can’t find it” is often more important than the ability to write a lengthy response.

A citation is a metadata contract, not a footnote.

It is difficult to think of citations as decorations that can be added later. Source indication is possible only when metadata is left in the document loading stage, metadata is preserved in the chunking stage, and metadata is returned together in the retrieval stage.

The minimum metadata required for an answer varies from project to project, but for learning examples, the following can be considered.

Copy1# To create a citation, tracking fields such as source, section, and chunk ID must remain in the search results.2Document(3    page_content="Meeting room reservations can be made up to 1 hour before the start of use.",4    metadata={5        "source": "sample-office-guide.md",6        "section": "reservation",7        "chunk_id": "sample-office-guide.md#reservation:001",8        "updated_at": "2026-06-01",9    },10)

If you have this metadata, you can create an answer like below.

Copy1# Add evidence next to the answer sentence so that the user can check it again.2Presentation monitors can be used after recording rental at the information desk. [sample-office-guide.md, equipment]3Meeting room reservations can be made up to 1 hour before the start of use. [sample-office-guide.md, reservation]

The citation format may vary depending on the service. It may appear as [file, section], or if it is a web service, it may appear as a link or document title. Whichever format you choose, there must be a link back between the response text and the supporting documents.

Attribution is also part of the user experience

source attribution is not only logs for internal evaluation. It is an interface that allows users to trust their answers. In particular, if the RAG response addresses responsible content such as policies, contracts, or operating procedures, a way to show “where the word is coming from” is needed.

However, if you pour all the evidence onto the screen, it becomes difficult to read. Divide according to the level the user needs to check.

When sources are indicated, the way answer errors are handled also changes. Instead of “the answer is strange,” users can provide feedback by saying “this section is irrelevant” or “it’s been a while since it was updated.” From a developer's perspective, retriever, metadata, and document update issues can be viewed separately.

Application example: Tuning direction is visible only when there is an evaluation record

The most frustrating situation when editing a RAG answer is when the cause appears to be lumped together. If all that remains is the fact that the answer is wrong, it is difficult to know whether the retriever should be modified, the chunk should be re-divided, or the prompt should be changed. Therefore, the evaluation records of the five pieces are not just logs, but serve as a standard for choosing the next direction for revision.

The cautionary tale is simple. If you only change the prompt because the answer is incorrect, the actual problem of missing metadata filters or search candidates remains. Conversely, if you only change the retriever, you may miss the answer question that adds an out-of-context sentence.

Even if the question is the same, the place to address changes depending on the location of the failure.

As this record accumulates, improvement tasks become smaller. Rather than redesigning the entire pipeline every time, we can narrow down whether this failure is a condition separation issue before search, a candidate selection issue, or a verification issue after answer generation. This may seem excessive for a small RAG example, but as the number of documents increases, quality improvement is likely to go by the wayside without such records.

What to leave before moving on to the next step

Coming to Part 5, the RAG flow is not simply “code for inserting a document and receiving a response.” Documents become chunks, chunks become search candidates, search candidates become prompt contexts, and answers are linked back to evidence.

In the next article, we look at the boundaries that arise when transferring this flow to actual documents and production environments. In PDFs, the text may not be as clean as you think, and in web documents, you need to pay attention to freshness and duplication. Markdown is strong when the heading structure is stable, but if the document writing rules fluctuate, the splitting quality also fluctuates. Additionally, token budget and prompt injection risk directly affect search quality and operational safety.

The standards to be taken in Part 5 are as follows.

• Before responding, the retriever candidate is evaluated.
• After answering, check whether the sentence went out of context.
• Citations are not a decoration at the end of an answer, but are a result of metadata preservation.
• source attribution is a device that allows users and developers to see the same evidence.

Frequently Asked Questions

Do all RAGs need an evaluation framework?

There is no need for a grand evaluation system from the beginning. However, at least the question, search candidates, selected context, excluded candidates, and final answer must be left together. Without this record, you can only judge whether the quality of the answers has improved or deteriorated.

Will the hallucination problem be solved just by adding a citation?

no. A citation is just a sign that connects the answer and the evidence. There must be a process to check whether the answer sentence follows the actual context. Even if the source is attached, if the sentence itself deviates from the evidence, it is an incorrect answer.

When is self-query worth reviewing first?

If metadata such assource,section,date,product, androleare stable in the document, and users often mention the conditions in questions, it is worth considering. Conversely, if metadata quality is low, self-query may incorrectly narrow the search scope.