Where chunks meet questions
In the previous article,sample-office-guide.mdbecameDocument, andDocumentwas divided into a searchable list of chunks. Now when a question comes in, these chunks should be selected again.
Now see how the chunk moves up to just before the prompt. Rather than memorizing the embedding model, vector store, retriever, prompt template, chain, and query transformation separately, it follows which intermediate output a single question passes through.
The question is put like this:
Until when can I reserve a conference room?Before answering this question, the first thing to check is not the answer sentence. What chunk the retriever retrieved, whether the chunk's metadata is alive, and what context is included in the prompt.
Embedding is a representation transformation for search.
A chunk is human-readable text. However, to compare similarity in a vector store, chunks and questions must be converted into comparable expressions. At this time, embedding comes into play.
In the actual service, you must select an embedding provider or vector database. However, let's leave the provider setting aside for a moment and focus on the flow of "the text is changed to a comparable value, and the chunk is found again with that value."
The example below is a simple embedding to view the structure without an API key. Although it is not an actual semantic embedding, it is enough to visually check the flow of questions being asked and related chunks being returned.
from langchain_core.documents import Documentfrom langchain_core.embeddings import Embeddingsfrom langchain_community.vectorstores import FAISS class KeywordEmbeddings(Embeddings): terms = ["cabinet", "reservation", "equipment", "security"] def _embed(self, text: str) -> list[float]: return [float(text.count(term)) for term in self.terms] def embed_documents(self, texts: list[str]) -> list[list[float]]: return [self._embed(text) for text in texts] def embed_query(self, text: str) -> list[float]: return self._embed(text)The first thing to check before elaborating the implementation is whether the chunk and query are expressed in the same way. If you change only the question in a different way or save the document with different standards, it is difficult to trust the retriever results.
The vector store contains both text and metadata.
The vector store appears to store only vectors, but in RAG flow, chunk text and metadata must live together. If only a vector score is returned as a search result, there is no context to put in the prompt, and the source cannot be explained.
chunks = [ Document( page_content="Meeting room reservations can be made up to 1 hour before the start of use.", metadata={ "source": "sample-office-guide.md", "section": "reservation", "version": "2026-06", }, ), Document( page_content="Equipment rentals are recorded at the information desk and then returned.", metadata={ "source": "sample-office-guide.md", "section": "equipment", "version": "2026-06", }, ), Document( page_content="Secure access cards can only be applied for by approved entities.", metadata={ "source": "sample-office-guide.md", "section": "security", "version": "2026-06", }, ),] vector_store = FAISS.from_documents(chunks, KeywordEmbeddings())retriever = vector_store.as_retriever(search_kwargs={"k": 2})At this stage,k=2is also not the correct answer. How many chunks are needed for one question must be judged by the output. A single reservation chunk may be sufficient for a conference room reservation question, or even a sentence confirming the equipment after use may be required.
Running example: retriever result is not an answer
When you run retriever, a chunk close to the question is returned.
question = "Until when can I reserve a conference room?"results = retriever.invoke(question) for doc in results: print(doc.page_content) print(doc.metadata)The retriever result is not an answer. The retriever selects a context candidate, not an answer candidate. If the room reservation chunk comes back first, that's a good sign, but if the second result includes a security access chunk, you should think twice before putting it in the prompt.
| confirmation point | What you see |
|---|---|
| top results | Check whether the chunk closest to the question came first |
| metadata | Check if source, section, and version are returned together. |
| k value | Check whether there is too little or too much context |
| repeat | Check to see if the same content is posted repeatedly. |
| mixed | Check whether chunks from other topics have come in together. |
If the search is shaky, the answer will be shaky even if you use prompt well. So in this step, the resulting chunk is read first before calling the model.
The prompt template defines input boundaries.
Once you have checked the retriever results, you can now move to the prompt. At this time, the prompt template is not a simple string format.question,context, andsourceare input boundaries that determine where each goes.
from langchain_core.prompts import ChatPromptTemplate prompt = ChatPromptTemplate.from_messages( [ ( "system", "You answer only based on in-house knowledge documents. Do not make up content outside of context.", ), ( "human", "Question: {question}\n\ncontext:\n{context}\n\nsource: {source}", ), ]) selected_docs = results[:1] payload = { "question": question, "context": "\n".join(doc.page_content for doc in selected_docs), "source": ", ".join( f"{doc.metadata['source']}#{doc.metadata['section']}@{doc.metadata['version']}" for doc in selected_docs ),} messages = prompt.format_messages(**payload) for message in messages: print(f"{message.type}: {message.content}")This output is the input just before the model call. Check whether the question remains as is, whether only the chunk selected by the retriever is included in the context, and whether the source is missing. If you do not check this, it is difficult to separate whether it is a retrieval problem or a prompt problem when the answer is strange.
chain is not an autorun button
Because of the word chain, it may seem that all processes follow automatically. However, in RAG, the chain is meaningful when the handoff of each step is visible.
If you solve the flow like a log, it reads like this.
question-> transformed query or original query-> retriever result-> prompt payload-> rendered prompt messages-> model answer-> answer with sourceIt must be possible to output intermediate values at each step. In particular, you must see the retriever result and rendered prompt. The retriever result is wrong, but just fixing the prompt will hide the problem. The wrong context is entered in the prompt, but if you change the model, the same problem repeats.
Boundary case: query transformations must be compared
Sometimes questions are changed to improve search quality. For example, changing a user's natural language question into an expression better suited to searches.
original_query = "Until when can I reserve a conference room?"transformed_query = "Example in-house knowledge document Meeting room reservation deadline" print("original:", original_query)print("transformed:", transformed_query)Query transformation is useful, but it also has risks. This is because the intent of the original question may change as you increase search hints. If the question of ‘how long can this be possible’ becomes as broad as simply ‘meeting room policy’, the retriever can retrieve more documents, but the focus of the answer becomes blurred.
Therefore, the converted question should always be viewed side by side with the original question. Even if the question used in the search changes, the user's original question must remain in the prompt. Expressions changed for search purposes should not change the intent of the answer.
Checklist to leave before calling LLM
Before calling LLM, the input boundaries must first be cleaned up. The chunk goes through embedding and enters the vector store. The retriever retrieves the chunk closest to the question. The prompt template puts the chunk into context and passes the source information along. chain connects these handoffs, but it is difficult to debug without seeing the intermediate output.
There are five things to check in this flow:
- Are questions and chunks compared based on the same embedding?
- Are chunk text and metadata left behind in vector store results?
- Are the retriever results sufficiently narrowed down to answer material?
- Does the prompt contain only the intended context?
- Does the query transformation maintain the intent of the original question?
Only when these five things are correct are you ready to see the model answer. In RAG, the answer is the last sentence, and the quality is determined by the input boundaries that came before it.
Practical example: Don't mistake retriever results for answers
The first chunk returned by the retriever is not always the correct answer. For example, if a user asks “How long will data be kept after deleting an account?”, the retriever can retrieve the “How to delete an account” document first. The document title is similar, but what is needed for an actual answer is the retention period and legal basis. So we need a small criterion to evaluate the retrieved chunk before calling LLM.
| Confirmation criteria | Things to see |
|---|---|
| Question Match | Does the chunk contain the key noun of the question? |
| Answer possibilities | Can you answer by just looking at the chunk? |
| source | Are the document id and section remaining? |
| crash | Doesn't another chunk say the opposite? |
A case of failure is when the prompt template causes search results to be unconditionally trusted. The rule of “answer only by looking at the context below” is good, but if the context is wrong or lacking, you should be able to answer “lack of evidence.” A chain is not an autorun button, but an assembly step that connects search results, prompt boundaries, and output rules. Here, you need to distinguish between empty results and low relevance to fix the RAG quality.
Comparative experiments can also start small. For the same three questions, the original question search, converted question search, and metadata filter search are output side by side. The location of the problem becomes clear when you look at which chunks were selected in the stage before creating the answer.
| experiment | What to check |
|---|---|
| Original question search | Do the necessary chunks come from just the user’s expression? |
| Search for conversion questions | Does the candidate get better without the intention changing? |
| metadata filter search | Are authority and document scope reduced first? |
If you leave this table, you don't have to lump it in as a "prompt problem" later when the model's answer is wrong. It is possible to determine whether the search candidate is incorrect, whether the source is missing from the prompt, or whether the model created content outside of the context. RAG debugging is faster with intermediate output comparisons than with the last answer.
It is better to create a small test set. Choosing just 5 questions is enough. One is a question for which the correct answer is in the document, one is a question that is confusing because there are many similar documents, and one is a question for which there is no answer in the document. The retriever must handle the three cases differently. In questions with answers, relevant chunks must appear, in confusing questions, the source must be separated, and in questions without answers, the lack of evidence must be able to be stated.
The prompt chain reflects this distinction. If the context is empty, no answer is created, if there are multiple sources, a possibility of conflict is indicated, and if there is no section metadata, the citation quality is recorded as low. This way, the chain becomes a boundary that protects RAG quality rather than a thin wrapper around model calls.
Operation logs are also left in the same structure. Savingquery,transformed_query,retrieved_chunk_ids,prompt_token_count, andsource_countmakes quality retrospective easier in the future. When the answer is wrong, you can immediately see how different the original question was from the converted question, how many search results there were, and whether the prompt was too long.
The LangChain example is good for quickly connecting chains, but in services, observability is more important than connectivity. A chain that cannot output intermediate values is convenient for demos but inconvenient for operation. So, it is better to get into the habit of checking the payload of each step from the beginning.
Frequently Asked Questions
Q. Can I apply to LLM immediately after the retriever results come out?
A. It is possible to run it, but it is better to read the results first. If the search result is wrong and you put it directly into LLM, the model can create a plausible answer based on the wrong context.
Q. Should I always use query transformation?
A: No. If the question is short or fits well with the document expression, the original question may be sufficient. If you use a conversion, you should print the original question and the converted question together to check whether the intent is maintained.
Q. Is it necessary to include the source in the prompt?
A. If source indication or debugging is necessary, it is better to include it. At least the source must be maintained in the payload inside the chain so that the answer and evidence can be linked later.
댓글
GitHub 계정으로 로그인하면 댓글을 남길 수 있습니다. 댓글은 GitHub Discussions를 통해 운영됩니다.