LangChain RAG Review 03: Retriever Strategies for Better Search Quality

The search is successful, but it's difficult to answer

In Part 2, we saw the flow of putting the chunk created insample-office-guide.mdinto the vector store and having the retriever retrieve the chunk close to the question. There was a sentence that I deliberately emphasized at that time. The retriever result is not an answer, but an answer candidate.

Part 3 starts from the next scene. The question is still simple.

Until when can I reserve a conference room?

This question should include the following chunk: ‘Conference room reservations can be made up to 1 hour before the start of use.’ However, if the actual document becomes slightly larger, the search results fluctuate like this. The conference room reservation sentence comes first, but the equipment return sentence comes second, and the security access card sentence comes third. Although it is not completely unrelated to the question, it is a result that is too broad to use as a basis for an answer.

A common mistake here is to change the value ofkor write the prompt statement more strictly. Of course, both can help. However, when the search results themselves come in broadly, the prompt receives already mixed ingredients. The model generates an answer within it, but it becomes increasingly unclear which chunk was the actual basis.

When search results are ambiguous, the retriever strategy should not be attached as a list of functions. First, narrow the search scope with a metadata filter, expand the question expression with multi-query retrieval, remove unnecessary sentences in the imported chunk with context compression, and reorder the final candidates with reranking.

The criteria for judgment are simple. Before putting it in the prompt, you should be able to explain which candidates should be kept and which candidates should be removed.

Running example: Read search results like a failure log

Before committing to a strategy, you should read the current results. If you look at the search results like a success log, it stops at “Results found.” Looking at the failure log raises the question, "Why did this chunk come in?"

Let's increase the example chunk a little. The samesample-office-guide.mdincludes conference rooms, equipment, security, and visitor information.

Copy1from langchain_core.documents import Document2 3# Create a minimal example chunk to use for search quality comparison.4chunks = [5    Document(6        page_content="Meeting room reservations can be made up to 1 hour before the start of use.",7        metadata={"source": "sample-office-guide.md", "section": "reservation"},8    ),9    Document(10        page_content="Please check the condition of the shared equipment after using the conference room.",11        metadata={"source": "sample-office-guide.md", "section": "reservation"},12    ),13    Document(14        page_content="Equipment rentals are recorded at the information desk and then returned.",15        metadata={"source": "sample-office-guide.md", "section": "equipment"},16    ),17    Document(18        page_content="Visitor access is subject to security desk approval and a temporary card is issued.",19        metadata={"source": "sample-office-guide.md", "section": "security"},20    ),21]

Assume that the retriever returns a result as below.

1.reservation- Meeting room reservations are available up to 1 hour before the start of use. 2.equipment- Equipment rental is recorded at the information desk and then returned. 3.security- Temporary cards are issued for visitor access after security desk approval.

The first results are good. The problems are the second and third. Equipment rental and visitor access are similar in the larger topic of “house guidance,” but they are not necessary when it comes to asking when meeting room reservations are due. If you put all three chunks in the prompt in this state, the model may give you the right answer. However, this is not because the search quality was good, but because the first chunk was a lucky mix.

Before creating an answer, write down the following three things first.

• What number of chunks are needed directly for the correct answer?
• Does the auxiliary context of the same section come together?
• Why is a chunk from another section interjected?

This record allows you to compare whether things got better or worse when you changed your strategy behind the scenes.

First reduce the search space with metadata filter.

The first thing to look at is metadata. The reason why we decided to keep metadata in Part 1 is revealed here. If you havesectionleft per chunk, you can narrow your search to thereservationarea when your question is explicitly about conference room reservations.

Copy1# When the question category is clear, the candidate space is reduced using metadata before semantic search.2retriever = vector_store.as_retriever(3    search_kwargs={4        "k": 3,5        "filter": {"section": "reservation"},6    }7)

The syntax of this code may vary depending on the vector store used. The first thing to look at before specific grammar is the order. Before making semantic search smarter, see if you can exclude areas that shouldn't be searched in the first place.

Metadata filters are especially effective when document categories are clear. If it is an in-house document, values ​​such assection,department,doc_type, andversionare candidates. For product documentation, values ​​likeproduct,feature,release, andlocaleare helpful.

However, you should not trust the filter too quickly. If a user asks, “Where can I rent a monitor for use in a conference room?”,reservationandequipmentmay be needed at the same time. If the question covers several areas and you fix it as one section, you are throwing away the necessary evidence.

After using the metadata filter, compare the results like this.

before

1.reservation- Meeting room reservations can be made up to 1 hour before the start of use. 2.equipment- Equipment rental is recorded at the information desk and then returned. 3.security- Temporary cards are issued for visitor access after security desk approval.

after:section = reservation

1.reservation- Conference room reservations can be made up to 1 hour before the start of use. 2.reservation- Please check the status of the shared equipment after using the conference room.

What's better is that other sections are missing. There are still questions remaining. Is the second chunk really necessary for the answer? The first chunk is sufficient for the question “How long?” At this time, the following strategy becomes necessary.

Multi-query retrieval broadens question expressions

If metadata filter is a strategy to reduce the search space, multi-query retrieval is a strategy to broaden the question expression. This is useful when the user's question and the wording of the document are different.

For example, a user might ask:

What is the deadline for meeting room applications?

In the document, there is no expression 'application deadline', but there may be expressions such as 'reservation' and 'one hour before start of use'. With a single query, the desired chunk may be pushed back due to differences in expression. At this time, you can create several search queries from the original question, search each one, and then combine the results.

Copy1# Reduce expression differences by solving the same intent into multiple search expressions.2queries = [3    "Meeting room application deadline",4    "When can I reserve a meeting room?",5    "Reservation criteria before meeting room use begins",6]7 8candidate_docs = []9 10for query in queries:11    candidate_docs.extend(retriever.invoke(query))12 13unique_docs = deduplicate_by_source_and_text(candidate_docs)

The caveat here is that multi-query broadens the search intent. If the direction of expansion is correct, the difference in expression is reduced. On the other hand, if you broaden it too much, the question becomes blurry. If the ‘conference room application deadline’ is changed to ‘in-house facility use policy,’ equipment rental, visitor access, and seat reservations can also be included.

So, when using multi-query, the generated query must be left along with the results.

query: Meeting room application deadline

hit:reservation- Meeting room reservations are available up to 1 hour before use.

query: When can I reserve a conference room?

hit:reservation- Meeting room reservations can be made up to 1 hour before the start of use.

query: Reservation criteria before starting to use the conference room

hit:reservation- Meeting room reservations can be made up to 1 hour before the start of use.

This output is not just a debug log. This is the basis for judging search quality. It is a good sign if the same correct chunk comes up repeatedly in several expressions. Conversely, if a completely different section is posted for each query, the question expansion may be too broad or the embedding criteria may not match the document expression.

Context compression is reduced after importing.

The chunk retrieved by the retriever does not always have to be displayed as is in the prompt. In particular, if the chunk is long or multiple sentences are mixed within one chunk, it is necessary to take the step of leaving only the sentences necessary for the answer. This flow is context compression.

Let's change the example a bit.

Meeting room reservations can be made up to 1 hour before the start of use.

Please check the condition of the shared equipment after using the conference room.

Reservation changes must be separately notified to attendees.

If the question is “How long can I reserve a conference room?”, the first sentence is key. The second and third sentences are in the same section, but are not directly needed in the answer. Compression plays the role of converting these chunks into shorter contexts.

Copy1# Leave only the sentences directly necessary for the question, but maintain metadata.2def compress_for_question(question: str, docs: list[Document]) -> list[Document]:3    compressed = []4 5    for doc in docs:6        sentences = split_sentences(doc.page_content)7        kept = [8            sentence9            for sentence in sentences10            if "reservation" in sentence and ("Until" in sentence or "start" in sentence)11        ]12 13        if kept:14            compressed.append(15                Document(16                    page_content=" ".join(kept),17                    metadata=doc.metadata,18                )19            )20 21    return compressed

Actual implementations may use more sophisticated compressors. Here, it is expressed as a simple function to demonstrate the principle. The key is to reduce the body of the prompt while maintaining metadata. If the context is reduced but the source or section disappears, it is difficult to connect the answer and evidence later.

Compression helps reduce cost and noise. However, if it is reduced too much, the evidence becomes weak. Only the sentence “Conference room reservations can be made up to 1 hour in advance” remains, and a follow-up condition such as “Attendees must be notified of reservation changes” may be omitted from the question. So the compression result should be viewed side by side with the original chunk.

reranking looks at the order of final candidates again

Even after passing through metadata filter, multi-query, and compression, there is one final problem that remains. This is a case where the candidate is correct, but the order is ambiguous. Vector similarity looks at closeness to the question, but doesn't always guarantee an order that will actually help answer the question.

reranking is the step of rescoring the searched candidates and changing their order. The first search is done broadly, and reranking is done to narrow it down based on the possibility of answering.

Copy1# Candidate chunks are reordered based on the answerability signal.2def rank_for_answer(question: str, docs: list[Document]) -> list[Document]:3    # score is an example heuristic and can be replaced with a separate reranker in actual services.4    def score(doc: Document) -> int:5        text = doc.page_content6        points = 07 8        if "cabinet" in text:9            points += 110        if "reservation" in text:11            points += 112        if "Until" in text or "1 hour" in text:13            points += 214        if doc.metadata.get("section") == "reservation":15            points += 116 17        return points18 19    return sorted(docs, key=score, reverse=True)

This example is intentionally simple. The actual reranker may use a separate model or scoring logic. The point to keep in mind in the blog example is that the retriever does not trust the retrieved order, but rearranges it based on the signals needed for the answer.

After reranking, at least the top few results should be printed and read.

1.reservation- Meeting room reservations can be made up to 1 hour before the start of use. 2.reservation- Please check the status of the shared equipment after using the conference room. 3.equipment- Equipment rentals are recorded at the information desk and then returned.

With this result, you can decide whether to enter only number 1 in the prompt, or enter number 1 and number 2 together. Number 3 comes from the same internal guidance document, but it is better to leave it out for the current question.

Edge case: Strategies don’t turn on all at once

The retriever strategy is not a list of features. It is close to the adjustment order of reading search results and attaching only what is necessary.

Metadata filters reduce the search space. Multi-query retrieval broadens the question expression. Context compression leaves only the parts necessary for the answer within the imported chunk. reranking rearranges the order of candidates.

Even if you turn on these four things at once and the results improve, it is difficult to know why. Conversely, even when it gets worse, it is difficult to find where it went wrong. So we need to change it to small experimental units.

baseline retrieval

Apply to metadata filter

Add to multi-query

to apply compression

to apply reranking

to prompt context confirmed

Ask the same questions at each step and compare the results. Look at “what is the chunk number of the correct answer?”, “have unnecessary sections been reduced?”, “is metadata maintained?”, and “has the context for the prompt been shortened?”

If you keep this order, the retriever strategy will not change to persimmon. It will keep a record of why the search results have improved, and you will know where to start looking again later when the number of documents increases or the question pattern changes.

Criteria for moving on to the next post

After this article, the process of narrowing down search candidates before smarter answers remains.

sample-office-guide.mdno longer remains just a list of chunks. Chunks contain section metadata, questions can be expanded into multiple search expressions, searched contexts can be compressed, and candidates can be reordered. Only after going through this flow can you select the context to enter in the prompt.

In the next article, we look at search quality from a broader perspective. So far, we have focused on “which chunk to import,” and next we move on to how to maintain the document structure. The methods of using keyword search and semantic search together, searching in small chunks and retrieving large parent documents, and indexing one document into multiple expressions are followed here.

The standards are continued from the previous articles. In RAG, the quality of answers does not suddenly improve from the last LLM call. The material of the answer is determined depending on what units the document was divided into, what candidates the retriever retrieved, and how the candidates were reduced and sorted.

Frequently Asked Questions

Q. Is it always right to use the metadata filter first?

A. When the category of the question is clear and the metadata is reliable, it is better to look at it first. However, if the question covers multiple categories, the evidence that requires filtering may be discarded, so the results must be compared.

Q. Is multi-query retrieval the same as query transformation?

A. There is some overlap, but the purpose is slightly different. Query transformation can be used as a flow that changes one search expression, and multi-query retrieval is closer to a method of combining candidates by searching with multiple expressions. You need to make sure that both maintain the intent of the original question.

Q. If you use reranking, is there no need for compression?

A: No. reranking changes the order of candidates, and compression reduces the context within the candidates. Even if long chunks are well placed at the top, compression may still be necessary if it is burdensome to put them as is in the prompt.