The Search Before the Answer

Imagine asking a shop's AI assistant:

I bought X200 headphones last week. They've stopped working. Can I bring them back?

The assistant needs the shop's policy to answer. One way to supply it is to search the shop's documents, find useful passages, and give those passages to AI alongside the question. AI then uses that material to draft an answer.

This approach is called retrieval-augmented generation, usually shortened to RAG: find the information, give it to AI, then generate the answer. We will follow a system that searches by both meaning and words.

Before the customer asks a question, the software behind the assistant prepares the handbook for search. We will call this software the application. It first gets readable text from the documents. For a PDF, it extracts the words from the pages. A scanned page may need a tool that recognises text in an image.

The handbook covers deliveries, repairs, payments and returns. We divide its text into smaller passages so a returns rule can be searched separately from delivery instructions. These passages are called chunks.

One passage says:

Opened headphones may be returned if they are faulty. Proof of purchase is required.

Those sentences belong together. Cutting after "Opened headphones may be returned" loses the fault condition and the proof-of-purchase requirement. A chunk might be a sentence, a paragraph or a section. Keep enough text for the rule to make sense on its own.

Our customer says "bring them back". The handbook says "returned". Searching only for matching words could miss the connection.

To search by meaning, the application sends each passage to an AI model that calculates a list of numbers from its text. The model is trained so that texts with related meanings tend to receive similar lists, even when their words differ. This is an embedding model. Its number list is an embedding, also called a vector.

The numbers help us compare texts. An individual number does not usually have a label we can read as "headphones", "returns" or "faulty".

For our example, use a list with just one number. The returns passage receives [24]: the brackets hold the list, and 24 is its one entry. We can draw it at 24 on a number line. Embedding models normally return longer lists; we use one-number lists here so we can follow the comparison on a line.

The assistant needs the rule's original words to answer the customer, so we save them alongside the number list used to find it. We also keep where the passage came from. One saved record contains:

Record ID: returns-01

Text: Opened headphones may be returned if they are faulty. Proof of purchase is required.

Number list: [24]

Source: Shop handbook → Returns

The ID identifies this particular passage. The source reference, including its link when available, lets us find the original section.

The search service organises the records into a lookup structure, called an index. Think of a book's index helping you find the right pages. A search index organises records so the application can look for matching words or similar number lists.

This preparation happens before the questions arrive. We reuse the prepared records for later questions, and update them when the documents change.

The handbook is now ready for search. When the customer asks a question, the application sends its complete wording through the same embedding model used for the passages. Its number list can then be compared with the lists saved for the handbook.

In our small example:

I bought X200 headphones last week. They've stopped working. Can I bring them back? → embedding model → [25]

The question receives [25], so we draw it at 25. The returns passage has [24], so it sits at 24, one step away. The model calculated both lists; the drawing helps us see their comparison.

Suppose the saved passages have lists [20], [23], [24], [27], [28], [50] and [70]. Each has one number, so each appears at that position on the line.

In this example, closest means the smallest numerical gap. The passage at 24 is one step from the question at 25. The passages at 23 and 27 are both two steps away. The passage at 70 is 45 steps away.

If we request the five closest passage lists, we select [24], [23], [27], [28] and [20]:

This setting is often written as k = 5: return five matches. Five is our choice for this example. For longer embeddings, search compares the complete lists using a comparison method suited to the model.

Each matching list belongs to a saved passage record. Selecting [24] leads the application to returns-01, where it retrieves the original faulty-headphones rule and its source reference. It retrieves the other matching records in the same way. The words were saved alongside the numbers all along.

Those passages are candidates: possible evidence for an answer. A passage can be close in meaning but describe unopened products, repairs or another product's returns. Even a passage about faulty headphones could be missing the proof-of-purchase condition.

Making the whole handbook one passage can hide a specific rule among unrelated topics. Cutting it too finely can lose the condition that makes the rule useful. Test the split with actual questions and inspect the text the search returns.

The passages are only one possible source of confusion. The search question can lose context too.

Our full question already includes the product, timing and fault, so the application can use it directly. Everyday wording such as "bring them back" is fine. In a longer conversation, though, those details might be spread across several messages.

But suppose those details appeared in earlier chat messages, and the application sends only "Can I bring them back?" to search. That short question no longer says what "them" means or why the customer wants a return.

An optional AI step can receive the short question and the relevant earlier messages, then write a search question that makes sense on its own:

What does this shop's policy say about returning faulty X200 headphones bought last week?

This is query rewriting. It carries known details into the search; it should not assume a receipt or invent anything the customer has not said.

The rewritten question then goes through the same embedding model to calculate the number list used for search.

When this optional step is needed, it happens before both searches. Word search uses the rewritten question's words. Meaning search uses its new number list. The closer passage check described below, called reranking, uses that complete search question too. The application keeps the customer's original question and relevant earlier messages for the final answer.

For our headphone example, the question was complete already, so we continue with its original wording. It also includes the product code "X200". Searching for that exact code gives us another way to find useful passages alongside the meaning search.

For the complete question in our example, meaning-based search finds the five passages at 24, 23, 27, 28 and 20. They include the faulty-headphones rule at 24 and the unopened-products rule at 23. It connects "bring them back" with "returned" by comparing number lists.

Word-based search looks for words and codes from the question in the saved text. The code "X200" might find repair information at 50 and a rule about unopened X200 headphones at 23. Both searches return passage records with readable text and source references.

Each method can find something the other misses. The faulty-headphones rule might never mention "X200". A passage with "X200" might concern repairs rather than returns. Combining word and meaning search is called hybrid search.

We have five meaning matches and two word matches. The unopened-products passage has ID unopened-01. Both searches return that same ID, so the application keeps the record once. That leaves six different passages. The ID identifies the saved record, whichever search found it.

Combining the searches widens our selection. The application also combines the two searches' rankings to put the passages in one order. We now need to check which of those six passages will help answer the customer.

We now have six candidates, including passages about different situations. The application can give them a closer check before asking AI for an answer. This step is called reranking.

The meaning search compared number lists. A common text-based reranker gets the search question and each candidate's actual words. It considers the question and passage together and estimates relevance, then puts the most useful-looking passages first.

A passage found by both methods can move higher in the combined order even when it describes the wrong situation. Suppose three of our six candidates arrive in this order: an unopened-products rule, X200 repair information, and the faulty-headphones rule. A useful reranking could move the faulty-headphones rule ahead of the other two:

The unopened rule concerns a different situation. The repair passage names the right product but does not say whether it can be returned. The faulty-headphones rule includes both the fault condition and the proof-of-purchase requirement.

After the reranker orders the six candidates, the application chooses how many of the leading passages to include, with a maximum of five in our example. Five is a limit, so it can send fewer. The checks below help us test whether the selected text contains the evidence needed for an answer. The earlier k = 5 controlled how many meaning matches we requested. This later limit controls how much selected text reaches AI.

A reranker can still get the order wrong. It only considers the candidates supplied to it; it does not discover a missing policy or restore text cut out during chunking. Reranking is optional and adds processing, so test whether it improves results for your questions.

The application now assembles the request for an answer. It sends the customer's original question, the selected passages and their source references, plus instructions for using that evidence. If the question depends on earlier messages, it includes those known chat details too.

For our customer, part of that request could look like this:

Customer question:

I bought X200 headphones last week. They've stopped working. Can I bring them back?

Selected passage:

Opened headphones may be returned if they are faulty. Proof of purchase is required.

Source reference: Source 1, Shop handbook → Returns, with the link saved alongside the passage.

Instructions:

Answer using the supplied policy. Keep its conditions. Point to the source. If it does not answer part of the question, say what is missing.

A supported answer could be:

According to the shop's returns policy, opened headphones may be returned if they are faulty, and proof of purchase is required. Do you have a receipt or another proof of purchase? [Source 1]

The application can display Source 1 as a link to the policy. That helps the customer check the answer, but the passage still has to support the claim. "You'll get a full cash refund today without a receipt" would go beyond this text: it requires proof of purchase and says nothing about refund method or timing.

We can now check whether this reply gives our customer an answer the handbook supports. Keep the question, selected passages and answer visible together. Start with this example, then try a few other questions you can check against the handbook.

Did the needed rule reach AI? For our customer, the selected text should include the fault condition and proof-of-purchase requirement. If they are missing, inspect the chunk boundaries and search results. Also check that the cited policy is current and applies to this product.

Did the answer preserve that rule? It should retain the conditions and avoid adding promises about refunds or timing. Open the cited section and compare its words with the answer.

Also try: "Will I receive a cash refund today?" The supplied passage does not answer that question. The assistant should say what the policy does not establish.

These checks help separate two failures: the application supplied incomplete information, or AI received the right passage but wrote beyond it. Seeing the evidence alongside the answer tells you where to start fixing the system.

Our customer asked in everyday language, while the answer depended on a rule in the shop's handbook. RAG connects that question to the document by finding passages AI can use when writing its reply.

The work begins before the question arrives. Preparing searchable passages keeps the rule with its conditions. Here, meaning search connects "bring them back" with "returned", while word search helps find details such as "X200". Reading the shortlist more closely can bring the passage that applies to the customer's situation to the front.

Those stages lead to the same handoff: AI receives the question with useful, readable text from the documents and a reference to its source. That is how RAG brings information from a document into an AI answer.