What is RAG? Retrieval augmented generation explained with a real pipeline

RAG (retrieval augmented generation) is a technique where an AI system, before answering, searches your own content for the passages most relevant to the question and passes them to a language model with an instruction to answer from them. The model does not learn your content. It reads it just in time, every time.

Searches for “qué es RAG” are growing fast in Spanish (Google Ads data puts Mexico at 390 a month on average, with the last three months 72% above the first three of the year). We would rather explain it with a system that runs in production than with a diagram: ChatWise, our support chat product, which answers on a website and on WhatsApp from each business’s own content. Every number about ChatWise below comes from its code.

The problem RAG solves

A language model knows what it saw in training. That leaves three gaps when you put it in front of your customers:

• It does not know your content. Your opening hours, your returns policy and your catalogue were never in its training data.
• It makes things up when it does not know. Ask it something it has no answer for and it tends to produce a plausible reply rather than admit it. That is a hallucination.
• Its knowledge is dated. Even if it knew something about your company, it would be the version from months ago.

RAG addresses all three at once. The knowledge lives outside the model, in an index you control and update, and the model only writes the answer from what the index hands it.

The usual alternative is fine-tuning, which retrains the model on your examples. It is good for changing how a model behaves (tone, format, a very specific task) and a poor way to teach it facts that change: every update means another training run, and you cannot point to the document an answer came from. With RAG, changing a price means re-indexing one page.

How RAG works, step 1: indexing your content

Before any question can be answered, the content has to be prepared. This happens once per document and again whenever the document changes.

1. Collect the content. Web pages, PDFs, text documents, FAQs, converted to plain text or Markdown.
2. Split it into chunks. A whole document is too large and too general to compare with a question, so it is cut into pieces of a few hundred words, usually with some overlap so an idea is not cut in half.
3. Embed each chunk. An embedding model turns each chunk into a vector: a list of hundreds or thousands of numbers that represents its meaning. Two texts that say similar things end up with nearby vectors even when they use different words.
4. Store it in a vector database with metadata. Next to each vector you keep the original text and data such as the URL, the page number or the source document. That metadata is what later lets you cite the source.
5. Keep one collection per customer or site. If the system serves several businesses, each needs its own collection, so a search for one business can never return another’s text.

How RAG works, step 2: answering the question

When a question arrives, the system:

1. Embeds the question with the same model used for the chunks. It has to be the same one, because vectors from different models cannot be compared.
2. Retrieves the closest chunks (the top-k, for example the five most similar).
3. Optionally improves that list. A reranker can reorder the candidates with a more precise model, and hybrid search can add results from keyword search.
4. Builds the prompt: system instructions, the retrieved chunks and the question, with an instruction to answer only from that context.
5. Generates the answer with the language model.
6. Cites or falls back. Ideally it records which chunks were used, so they can be shown or audited, and has a way out when the content is not enough: saying it does not know, or handing the conversation to a person.

The real case: how ChatWise does it

ChatWise has a Python backend (FastAPI) that owns the whole RAG pipeline, built on LlamaIndex with Qdrant as the vector database. This is what it does at each step, according to its code.

Indexing

• Sources. A business can add files, URLs, pasted text or a sitemap. PDFs are read page by page and each page keeps its page number. URLs are fetched through a scraping service (Firecrawl) and stored as Markdown. Pasted text must be between 50 and 100,000 characters and is uploaded as one more file. For a sitemap, the system discovers every URL, creates a source for each one that does not exist yet and queues its own job.
• Job queue. Each source becomes a job in Postgres that a separate worker picks up. A failed job is retried automatically, and the source is marked as failed once retries run out. There is also a re-ingestion job, so any source can be processed again.
• Chunks. ChatWise uses LlamaIndex’s sentence splitter with its standard settings, which cuts chunks of up to 1,024 tokens with a 200-token overlap.
• Embeddings. By default, OpenAI’s text-embedding-3-large, which produces 3,072-dimension vectors. A site can be configured with another model, and its collection’s vector size is set from that model when the collection is created.
• Storage. Every site gets its own Qdrant collection, with a random name and cosine distance. Each chunk carries the source id, the site id, the source type and, depending on the source, the URL, the file name or the page number as metadata.

At question time

• First, decide whether this is a RAG question. A language model acts as a router: it reads the message and returns JSON with a target and a confidence score. When it is unsure, or something fails, the message goes to RAG by default. Handing off to a person is reserved for explicit requests (“I want to talk to a human”) and goes through a second check that throws out messages like “thanks” or “ok” without spending another model call. How to design that handoff is covered in a RAG chatbot on your own content.
• The query includes the conversation. ChatWise searches with the latest message and with what came before it. It builds a query from up to eleven previous messages plus the current one (“Conversation so far: … Latest user message: …”, in the original Spanish template), and that whole text is what gets embedded and searched. That is how a follow-up like “and how much is it?” arrives knowing which product was being discussed.
• Retrieval. It runs a vector search over the site’s collection and keeps the 5 most similar chunks, with the conversation already folded into the query.
• Prompt. It joins the system prompt the business configured, the retrieved chunks between separators, and the instruction to answer “given the context information and not prior knowledge”.
• Generation. A language model writes the answer, with fallbacks to other providers: if one model fails, it tries the next.
• Sources. Every answer from the knowledge base is saved with the chunks behind it: their id, similarity score, source type and metadata (the URL or the PDF page). Tokens and latency are recorded too.

Three retrieval rules from our other systems

ChatWise is not the only place we retrieve information. In the systems that hold our own agents’ memory (described in agent memory in practice) we follow three rules that apply to any RAG system.

Pair semantic search with full-text search. Vector search finds similar meanings and tends to miss exactly what people search for literally: product names, IDs and error codes. In our search over past sessions we combine vectors with BM25 keyword search and merge the two lists with reciprocal rank fusion, which rewards whatever ranks high in either one. We recommend hybrid search for any RAG system whose users search for names, codes or acronyms.

Use one embedding model per system. Every model produces its own vector space, so changing it means re-embedding the whole corpus. Our memory service uses Qwen3-Embedding-8B through the API of Tessera. The model outputs 4,096 dimensions and supports truncation; we store 1,536 because pgvector’s HNSW index on the standard vector type tops out at 2,000 dimensions. ChatWise, on the other hand, uses text-embedding-3-large at 3,072 dimensions in Qdrant. Different systems, different models, and each uses the same model to index and to query. In ChatWise’s code the rule is visible: each collection’s vector size is fixed by the model when it is created.

Keep the index rebuildable from the source. An upgrade to a search tool can change its index format. Our session search builds its index from Markdown summaries, so when the format changes, the old index is set aside and rebuilt from those summaries with nothing lost. The index is derived; the source is what you protect. ChatWise follows the same principle: it keeps each source’s original files and URLs, so re-ingestion is just another job in the queue.

RAG vs fine-tuning vs an LLM wiki

There are three common ways to give an AI system knowledge, and they solve different problems:

They are not mutually exclusive. Our company runs on an LLM wiki, and nothing stops you from putting RAG on top of one.

When do you need RAG?

It makes sense when several of these are true:

• Your content is larger than fits comfortably in a prompt.
• It changes often: prices, stock, policies, documentation.
• You need to know where each answer came from.
• You serve several customers or sites whose information must stay separate.

You probably do not need it if all your content fits in a system prompt (a few pages of FAQs), or if what you want to change is the model’s style rather than what it knows. And before putting it in front of customers, measure whether it retrieves the right chunks: that is what the evals you actually need covers.

If you are weighing RAG for your support or your internal documents, talk to us. It is part of the AI agents and automations we build and run for our clients.