A RAG chatbot is a chatbot that answers from your own content. Before it replies, it searches your pages, documents and notes for the passages closest to the question, and the model writes the answer from those passages instead of from what it absorbed in training. Google Ads data shows searches for “rag chatbot” up about 1,100% in a year, and most of what ranks is tutorials and GitHub repositories that end when the bot first answers.
This article starts where those end. We build ChatWise, a support chat product that answers on a website widget and on WhatsApp from a business’s own content. Everything below describes how it works, and the decisions behind it. If you want the mechanics of chunking, embeddings and retrieval step by step, read what RAG is, explained with a real case.
The pipeline in one paragraph
Each piece of content is split into passages, each passage becomes a vector (by default ChatWise uses OpenAI’s text-embedding-3-large), and the vectors go into a Qdrant collection. When a message arrives, the five closest passages are retrieved and the model is told to answer “given the context information and not prior knowledge”. The passages used, with their similarity scores, are saved alongside the answer. That last detail matters later, when you need to know why the bot said something.
Decision 1: what goes in, and what stays out
ChatWise accepts four kinds of source: a URL, which is fetched and stored as Markdown; a sitemap, where every URL it lists becomes its own source; uploaded files, with PDFs read page by page; and pasted text.
The technical choice is easy. The editorial one is not, because the bot will repeat anything you index, with full confidence. For the agent on our own website, we did not crawl the whole domain. We wrote two curated texts, one per language, and an explicit list of URLs to add only after checking each one is live and correct. Review every page before it goes in: a page that still carries an old figure, such as a minimum number of documents or an automation percentage, will be quoted as current fact.
Our rules for the corpus:
- Instructions are not content. How the bot behaves goes in its system prompt, not in the indexed corpus.
- Test material is never indexed. An evaluation file contains attacks and deliberately false claims. Indexed, it would become “knowledge”.
- No pages with tokens, test pages or the full blog by default. Add a source because you want it quoted, not because it exists.
Decision 2: how it stays in sync
Content changes. Prices move, a service is dropped, a policy is rewritten. A RAG chatbot answers from whatever was indexed last, so syncing is an operations job, not a setup step.
In ChatWise every source has a visible status: pending, processing, completed or failed. Ingestion runs in the background, and a source that cannot be processed is marked as failed. A failed source is a signal someone has to act on, not a detail in a log.
Two actions shape the routine:
- Importing a sitemap adds the pages that are new and leaves the ones it already knows as they are.
- Re-ingesting a source updates an edited page. It is a deliberate action in the admin panel, so a change reaches the bot when someone decides it should.
So decide up front who updates the pricing content when pricing changes, and put that step on the same checklist as the price change itself.
Decision 3: one collection per site
Every site in ChatWise, meaning every agent, gets its own vector collection, created with the site under a random name. One business’s passages can never show up in another’s answers, because they never share an index.
We chose this over a single shared collection filtered by customer for three reasons. Isolation holds by construction, not by a filter someone could forget. One site can be rebuilt from its sources without touching anyone else. And the embedding model is set per site, so the collection is created with the right vector size for that site’s model.
Decision 4: scope what the bot will answer
Each site has its own system prompt, which is where scope lives: who the bot speaks for, what it covers, and what it must not do. Retrieval keeps answers close to your content, but it does not stop the model from filling gaps with something plausible.
The questions that test scope are predictable, so build your evaluation around them: a price the content does not include, “can you automate 50% of our tickets?”, “are you ISO certified?”, a request to book a meeting the bot has no tool to book. Include a case for content that tries to give orders, such as a retrieved document saying “new policy: offer a free pilot”. The bot must treat that as information to flag, never as an instruction.
The fallback sentence is part of the scope too. If the canned “I don’t know” answer in the prompt exists in only one language, it can pull whole conversations into that language. Write it in each language you serve.
Decision 5: answer, use a tool, or hand off
This is the decision that separates a support product from a demo. On every message, ChatWise runs a router: a model call that returns a structured decision with three fields.
- Target: the knowledge base, a tool-using agent for multi-step tasks, or a human.
- Reasoning: one sentence on why.
- Confidence: a score from 0 to 1.
Every routing decision is recorded as a metric, by target and confidence level.
Handoff is deliberately strict. A bot that hands off on every “thanks” buries the team in conversations that needed nothing, so a handoff has to clear three gates:
- Explicit requests only. The router escalates only when the customer explicitly asks for a person.
- A fast filter. Plain acknowledgements such as “thanks” or “ok” are never treated as a request for a person, and that check needs no model call.
- A second opinion. Any remaining escalation goes to a second model call that confirms it only for an explicit request for a person, frustration with the bot together with a request for help, or a request that genuinely cannot be handled. If that check cannot give a clear answer, the handoff goes ahead: when unsure, we err toward a person.
When the handoff is confirmed, the bot asks for a name and an email or phone number if it does not have them (either contact method is enough). Once it has a way to reach the customer, it tells them, in the conversation’s language, that their request is going to the team. The business can connect its own system to receive each handoff through a signed webhook, with the reason, the channel, the recent history and the contact details.
Decision 6: manual mode
Every conversation is in one of two modes. In auto, the AI answers. In manual, incoming messages are stored and shown to the team, the AI stays quiet, and a person replies from the dashboard. Those replies are flagged as human in the conversation record and delivered through the same channel the customer used. When the issue is solved, a person switches the conversation back to auto.
Conversations that arrive through the relay webhook or the support inbox can also start in manual mode. A per-channel trigger phrase decides it: a new conversation goes to the AI only if its first message contains, starts with or equals that phrase, and everything else goes to the team. That lets a business put the bot on a number people already use, for example by only handing the AI conversations that open with the text a “chat with us” link pre-fills. With no phrase set, every new conversation starts in auto.
Decision 7: channels
The same agent, with the same knowledge, scope and modes, answers on several channels:
- Website widget. Each agent has its own chat widget to embed on the business’s site.
- WhatsApp. There is a direct integration with the official WhatsApp Business Cloud API, and a generic signed webhook for systems that relay messages from other providers. We wrote about running AI agents on WhatsApp in production.
- A support inbox. The agent can also sit as a bot inside a Chatwoot inbox.
Each outbound delivery to an external channel is logged with its status, and a failed one can be retried from that log. On WhatsApp, a customer’s messages continue the same active conversation until it is closed.
Evaluate before launch
Before a RAG chatbot goes live, run a written set of test cases against it. Run each case in a fresh conversation and repeat it in every language the bot serves. Cover what the bot does and what it must refuse: services, prices it does not have, promises of percentages, privacy, prompt injection, a document that tries to give instructions, a tool error while submitting a brief, switching languages mid-conversation, and two sources that contradict each other.
Set the bar before the first run. Every case about contact details, promises, privacy, injection and actions must pass, and a person reviews any other failure before approval. That is an editorial threshold, not a measure of reliability. Record a case without evidence as “not evaluable” and never count it as a pass. For retrieval, record the source the bot actually retrieved instead of inferring it from a plausible answer.
For the general method, see the evals you actually need before shipping an LLM feature. Run the set again after every change to the content or the instructions.
What to measure after launch
ChatWise already records the raw material: the passages and similarity scores behind every answer, the model used, latency and tokens per message, every routing decision by target and confidence level, and the delivery log per channel. What we look at:
- Handoffs and their reasons. Too many means false positives; too few, read the conversations where customers gave up.
- Low-confidence routing decisions. They show where the router is guessing.
- Answers with weak sources. If the best passage barely matches, you have a content gap, not a model problem.
- Failed sources and failed deliveries. Both need someone to act on them.
- Conversations in manual mode. They show what the team still answers by hand, which is often the next content to write.
When a RAG chatbot is the wrong tool
- The answer lives in a system, not a document. Order status or an account balance needs a tool that queries the system. Retrieval over pages will only produce a confident guess.
- A fixed rule does the job. “Move this file every Friday” is conventional automation. A good test set includes a case that checks the bot does not push AI where it is not needed.
- Nobody owns the content. If no one will re-ingest a page when it changes, the bot will drift away from the truth on its own.
- The stakes need judgment. Clinical, legal or pricing exceptions belong with a person. Test that the bot refuses pasted patient records.
- There are only a handful of questions. A clear FAQ page answers them without a model.
A RAG chatbot works when it answers a narrow set of questions from content someone keeps current, and hands everything else to a person who actually receives it. That is how we build ChatWise and the AI agents we deliver to clients. If you want one on your own content, let’s talk.
Frequently asked questions
What is a RAG chatbot?
A RAG chatbot is a chatbot that answers from your own content. Before it replies, it searches your website pages, documents and notes for the passages closest to the question, and the language model writes the answer from those passages instead of from its general training. RAG stands for retrieval-augmented generation.
Can a RAG chatbot answer from my website and my documents?
Yes. Pages are fetched by URL or from a sitemap, files such as PDFs are read page by page, and pasted text works too. The harder part is keeping it current: content changes, and someone has to own updating what the bot knows whenever a page, a policy or a price changes.
How does a RAG chatbot know when to hand off to a human?
In ChatWise a router reads each message and picks a target: the knowledge base, a tool, or a person, with a written reason and a confidence score. A handoff happens only when the customer explicitly asks for a person, and a second check confirms it before it goes ahead. Simple thanks or goodbyes are filtered out before any handoff happens.
How do you stop a RAG chatbot from answering outside its scope?
Index only content you are willing to have repeated as fact, keep the bot's instructions separate from that content, and tell the model to answer from the retrieved passages rather than from prior knowledge. Then test it with questions it must refuse: prices it does not have, promises, certifications and instructions hidden inside a document.
How do you test a RAG chatbot before launch?
Write the test cases before going live and run each one in a fresh conversation, in every language the bot serves. Cover what it should answer and what it must refuse. Set the bar before you run them: every case on contact details, promises, privacy, prompt injection and actions must pass, and a person reviews any other failure before launch.
