Skip to content

RAG chatbot over your documents: cite sources, admit gaps

From the work/ 14 min read

Build a RAG chatbot: split documents into chunks, search by embeddings, a rules prompt and Claude API citations. TypeScript code with tests and a PDF guide.

  • AI
  • Databases
  • JavaScript
In this article (9)
  1. What RAG is
  2. Splitting documents into chunks
  3. Embeddings and searching the chunks
  4. The rules prompt: only from this, cite the source, say you don't know
  5. A minimal RAG chatbot example
  6. How to evaluate the chatbot's answers
  7. Common mistakes
  8. Frequently asked questions
  9. Checklist

A RAG chatbot answers from your documents, not from whatever the model remembers. You split each document into chunks and store each one’s embedding; for every question you fetch the closest ones and give them to the model with three rules: answer only from them, cite the source, and say “I don't have that information” when it isn't there.

Below you get the flow, a minimal TypeScript example with the Claude API, and how to evaluate the answers. In Node we tested the document splitter, the search and the request builder; the model call and the embeddings call are replaced with test versions, so we did not measure the quality of any real model. Meaning-based search is already covered in semantic product search with embeddings and pgvector; this builds on top of it.

What RAG is

RAG stands for retrieval-augmented generation: before asking the model, your code searches your documents and pastes what it found into the prompt. The model does not learn your policies; it reads them on every question. Edit a document and the next answer already reflects it, and you can show where each fact came from.

There are two phases. Indexing happens once, and again when a document changes: split into chunks, compute an embedding for each and store it. Asking happens on every message: embed the question, take the k closest chunks, build the prompt and get the answer.

RAG in two phases: index once, ask every time1 · index, and again when a document changesdocumentspolicies, FAQchunkschunkText()embeddingsembed()indexpgvector or memory2 · ask, on every messagequestionsame embed()search top-kretrieve() · minScoreprompt + rulessearch_result blocksmodelclaude-opus-5-5“Shipping to Denver takes 2 to 4 business days.” [shipping-policy]or, if it is not in the documents: “I don't have that information.”the model reads your documents on every questiontested locally with stubbed embeddings: no provider was called

Scroll the figure sideways to see all of it

Is it always needed? No. In September 2024, Anthropic wrote in Introducing Contextual Retrieval that if your knowledge base is smaller than 200,000 tokens (about 500 pages), you can put the whole thing in the prompt, and that prompt caching makes that faster and cheaper. A store's policy pages usually fit. Try both options on your own questions before choosing.

Splitting documents into chunks

The embedding of a whole page blends topics; the embedding of a chunk with one idea looks a lot like the questions about that idea. According to the same Anthropic article, chunk size, boundaries and overlap affect retrieval, and RAG chunks are usually at most a few hundred tokens. Our starting point: whole paragraphs, up to 800 characters, with the last paragraph repeated at the start of the next chunk. This is chunkText:

// Whole paragraphs only; the last `overlap` paragraphs of a chunk open the next one.
// shortcut: a single paragraph longer than maxChars stays one chunk, split it by sentences if your docs have them.
export function chunkText(text: string, maxChars = 800, overlap = 1): string[] {
  const paragraphs = text.split(/\n{2,}/).map((p) => p.trim()).filter(Boolean);
  const chunks: string[] = [];
  let current: string[] = [];
  let fresh = 0; // paragraphs in `current` that are not overlap
  for (const p of paragraphs) {
    if (fresh > 0 && current.join("\n\n").length + p.length > maxChars) {
      chunks.push(current.join("\n\n"));
      current = current.slice(Math.max(0, current.length - overlap));
      fresh = 0;
    }
    current.push(p);
    fresh++;
  }
  if (fresh > 0) chunks.push(current.join("\n\n"));
  return chunks;
}
  1. split(/\n{2,}/) splits at blank lines; trim and filter(Boolean) drop the empty ones.
  2. current holds the paragraphs of the chunk being built and fresh counts the new ones, the ones that are not overlap.
  3. If there are new paragraphs and the next one would exceed maxChars, it closes the chunk and opens another with the last overlap paragraphs. fresh > 0 stops it from emitting a chunk made only of overlap.
  4. It emits whatever is left. A paragraph longer than maxChars stays as one chunk (the shortcut comment); split it by sentences if your documents have huge ones.

With a three-paragraph document and maxChars = 90, only so the effect is visible, the real output was:

chunkText(shippingPolicy, 90, 1)
[
  'Orders are prepared within 24 business hours.',
  'Orders are prepared within 24 business hours.\n\nShipping to Denver and Austin takes 2 to 4 business days.',
  'Shipping to Denver and Austin takes 2 to 4 business days.\n\nShipping is free on purchases over 75 dollars.'
]

The second chunk opens with the paragraph that closed the first: that is the overlap. Before computing the embedding we prepend the document title, a cheap version of Anthropic's idea of adding context to every chunk. Their full technique, which generates that context with a model, cut the top-20 retrieval failure rate in their own tests by 35% with contextual embeddings (from 5.7% to 3.7%), 49% when adding contextual BM25 and 67% with a reranker. Those are their numbers; we did not measure them.

Embeddings and searching the chunks

Each chunk becomes an embedding with the model of an embeddings provider (Anthropic does not offer one of its own). Use the same model for chunks and questions, and store each vector with its source, its title and the text. Searching means sorting by distance and keeping the k closest; 3 to 5 is a good start. In Postgres with pgvector it is an ORDER BY embedding <=> $1 LIMIT 4; in the example, to avoid depending on a database, it is an in-memory pass:

// Embeddings come back with length 1, so the dot product is the cosine similarity.
const dot = (a: number[], b: number[]) => a.reduce((sum, x, i) => sum + x * b[i], 0);

export function retrieve(query: number[], index: Chunk[], k = 4, minScore = 0.3): Hit[] {
  return index
    .map((chunk) => ({ chunk, score: dot(query, chunk.embedding) }))
    .filter((hit) => hit.score >= minScore)
    .sort((a, b) => b.score - a.score)
    .slice(0, k);
}
  1. Embeddings come back with length 1, so the dot product (dot) is the cosine similarity.
  2. map scores every chunk against the question and filter drops whatever does not reach minScore: the threshold.
  3. sort and slice(0, k) keep the k best.

Without a threshold there are always four “closest” chunks, even when they have nothing to do with the question, and the model will try to answer from them. The 0.3 above suits the stubbed embeddings of the test; with a real model, look at the scores of a few questions that do have an answer and a few that don't, and set the cut between the two. In our test, the Denver question scored 0.41 and 0.33 on two chunks of the shipping policy and the question outside the documents scored 0 on all of them: no chunks came back and the model was never called.

The rules prompt: only from this, cite the source, say you don't know

The rules go in system: answer only from the search results, every fact comes from a result and is cited, and if the answer is not there, say exactly that you don't have the information. We add a fourth: do not invent time frames or amounts. For citations we use the Claude API's search_result blocks, described in its search results documentation: each chunk travels with source, title, content and citations: { enabled: true }, and Claude answers with text blocks whose citations are of type search_result_location (source, title, cited_text and the block position). What the documentation says:

  • search_result blocks can only appear in user messages, including tool results, and only with text. Claude cites whole blocks: for finer citations, split the content into smaller blocks. All the results in a request must use the same citations.enabled value.
  • According to the citations documentation, citations are valid pointers to the documents you sent, and cited_text does not count toward output tokens. Whether the sentence is actually supported by that chunk is not guaranteed by anyone: you evaluate that.
  • Citations do not combine with structured outputs: enabling them together with output_config.format returns a 400 error. If you need structured data, ask for one line per field and parse it, or read how to get reliable JSON from an LLM.
export const NO_INFO = "I don't have that information in the documents. Do you want me to connect you with a person?";

export const RULES = `You are the support assistant of an online store.
Rules:
1. Answer ONLY from the search results you receive. Use no outside knowledge.
2. Every fact (time frame, price, condition) must come from a result and be cited.
3. If the answer is not in the results, reply exactly: "I don't have that information in the documents. Do you want me to connect you with a person?"
4. Do not invent policies or amounts. Answer in English, in three sentences at most.`;
  1. NO_INFO is the exact phrase for “I don't know”. The code uses it as well when there are no chunks, so the reply is identical on both paths.
  2. RULES holds the four rules; the third repeats that phrase in quotes so the model returns it verbatim.
export function buildRequest(question: string, hits: Hit[]): Anthropic.MessageCreateParamsNonStreaming {
  const results = hits.map(({ chunk }) => ({
    type: "search_result" as const,
    source: chunk.source,
    title: chunk.title,
    content: [{ type: "text" as const, text: chunk.text }],
    citations: { enabled: true },
  }));
  return {
    model: "claude-opus-5-5",
    max_tokens: 1024,
    system: RULES,
    messages: [{ role: "user", content: [...results, { type: "text" as const, text: question }] }],
  };
}
  1. model is the one the Claude documentation examples use as of November 12, 2026; swap in the one you use. system carries the rules.
  2. One search_result block is created per retrieved chunk, with citations enabled; as const keeps the literals the way TypeScript expects them.
  3. The question goes last, as a text block, after the results.
every fact in the answer points at a chunk you sentchunks sent[1] shipping-policyShipping to Denver and Austintakes 2 to 4 business days.[2] returns-policyWe accept returns for 30days after delivery.question and answerquestionHow long does shippingto Denver take?answer2 to 4 business days [1].search_result_locationquestion outside the documents: “Do you sell hiking boots?”retrieve() returns [] and the model is not even called“I don't have that information in the documents.”the citation carries source, title and cited_textsample documents invented for this figure

Scroll the figure sideways to see all of it

A minimal RAG chatbot example

With the import and the types at the top of the file, answer takes the API client, your provider's embeddings function, the index and the question:

import Anthropic from "@anthropic-ai/sdk";

export interface Chunk { source: string; title: string; text: string; embedding: number[] }
export interface Hit { chunk: Chunk; score: number }
export async function answer(
  client: Pick<Anthropic, "messages">,
  embed: (text: string) => Promise<number[]>,
  index: Chunk[],
  question: string,
) {
  const hits = retrieve(await embed(question), index);
  if (hits.length === 0) return { text: NO_INFO, sources: [] as string[] }; // no model call
  const res = await client.messages.create(buildRequest(question, hits));
  if (res.stop_reason === "refusal") return { text: NO_INFO, sources: [] as string[] };
  const text = res.content.map((b) => (b.type === "text" ? b.text : "")).join("");
  const sources = new Set<string>();
  for (const b of res.content) {
    if (b.type !== "text") continue;
    for (const c of b.citations ?? []) if (c.type === "search_result_location") sources.add(c.source);
  }
  return { text, sources: [...sources] };
}
  1. retrieve(await embed(question), index) searches with the question's embedding. If no chunk reaches the threshold, it returns NO_INFO without calling the model: cheaper, and nothing to invent from.
  2. buildRequest builds the request and client.messages.create sends it. If it ends with stop_reason equal to refusal, it also returns NO_INFO.
  3. text joins the text blocks and sources collects the source of the citations, without repeats.

This is how the index is built and a question is asked (the same loop the test uses, with the stubbed embedding):

const docs = [
  { source: "shipping-policy", title: "Shipping policy", text: "…the document text…" },
];

const index: Chunk[] = [];
for (const doc of docs) {
  for (const text of chunkText(doc.text)) {
    // embed(): the call to your embeddings provider
    index.push({ source: doc.source, title: doc.title, text, embedding: await embed(`${doc.title}\n${text}`) });
  }
}

const reply = await answer(new Anthropic(), embed, index, "How long does shipping to Denver take?");
console.log(reply.text, reply.sources);

We ran 8 tests with Node 22.22.3 (node --test), 4 in Spanish and 4 in English, and all 8 passed: chunkText neither loses nor duplicates paragraphs and repeats the overlap; the search puts the right source first and returns nothing outside the documents; the request carries the search_result blocks with citations, the question last and the claude-opus-5-5 model; and answer reads the sources from the citations, does not call the model outside the documents and returns NO_INFO on a refusal. The code passes tsc in strict mode against @anthropic-ai/sdk 1.8.0. We ran no real call to the model or to an embeddings provider: both are stubbed, so the tests say nothing about answer quality. There are no retries, streaming or history either.

How to evaluate the chatbot's answers

You need no tooling: a sheet with 20 to 30 real customer questions (from emails or chats), the source that should answer each one, and about five your documents do not answer. Run it every time you change something, and change one thing at a time: chunk size, k, minScore, the model or the rules. Measure the search and the answer separately: if the right chunk never reaches the model, no rule fixes it. Anthropic measures retrieval the same way, as the share of cases where the right chunk is not among the top 20.

What to measureHowIt is wrong if…
RetrievalIs the expected source among the k chunks?It is missing on questions that do have an answer
FaithfulnessDoes every fact appear in the cited chunk? Check by hand at firstThere are time frames, prices or conditions that are not there
RefusalQuestions outside the documents return NO_INFOIt answers something instead of saying it doesn't know
CitationThe cited source is the chunk that contains the factIt cites another document or none
typical RAG chatbot mistakes and where to looksymptomlikely causewhat to checkinvents a time frameno threshold, k too highminScore and rule 3says no when it had itchunk cut in halfsize and overlapcites another documentnear-identical titlesunique source and titlestale informationindex not rebuiltre-index on editmixes customersindex without permissionsfilter before searchingmeasure the search first, then the promptif the right chunk never reaches the model, the rules cannot fix it

Scroll the figure sideways to see all of it

Common mistakes

  • Mixing embedding models. If the chunks were indexed with one model and the question is computed with another, the vectors are not comparable. If you change models, re-index everything.
  • Having no threshold. There is always a “closest”. Without minScore, the model gets irrelevant chunks and tries to answer anyway.
  • Trusting the prompt alone. The rules help, they do not guarantee: check with your question sheet that the chatbot refuses what it does not know.
  • Leaving old documents in. If you edit a policy and don't re-index, the chatbot answers with the previous one.
  • Indexing what the customer must not see. If the index mixes data from several customers, filter by permissions before searching, not after answering.
  • Treating documents as instructions. Retrieved text is content, not orders. If someone else edits the documents, a chunk could ask to ignore the rules: do not give the model tools with side effects and no confirmation.

Frequently asked questions

RAG or fine-tuning?

If what changes is the data, such as time frames, prices or policies, RAG: you edit the document and the next answer uses it, with the source cited. Fine-tuning is for changing the style or format of the answers; we did not evaluate it here.

Do I need a vector database?

Not necessarily. For a handful of documents, the in-memory pass in the example is enough. When it grows, store the vectors in your Postgres with pgvector, as in the semantic search article.

What do I use to generate the embeddings?

The model of an embeddings provider: Anthropic does not offer one of its own. Pick one, test it with your question sheet and use the same one for chunks and questions. The code does not depend on the provider: embed is a function you pass in.

What happens if someone asks something that is not in my documents?

There are two safeguards: the threshold, which avoids calling the model when no chunk is close, and the third rule, which asks for the exact NO_INFO phrase when the chunk does not answer. Always offer to hand over to a person.

Checklist

  • Documents are split into one-idea chunks, with the title and source stored next to each vector.
  • Chunks and questions use the same embeddings model.
  • minScore was calibrated with questions that have an answer and others that don't.
  • Every chunk travels as a search_result with citations.enabled set to true.
  • The system asks to answer only from the results, cite them and say when it doesn't know.
  • With no chunk above the threshold, the code replies NO_INFO without calling the model.
  • There is a sheet of 20 to 30 questions, five unanswerable, and it runs on every change.
  • Documents are re-indexed when edited and the index respects each customer's permissions.

If you don't have the vectors yet, start with semantic product search with embeddings and pgvector. And if the chatbot has to return data rather than text, see how to get reliable JSON from an LLM, remembering that citations and structured outputs cannot be used together.

Saw this on Instagram? Comment RAG on the Reel and I'll send you the guide.

Written by

Luis Morales

Luis Morales

Frontend and Shopify developer, EcomLabs

@luismoralesdev on Instagram

Published November 12, 2026 · updated November 18, 2026

← Blog