RAG chatbot over your documents: cite sources, admit gaps
From the work/ 14 min read
Build a RAG chatbot: split documents into chunks, search by embeddings, a rules prompt and Claude API citations. TypeScript code with tests and a PDF guide.
- AI
- Databases
- JavaScript
In this article (9)
A RAG chatbot answers from your documents, not from whatever the model remembers. You split each document into chunks and store each one’s embedding; for every question you fetch the closest ones and give them to the model with three rules: answer only from them, cite the source, and say “I don't have that information” when it isn't there.
Below you get the flow, a minimal TypeScript example with the Claude API, and how to evaluate the answers. In Node we tested the document splitter, the search and the request builder; the model call and the embeddings call are replaced with test versions, so we did not measure the quality of any real model. Meaning-based search is already covered in semantic product search with embeddings and pgvector; this builds on top of it.
What RAG is
RAG stands for retrieval-augmented generation: before asking the model, your code searches your documents and pastes what it found into the prompt. The model does not learn your policies; it reads them on every question. Edit a document and the next answer already reflects it, and you can show where each fact came from.
There are two phases. Indexing happens once, and again when a document changes: split into chunks, compute an embedding for each and store it. Asking happens on every message: embed the question, take the k closest chunks, build the prompt and get the answer.
Scroll the figure sideways to see all of it
Is it always needed? No. In September 2024, Anthropic wrote in Introducing Contextual Retrieval that if your knowledge base is smaller than 200,000 tokens (about 500 pages), you can put the whole thing in the prompt, and that prompt caching makes that faster and cheaper. A store's policy pages usually fit. Try both options on your own questions before choosing.
Splitting documents into chunks
The embedding of a whole page blends topics; the embedding of a chunk with one idea looks a lot like the questions about that idea. According to the same Anthropic article, chunk size, boundaries and overlap affect retrieval, and RAG chunks are usually at most a few hundred tokens. Our starting point: whole paragraphs, up to 800 characters, with the last paragraph repeated at the start of the next chunk. This is chunkText:
// Whole paragraphs only; the last `overlap` paragraphs of a chunk open the next one.
// shortcut: a single paragraph longer than maxChars stays one chunk, split it by sentences if your docs have them.
export function chunkText(text: string, maxChars = 800, overlap = 1): string[] {
const paragraphs = text.split(/\n{2,}/).map((p) => p.trim()).filter(Boolean);
const chunks: string[] = [];
let current: string[] = [];
let fresh = 0; // paragraphs in `current` that are not overlap
for (const p of paragraphs) {
if (fresh > 0 && current.join("\n\n").length + p.length > maxChars) {
chunks.push(current.join("\n\n"));
current = current.slice(Math.max(0, current.length - overlap));
fresh = 0;
}
current.push(p);
fresh++;
}
if (fresh > 0) chunks.push(current.join("\n\n"));
return chunks;
}
split(/\n{2,}/)splits at blank lines;trimandfilter(Boolean)drop the empty ones.currentholds the paragraphs of the chunk being built andfreshcounts the new ones, the ones that are not overlap.- If there are new paragraphs and the next one would exceed
maxChars, it closes the chunk and opens another with the lastoverlapparagraphs.fresh > 0stops it from emitting a chunk made only of overlap. - It emits whatever is left. A paragraph longer than
maxCharsstays as one chunk (theshortcutcomment); split it by sentences if your documents have huge ones.
With a three-paragraph document and maxChars = 90, only so the effect is visible, the real output was:
chunkText(shippingPolicy, 90, 1)
[
'Orders are prepared within 24 business hours.',
'Orders are prepared within 24 business hours.\n\nShipping to Denver and Austin takes 2 to 4 business days.',
'Shipping to Denver and Austin takes 2 to 4 business days.\n\nShipping is free on purchases over 75 dollars.'
]
The second chunk opens with the paragraph that closed the first: that is the overlap. Before computing the embedding we prepend the document title, a cheap version of Anthropic's idea of adding context to every chunk. Their full technique, which generates that context with a model, cut the top-20 retrieval failure rate in their own tests by 35% with contextual embeddings (from 5.7% to 3.7%), 49% when adding contextual BM25 and 67% with a reranker. Those are their numbers; we did not measure them.
Embeddings and searching the chunks
Each chunk becomes an embedding with the model of an embeddings provider (Anthropic does not offer one of its own). Use the same model for chunks and questions, and store each vector with its source, its title and the text. Searching means sorting by distance and keeping the k closest; 3 to 5 is a good start. In Postgres with pgvector it is an ORDER BY embedding <=> $1 LIMIT 4; in the example, to avoid depending on a database, it is an in-memory pass:
// Embeddings come back with length 1, so the dot product is the cosine similarity.
const dot = (a: number[], b: number[]) => a.reduce((sum, x, i) => sum + x * b[i], 0);
export function retrieve(query: number[], index: Chunk[], k = 4, minScore = 0.3): Hit[] {
return index
.map((chunk) => ({ chunk, score: dot(query, chunk.embedding) }))
.filter((hit) => hit.score >= minScore)
.sort((a, b) => b.score - a.score)
.slice(0, k);
}
- Embeddings come back with length 1, so the dot product (
dot) is the cosine similarity. mapscores every chunk against the question andfilterdrops whatever does not reachminScore: the threshold.sortandslice(0, k)keep the k best.
Without a threshold there are always four “closest” chunks, even when they have nothing to do with the question, and the model will try to answer from them. The 0.3 above suits the stubbed embeddings of the test; with a real model, look at the scores of a few questions that do have an answer and a few that don't, and set the cut between the two. In our test, the Denver question scored 0.41 and 0.33 on two chunks of the shipping policy and the question outside the documents scored 0 on all of them: no chunks came back and the model was never called.
The rules prompt: only from this, cite the source, say you don't know
The rules go in system: answer only from the search results, every fact comes from a result and is cited, and if the answer is not there, say exactly that you don't have the information. We add a fourth: do not invent time frames or amounts. For citations we use the Claude API's search_result blocks, described in its search results documentation: each chunk travels with source, title, content and citations: { enabled: true }, and Claude answers with text blocks whose citations are of type search_result_location (source, title, cited_text and the block position). What the documentation says:
search_resultblocks can only appear in user messages, including tool results, and only with text. Claude cites whole blocks: for finer citations, split the content into smaller blocks. All the results in a request must use the samecitations.enabledvalue.- According to the citations documentation, citations are valid pointers to the documents you sent, and
cited_textdoes not count toward output tokens. Whether the sentence is actually supported by that chunk is not guaranteed by anyone: you evaluate that. - Citations do not combine with structured outputs: enabling them together with
output_config.formatreturns a 400 error. If you need structured data, ask for one line per field and parse it, or read how to get reliable JSON from an LLM.
export const NO_INFO = "I don't have that information in the documents. Do you want me to connect you with a person?";
export const RULES = `You are the support assistant of an online store.
Rules:
1. Answer ONLY from the search results you receive. Use no outside knowledge.
2. Every fact (time frame, price, condition) must come from a result and be cited.
3. If the answer is not in the results, reply exactly: "I don't have that information in the documents. Do you want me to connect you with a person?"
4. Do not invent policies or amounts. Answer in English, in three sentences at most.`;
NO_INFOis the exact phrase for “I don't know”. The code uses it as well when there are no chunks, so the reply is identical on both paths.RULESholds the four rules; the third repeats that phrase in quotes so the model returns it verbatim.
export function buildRequest(question: string, hits: Hit[]): Anthropic.MessageCreateParamsNonStreaming {
const results = hits.map(({ chunk }) => ({
type: "search_result" as const,
source: chunk.source,
title: chunk.title,
content: [{ type: "text" as const, text: chunk.text }],
citations: { enabled: true },
}));
return {
model: "claude-opus-5-5",
max_tokens: 1024,
system: RULES,
messages: [{ role: "user", content: [...results, { type: "text" as const, text: question }] }],
};
}
modelis the one the Claude documentation examples use as of November 12, 2026; swap in the one you use.systemcarries the rules.- One
search_resultblock is created per retrieved chunk, withcitationsenabled;as constkeeps the literals the way TypeScript expects them. - The question goes last, as a text block, after the results.
Scroll the figure sideways to see all of it
A minimal RAG chatbot example
With the import and the types at the top of the file, answer takes the API client, your provider's embeddings function, the index and the question:
import Anthropic from "@anthropic-ai/sdk";
export interface Chunk { source: string; title: string; text: string; embedding: number[] }
export interface Hit { chunk: Chunk; score: number }
export async function answer(
client: Pick<Anthropic, "messages">,
embed: (text: string) => Promise<number[]>,
index: Chunk[],
question: string,
) {
const hits = retrieve(await embed(question), index);
if (hits.length === 0) return { text: NO_INFO, sources: [] as string[] }; // no model call
const res = await client.messages.create(buildRequest(question, hits));
if (res.stop_reason === "refusal") return { text: NO_INFO, sources: [] as string[] };
const text = res.content.map((b) => (b.type === "text" ? b.text : "")).join("");
const sources = new Set<string>();
for (const b of res.content) {
if (b.type !== "text") continue;
for (const c of b.citations ?? []) if (c.type === "search_result_location") sources.add(c.source);
}
return { text, sources: [...sources] };
}
retrieve(await embed(question), index)searches with the question's embedding. If no chunk reaches the threshold, it returnsNO_INFOwithout calling the model: cheaper, and nothing to invent from.buildRequestbuilds the request andclient.messages.createsends it. If it ends withstop_reasonequal torefusal, it also returnsNO_INFO.textjoins the text blocks andsourcescollects thesourceof the citations, without repeats.
This is how the index is built and a question is asked (the same loop the test uses, with the stubbed embedding):
const docs = [
{ source: "shipping-policy", title: "Shipping policy", text: "…the document text…" },
];
const index: Chunk[] = [];
for (const doc of docs) {
for (const text of chunkText(doc.text)) {
// embed(): the call to your embeddings provider
index.push({ source: doc.source, title: doc.title, text, embedding: await embed(`${doc.title}\n${text}`) });
}
}
const reply = await answer(new Anthropic(), embed, index, "How long does shipping to Denver take?");
console.log(reply.text, reply.sources);
We ran 8 tests with Node 22.22.3 (node --test), 4 in Spanish and 4 in English, and all 8 passed: chunkText neither loses nor duplicates paragraphs and repeats the overlap; the search puts the right source first and returns nothing outside the documents; the request carries the search_result blocks with citations, the question last and the claude-opus-5-5 model; and answer reads the sources from the citations, does not call the model outside the documents and returns NO_INFO on a refusal. The code passes tsc in strict mode against @anthropic-ai/sdk 1.8.0. We ran no real call to the model or to an embeddings provider: both are stubbed, so the tests say nothing about answer quality. There are no retries, streaming or history either.
How to evaluate the chatbot's answers
You need no tooling: a sheet with 20 to 30 real customer questions (from emails or chats), the source that should answer each one, and about five your documents do not answer. Run it every time you change something, and change one thing at a time: chunk size, k, minScore, the model or the rules. Measure the search and the answer separately: if the right chunk never reaches the model, no rule fixes it. Anthropic measures retrieval the same way, as the share of cases where the right chunk is not among the top 20.
| What to measure | How | It is wrong if… |
|---|---|---|
| Retrieval | Is the expected source among the k chunks? | It is missing on questions that do have an answer |
| Faithfulness | Does every fact appear in the cited chunk? Check by hand at first | There are time frames, prices or conditions that are not there |
| Refusal | Questions outside the documents return NO_INFO | It answers something instead of saying it doesn't know |
| Citation | The cited source is the chunk that contains the fact | It cites another document or none |
Scroll the figure sideways to see all of it
Common mistakes
- Mixing embedding models. If the chunks were indexed with one model and the question is computed with another, the vectors are not comparable. If you change models, re-index everything.
- Having no threshold. There is always a “closest”. Without
minScore, the model gets irrelevant chunks and tries to answer anyway. - Trusting the prompt alone. The rules help, they do not guarantee: check with your question sheet that the chatbot refuses what it does not know.
- Leaving old documents in. If you edit a policy and don't re-index, the chatbot answers with the previous one.
- Indexing what the customer must not see. If the index mixes data from several customers, filter by permissions before searching, not after answering.
- Treating documents as instructions. Retrieved text is content, not orders. If someone else edits the documents, a chunk could ask to ignore the rules: do not give the model tools with side effects and no confirmation.
Frequently asked questions
RAG or fine-tuning?
If what changes is the data, such as time frames, prices or policies, RAG: you edit the document and the next answer uses it, with the source cited. Fine-tuning is for changing the style or format of the answers; we did not evaluate it here.
Do I need a vector database?
Not necessarily. For a handful of documents, the in-memory pass in the example is enough. When it grows, store the vectors in your Postgres with pgvector, as in the semantic search article.
What do I use to generate the embeddings?
The model of an embeddings provider: Anthropic does not offer one of its own. Pick one, test it with your question sheet and use the same one for chunks and questions. The code does not depend on the provider: embed is a function you pass in.
What happens if someone asks something that is not in my documents?
There are two safeguards: the threshold, which avoids calling the model when no chunk is close, and the third rule, which asks for the exact NO_INFO phrase when the chunk does not answer. Always offer to hand over to a person.
Checklist
- Documents are split into one-idea chunks, with the title and source stored next to each vector.
- Chunks and questions use the same embeddings model.
minScorewas calibrated with questions that have an answer and others that don't.- Every chunk travels as a
search_resultwithcitations.enabledset totrue. - The
systemasks to answer only from the results, cite them and say when it doesn't know. - With no chunk above the threshold, the code replies
NO_INFOwithout calling the model. - There is a sheet of 20 to 30 questions, five unanswerable, and it runs on every change.
- Documents are re-indexed when edited and the index respects each customer's permissions.
If you don't have the vectors yet, start with semantic product search with embeddings and pgvector. And if the chatbot has to return data rather than text, see how to get reliable JSON from an LLM, remembering that citations and structured outputs cannot be used together.
Saw this on Instagram? Comment RAG on the Reel and I'll send you the guide.
Written by

Published November 12, 2026 · updated November 18, 2026