Medium · 14 steps · ~45 min

Design a chatbot over company documents

Employees ask questions; answers come from your documents with citations. Size retrieval, reranking, cache and model within a budget, and survive a cold cache and a provider outage.

Employees ask a chatbot questions that should be answered from the company's own documents: policies, runbooks, handbooks. A model alone would guess, so the system retrieves relevant passages first and gives only those to the model, which answers with citations (retrieval-augmented generation, RAG). You must design both paths: the question path (client, API, semantic cache, vector search, reranker, then the model through an LLM gateway) and the ingestion path (documents uploaded, split, embedded and indexed through a queue and workers). It has to fit a monthly budget, survive a cold cache and a model-provider outage, and respect who is allowed to read what.

Same problem as Enterprise knowledge assistantsSupport copilotsInternal documentation chat

How it works

• Ingest: a document is stored, queued, split into chunks, embedded and indexed with its access list and version.
• Retrieve: the question finds candidate chunks by meaning (and keywords), filtered by what the user may read.
• Rerank: a sharper model re-scores the shortlist and keeps the best few.
• Generate: the model answers only from those passages and returns citations, or says it did not find the answer.

What you'll build

1. The API and the document, chunk and chat data
2. The question path and the ingestion path (queue and workers)
3. A semantic cache and a budget
4. Scaling to 7,000 questions a second with your own GPUs
5. A provider outage, a cold cache, and the full peak design
6. Deep dives: chunking, hybrid search, permissions, freshness, answer not found

Steps

1. 1Design the chat and document API
2. 2Design the document, chunk and chat data
3. 3Answer 400 questions a second from the documents
4. 4Index documents through a queue and workers
5. 5Deep dive: how big should a chunk be?
6. 61,500 questions a second, on a budget
7. 7Deep dive: vector-only or hybrid search?
8. 8Scale to 7,000 questions a second
9. 9Deep dive: permissions-aware retrieval
10. 10The model provider has an outage
11. 11Deep dive: keeping the index fresh, and forgetting
12. 12Survive a cold cache
13. 13Deep dive: when the answer is not in the documents
14. 14Put it together: 8,000 questions a second, provider down