Hard · 14 steps · ~60 min

Design a chat product at scale

Wire the whole stack behind a conversational AI product: streaming, per-plan limits, guardrails, a cache, hosted and self-hosted models and a history store that survives failures.

You are building the stack behind a conversational AI product used by millions of people: free and paid plans, long conversations, answers that arrive word by word, and a history that must still be there tomorrow. The earlier labs each built one piece: the LLM gateway, the moderation pipeline, the semantic cache and the inference service. Here you build a different thing, the whole product end to end: a connection layer, API servers, per-plan limits, a safety stage, a cache, a gateway over hosted models with your own GPUs behind it, and a conversation store. Then you break it: a provider outage, a cold cache, a lost database primary and a lost GPU, always inside a monthly budget.

Same problem as Consumer chat assistantsConversational AI products

How it works

• One chat turn touches many systems (connection, API, limits, safety check, cache, model, history), and each has its own capacity and its own way to fail.
• Cheap stages go in front of expensive ones: plan limits, then rules and classifiers, then the cache, and only then a model.
• Hosted models give you elasticity and your own GPUs give you a lower price per request; a gateway lets you use both and lose either one.
• History is the product's memory. It must survive a lost primary and keep growing, so it is replicated first and sharded later.

What you'll build

1. The chat API and the data model behind it
2. Connection layer, API servers and a conversation store on a hosted model
3. A limiter, a safety stage, a semantic cache and a read replica
4. Your own GPU fleet next to the hosted model
5. Designs that survive a provider outage, a cold cache, a lost primary and a lost GPU at peak

Steps

1. 1Design the chat API
2. 2Design the data model
3. 3Launch: connections, API servers and a hosted model
4. 4Storing and trimming conversation history
5. 5Grow to 6,000 a second: limits, safety, a cache, a replica
6. 6Quotas by plan and fairness
7. 7Add your own GPUs next to the hosted model
8. 8Safety layers and what to log
9. 9The model provider has an outage
10. 10Survive a cold cache at the evening peak
11. 11Streaming fan-out and connection capacity
12. 12Lose the conversation store's primary
13. 13Regional deployment and data residency
14. 14Millions of users: shard the store and lose a GPU at peak