Wire the whole stack behind a conversational AI product: streaming, per-plan limits, guardrails, a cache, hosted and self-hosted models and a history store that survives failures.
How it works
›One chat turn touches many systems (connection, API, limits, safety check, cache, model, history), and each has its own capacity and its own way to fail.
›Cheap stages go in front of expensive ones: plan limits, then rules and classifiers, then the cache, and only then a model.
›Hosted models give you elasticity and your own GPUs give you a lower price per request; a gateway lets you use both and lose either one.
›History is the product's memory. It must survive a lost primary and keep growing, so it is replicated first and sharded later.
What you'll build
The chat API and the data model behind it
Connection layer, API servers and a conversation store on a hosted model
A limiter, a safety stage, a semantic cache and a read replica
Your own GPU fleet next to the hosted model
Designs that survive a provider outage, a cold cache, a lost primary and a lost GPU at peak