Hard · 14 steps · ~50 min
Design an AI code assistant
Instant inline completions, streamed chat and sandboxed code runs, with per-user limits, a fast model fleet and an outage plan.
Developers use an assistant inside their editor in two ways. Inline completions are many tiny requests that must feel instant, or they are worthless. Chat and agent mode are fewer, longer streamed requests that may run code or tests. One heavy user must not slow everyone down, the bill must be bounded, untrusted generated code must be contained, and a model provider outage must not stop developers from typing.
Same problem as IDE coding assistantsPull-request review bots
How it works
- Completions are high-volume and latency-critical: a small fast model on your own servers plus a cache beats a large hosted model.
- Chat is long and streamed, so a realtime gateway holds the connections and hosted models supply quality.
- Generated code is untrusted: run it in short-lived sandboxes behind a queue.
- Limits at the user and the organisation layer keep both fairness and the bill under control.
- A fallback you already own turns a provider outage into a slightly slower day.
What you'll build
- The assistant API and its data model
- A limiter, gateway and fast completion fleet
- A completion cache and a streamed chat path
- A queued sandbox fleet that survives a lost host
- One combined design that survives a provider outage and a lost GPU
Steps
- 1Design the assistant API
- 2Design the data model
- 3Put a front door and a fast model in front of the editor
- 4Cache repeated completions
- 5Completion latency budget and stale requests
- 6Per-user quotas versus organisation quotas
- 7Stream chat answers through a realtime gateway
- 8Run generated code in a sandbox fleet
- 9Isolating untrusted generated code
- 10Lose a tool runner
- 11What code may be sent to the model
- 12Put it all together
- 13Survive a provider outage
- 14Lose a GPU server too