Labs › AI code assistant0 / 15

Step 1 of 15 · Intro

Design an AI code assistant

Instant inline completions, streamed chat and sandboxed code runs, with per-user limits, a fast model fleet and an outage plan.

How it works

  • ›Completions are high-volume and latency-critical: a small fast model on your own servers plus a cache beats a large hosted model.
  • ›Chat is long and streamed, so a realtime gateway holds the connections and hosted models supply quality.
  • ›Generated code is untrusted: run it in short-lived sandboxes behind a queue.
  • ›Limits at the user and the organisation layer keep both fairness and the bill under control.
  • ›A fallback you already own turns a provider outage into a slightly slower day.

What you'll build

  1. The assistant API and its data model
  2. A limiter, gateway and fast completion fleet
  3. A completion cache and a streamed chat path
  4. A queued sandbox fleet that survives a lost host
  5. One combined design that survives a provider outage and a lost GPU