Hard · 14 steps · ~60 min

Design a company AI platform

Build one shared AI platform for dozens of teams: gateway, retrieval, evaluation, guardrails, GPUs, metering and chargeback, sized for rising load and the worst day, inside a budget.

A company has dozens of product teams, and each one calls a model provider on its own: separate keys, separate retries, separate bills, no shared safety checks, and no answer to 'what do we spend, and what happens when the provider is down?'. Leadership wants one shared AI platform: a central LLM gateway (auth, per-team quotas, caching, routing and fallback), a shared retrieval service, an evaluation service that gates releases, safety guardrails by default, a self-hosted GPU fleet beside hosted models, usage metering for chargeback, and observability. You will size it as load rises, keep the bill within budget, and survive four incidents. It builds on the gateway, RAG, evaluation, inference and moderation labs, but the question here is platform-level: what is shared, what is isolated, and who pays.

Same problem as Internal AI platformsShared model gateways

How it works

• Sharing makes capacity cheaper (one busy fleet, one cache) and failures wider (one bad deploy hurts every team), so the platform needs quotas, priorities and isolation.
• Inside the platform one call becomes many: guardrails, cache, retrieval, model, metering. This lab counts requests per second including those internal calls.
• Money is a requirement: every call is metered to a team, and the monthly bill must fit a budget on the worst day too.
• Metering and evaluation are essential but must never block a user request, so they hang off a topic beside the request path.

What you'll build

1. The platform API and its data model
2. A shared front door: gateway, quotas, cache, hosted and GPU models
3. Shared retrieval and guardrails
4. Metering, budgets and an evaluation sampler beside the request path
5. Designs that survive a provider outage, a cold cache, a lost GPU and a lost metering node

Steps

1. 1Design the platform API
2. 2Design the platform's data model
3. 3One front door for every team
4. 4Central platform or team-owned stacks?
5. 5Shared retrieval and guardrails
6. 6Golden paths and guardrails by default
7. 7Meter every call and gate every release
8. 8Chargeback or showback?
9. 9The hosted provider goes down
10. 10Quotas and priority in a shortage
11. 11A prompt-version bump empties the cache
12. 12Versioning and deprecating models
13. 13Lose a GPU node at peak
14. 14Lose a metering node