Hard · 14 steps · ~50 min

Design a customer-support agent

One customer message becomes many model and tool calls. Size the workers, models, tool runners and approvals, cap the spend and survive outages.

A support agent reads a customer's message, then loops: it asks a model what to do, calls a tool (look up an order, check a policy, issue a refund up to a limit, send a reply), and finishes or hands the conversation to a person. One message therefore becomes many internal requests, each slow and some expensive. You must run these loops reliably, bound what they can spend and do, keep humans in charge of risky actions, and stay up when the model provider or a tool runner fails.

Same problem as Customer-support copilotsHelpdesk automationRefund and order assistants

How it works

• A run is a loop, so it lives on a queue and agent workers, not inside one web request. The customer gets an immediate 'accepted'.
• One message turns into about ten internal requests (five model calls, five tool calls), so the fleets are sized for that multiplied rate, not for the message rate.
• Caps on steps, time and spend are checked before every step; a tripped cap ends in a clean stop and a handoff.
• Tools that change money are tiered by risk, run in a sandbox, and take idempotency keys so a retry never repeats the effect.
• A model you host yourself keeps the agent alive when the provider fails, and a spare tool runner covers a lost machine.

What you'll build

1. The agent API and its data model
2. Queue, agent workers, gateway and a hosted model
3. A tool-runner fleet beside the model fleet
4. An approval queue for risky actions
5. A limiter and breakers that cap the spend
6. A design that survives a provider outage, a lost tool runner and a peak day

Steps

1. 1Design the support-agent API
2. 2Design the data model
3. 3Run the agent loop off the request path
4. 4Step, time and spend caps
5. 5Add the tool-runner fleet and size the workers
6. 6Which actions need a human?
7. 7Route risky actions into an approval queue
8. 8Idempotent tools and duplicate refunds
9. 9Cap the spend with a limiter and breakers
10. 10Prompt injection through customer text and tool output
11. 11Survive a model-provider outage
12. 12Lose a tool runner
13. 13Handing off to a human
14. 14Peak day with a failing inference server