One customer message becomes many model and tool calls. Size the workers, models, tool runners and approvals, cap the spend and survive outages.
How it works
›A run is a loop, so it lives on a queue and agent workers, not inside one web request. The customer gets an immediate 'accepted'.
›One message turns into about ten internal requests (five model calls, five tool calls), so the fleets are sized for that multiplied rate, not for the message rate.
›Caps on steps, time and spend are checked before every step; a tripped cap ends in a clean stop and a handoff.
›Tools that change money are tiered by risk, run in a sandbox, and take idempotency keys so a retry never repeats the effect.
›A model you host yourself keeps the agent alive when the provider fails, and a spare tool runner covers a lost machine.
What you'll build
The agent API and its data model
Queue, agent workers, gateway and a hosted model
A tool-runner fleet beside the model fleet
An approval queue for risky actions
A limiter and breakers that cap the spend
A design that survives a provider outage, a lost tool runner and a peak day