Hard · 13 steps · ~50 min

Design a durable multi-agent workflow

A planner, parallel researchers, a writer and a reviewer, built so a crash, a database failover or a model outage cannot lose or repeat work.

A research-and-report product runs a team of cooperating agents: a planner splits the request into sub-tasks, several researcher agents work in parallel and use tools, a writer assembles the result and a reviewer checks it. A run takes minutes to hours and spends real money on every model and tool call. Processes crash, databases fail over and model providers go down, and the run must survive all three without losing work or repeating a side effect. You will design the event log, the agent workers, the state store, the tool-runner fleet and the model capacity, within a budget.

Same problem as Research and report agentsWorkflow automation platforms

How it works

• Every change in a run is written to a durable event log first. State in the database is a fast copy that can be rebuilt from it.
• Delivery is at least once, so every step and every tool call carries an idempotency key and a repeat is harmless.
• Workers are replaceable. A coordinator resumes unfinished runs from the last checkpoint; retries use backoff and bad events go to a dead-letter store.
• One run fans out into many calls (about 20 model calls, 12 tool calls and 40 state writes). The simulator models this as step events per second, not per run.

What you'll build

1. The run API and its data model
2. An event log, workers and a state database
3. Agent, tool and state consumer groups with model capacity
4. A peak design inside a monthly budget
5. Survival of a worker crash, a state-store failover and a model-provider outage

Steps

1. 1Design the run API
2. 2Design the data model
3. 3Record every step in a durable event log
4. 4Exactly-once effects from at-least-once delivery
5. 5Add the agents, the tools and the model
6. 6Orchestrator-worker or peer agents
7. 7Scale to the evening peak within budget
8. 8Checkpoint granularity
9. 9Survive a worker crash
10. 10Side-effecting tools and compensation
11. 11Fail over the state store
12. 12Poison messages and dead letters
13. 13Survive a model provider outage