Medium · 8 steps · ~45 min

Design an order pipeline on AWS

Accept orders at any speed, process them at your own, and fan each one out to several systems.

Orders arrive in bursts. Writing each one straight to the database ties your speed to the slowest part and loses orders when it falls behind. You will decouple intake from processing with queues, then publish each order to several consumers without the API knowing who they are.

Same problem as Online checkoutTicketingFood ordering

This lab uses real AWS services. Limits are AWS's documented default quotas; figures AWS does not publish are marked as assumptions on each component. See every source and assumption.

How it works

• SQS standard queues take a very high, nearly unlimited number of requests a second and store each message in several Availability Zones before acknowledging it.
• Delivery is at least once: a message can arrive twice, and one that is not deleted becomes visible again after its visibility timeout (30 seconds by default).
• SNS publishes each message to every subscriber, often one SQS queue each. In us-east-1 an account can publish 30,000 messages a second by default.
• Limits come from AWS documentation (default quotas in us-east-1). Where AWS publishes no figure, such as how many requests a container or database handles, the lab states its assumption on the component.

What you'll build

1. A direct design at 1,800 orders a second
2. A queue and workers that absorb a 9,000 per second spike
3. SNS fan-out to billing and email
4. A dead-letter queue for poison messages
5. A worker pool that survives losing a worker
6. Idempotent processing
7. Per-customer ordering with FIFO
8. A database failover that the queue absorbs

Steps

1. 1Take 1,800 orders/s
2. 2Survive a 5× spike
3. 3Fan each order out to billing and email
4. 4A poison message
5. 5Lose a worker
6. 6Safe to run twice
7. 7Keep one customer's orders in order
8. 8Lose the Aurora writer mid-sale