Hard · 8 steps · ~55 min

Design a streaming analytics pipeline on AWS

Ingest a firehose of events, keep them in order where it matters, and feed storage and dashboards.

A product emits clickstream events, thousands a second. You need to collect them durably, process them in near real time, archive the raw events and power a dashboard. Throughput on Kinesis is bought in shards, so sizing is arithmetic you can do.

Same problem as Clickstream analyticsIoT telemetryApplication logs

This lab uses real AWS services. Limits are AWS's documented default quotas; figures AWS does not publish are marked as assumptions on each component. See every source and assumption.

How it works

• Each Kinesis shard accepts up to 1,000 records or 1 MB per second, whichever comes first. Records here are 1 KB or smaller, so the record limit binds.
• Each shard supports reads of up to 2 MB a second, shared by consumers that poll. Enhanced fan-out gives each registered consumer its own 2 MB a second per shard, up to 20 consumers.
• A record's partition key decides its shard. Records with the same key go to the same shard, in order. Retention is 24 hours by default, up to 365 days.
• Limits come from AWS documentation (default quotas in us-east-1). Where AWS publishes no figure, such as how many requests a container or database handles, the lab states its assumption on the component.

What you'll build

1. A stream sized for 3,000 events a second
2. A stream resized for 12,000
3. A plan for several consumers of one stream
4. Dashboards fed from the same stream
5. A partition key that keeps a customer's events in order

Steps

1. 1Ingest 3,000 events/s
2. 2Grow to 12,000 events/s
3. 3Three teams, one stream
4. 4Feed a dashboard
5. 5Keep a customer's events in order
6. 6Right-size to $1,850 a month
7. 7On-demand or provisioned?
8. 8Reprocess after a bug