Labs ›streaming analytics pipeline on AWS0 / 9 steps
Step 1 of 9 · Intro
Design a streaming analytics pipeline on AWS
Ingest a firehose of events, keep them in order where it matters, and feed storage and dashboards.
How it works
›Each Kinesis shard accepts up to 1,000 records or 1 MB per second, whichever comes first. Records here are 1 KB or smaller, so the record limit binds.
›Each shard supports reads of up to 2 MB a second, shared by consumers that poll. Enhanced fan-out gives each registered consumer its own 2 MB a second per shard, up to 20 consumers.
›A record's partition key decides its shard. Records with the same key go to the same shard, in order. Retention is 24 hours by default, up to 365 days.
›Limits come from AWS documentation (default quotas in us-east-1). Where AWS publishes no figure, such as how many requests a container or database handles, the lab states its assumption on the component.
What you'll build
A stream sized for 3,000 events a second
A stream resized for 12,000
A plan for several consumers of one stream
Dashboards fed from the same stream
A partition key that keeps a customer's events in order