Hard · 11 steps · ~75 min

Design a Kafka-like message queue

A durable, partitioned log that producers append to and consumer groups replay.

Design a distributed log: producers append messages to named topics, and many independent consumer groups read them at their own pace. It must scale to hundreds of thousands of messages a second, keep messages in order per key, survive a broker dying without losing acknowledged data, and let consumers replay history.

Same problem as Apache KafkaAmazon SQSRabbitMQ

How it works

• Produce: append a message to a partition chosen by its key.
• Fetch: consumers read a partition from an offset they track.
• Replicate: each partition has a leader and followers on other brokers.

What you'll build

1. The API and the log schema
2. A stream with consumer workers
3. Fan-out to several consumer groups
4. Partitioned brokers and wide consumer groups at 120k messages/s
5. A design that survives a lost broker, a lost consumer and a lost offset store

Steps

1. 1Design the API
2. 2Design the data model
3. 3One topic, one consumer group
4. 4A second consumer group for $2,600
5. 5Ordering guarantees
6. 6Scale to 120,000 messages/sec
7. 7Durability and replication
8. 8Lose a broker at 120,000 messages/sec
9. 9Lose a consumer
10. 10Offsets and delivery semantics
11. 11Lose the offset database at 20,000 messages/sec