Step 1 of 12 · Intro

Meet the product

Design a distributed log: producers append messages to named topics, and many independent consumer groups read them at their own pace. It must scale to hundreds of thousands of messages a second, keep messages in order per key, survive a broker dying without losing acknowledged data, and let consumers replay history.

Message queue

Steps · 1 of 12

Message queue

Labs › Kafka-like message queue0 / 12 steps

Step 1 of 12 · Intro

Design a Kafka-like message queue

A durable, partitioned log that producers append to and consumer groups replay.

How it works

›Produce: append a message to a partition chosen by its key.
›Fetch: consumers read a partition from an offset they track.
›Replicate: each partition has a leader and followers on other brokers.

What you'll build

The API and the log schema
A stream with consumer workers
Fan-out to several consumer groups
Partitioned brokers and wide consumer groups at 120k messages/s
A design that survives a lost broker, a lost consumer and a lost offset store