Step 1 of 11 · Intro

Design log aggregation and search

Ingest a firehose of logs, search them in seconds, and keep them cheaply.

How it works

• ›Ingest: a constant firehose of small writes with sudden spikes (write-heavy).
• ›Search: bursts of text queries over a time range (read, latency-sensitive).
• ›Retain: old data is rarely read but must be kept for months.

What you'll build

1. The API and the data model
2. A stream that buffers bursts, with indexing workers
3. A search index and a query cache
4. A cold archive tier to cut retention cost
5. Designs that survive a lost indexer and a cold cache