Hard · 10 steps · ~75 min

Design log aggregation and search

Ingest a firehose of logs, search them in seconds, and keep them cheaply.

Thousands of services write log lines. Engineers search them during incidents, and auditors need months of history. Ingest must never lose data or slow the apps, search must be fast, and storing everything must not bankrupt you.

Same problem as Elasticsearch / ELKSplunkDatadog Logs

How it works

• Ingest: a constant firehose of small writes with sudden spikes (write-heavy).
• Search: bursts of text queries over a time range (read, latency-sensitive).
• Retain: old data is rarely read but must be kept for months.

What you'll build

1. The API and the data model
2. A stream that buffers bursts, with indexing workers
3. A search index and a query cache
4. A cold archive tier to cut retention cost
5. Designs that survive a lost indexer and a cold cache

Steps

1. 1Design the API
2. 2Design the data model
3. 3Search a small deployment
4. 4Absorb a burst without losing logs
5. 5When indexers fall behind
6. 6Keep 90 days without paying for it
7. 7What makes search fast
8. 8Lose an indexer
9. 9Survive a cold query cache
10. 10Deep dive: retention and cost