Hard · 12 steps · ~80 min

Design a web search engine

Crawl the web, index it, and answer queries in milliseconds.

Build a search engine: discover and fetch pages, turn them into an index, and answer millions of queries a second with good results that reflect today's web.

Same problem as Google SearchBingDuckDuckGo

How it works

• Crawl: fetch pages politely and prioritise what changes (write-heavy, background).
• Index: turn pages into an inverted index split across machines.
• Query: parse, fan out to shards, rank, and return in tens of milliseconds (read-heavy).

What you'll build

1. The API and the crawl, page and index data
2. A cached query path
3. A crawl-to-index pipeline with a stream
4. A sharded index behind edge and result caches
5. Designs that survive a cold cache, a lost shard and a lost indexer

Steps

1. 1Design the API
2. 2Design the data model
3. 3The first index
4. 4Cache popular queries
5. 5The crawl-to-index pipeline
6. 6Crawl freshness
7. 7Serve 200,000 queries a second
8. 8How to shard the index
9. 9Ranking
10. 10Survive a cold cache
11. 11Lose an index shard
12. 12Lose an indexing worker