Step 1 of 13 · Intro

Design a web search engine

Crawl the web, index it, and answer queries in milliseconds.

How it works

• ›Crawl: fetch pages politely and prioritise what changes (write-heavy, background).
• ›Index: turn pages into an inverted index split across machines.
• ›Query: parse, fan out to shards, rank, and return in tens of milliseconds (read-heavy).

What you'll build

1. The API and the crawl, page and index data
2. A cached query path
3. A crawl-to-index pipeline with a stream
4. A sharded index behind edge and result caches
5. Designs that survive a cold cache, a lost shard and a lost indexer