Medium · 11 steps · ~60 min

Design object storage (S3-like) and pastebin

Start with a pastebin, grow it into a durable store for files of any size.

You begin with a pastebin: paste text, share a link. Then people want to upload photos, videos and backups. The service has to store bytes for years without losing any, serve popular files quickly, and keep the bill under control.

Same problem as Amazon S3PastebinGoogle Cloud Storage

How it works

• Write: create an object or upload a big file in parts (rare, large).
• Read: fetch by key, often the same popular objects (read-heavy).
• Keep: never lose a byte, even when disks and servers die.

What you'll build

1. The API and the metadata-versus-blob data model
2. A cached metadata path for a viral paste
3. Object storage for bytes, with a CDN for reads
4. Replication for metadata, and designs that survive a cold CDN, a lost storage node and a lost primary

Steps

1. 1Design the API
2. 2Design the data model
3. 3A pastebin that works
4. 4A viral paste
5. 5Why split metadata from blobs?
6. 6Add large files and a CDN
7. 7Durability: what does eleven nines mean?
8. 8Survive a cold CDN
9. 9Lose a blob storage node
10. 10Lose the metadata primary
11. 11Deep dive: big uploads