Step 1 of 9 · Intro

Design a video upload pipeline on AWS

Take uploads instantly and transcode them in the background without losing any.

How it works

• ›Uploads go to S3, which scales per key prefix (at least 3,500 writes a second per prefix). An S3 event goes to an SQS queue so processing is decoupled from upload.
• ›AWS Elemental MediaConvert runs transcoding jobs. A new on-demand queue in us-east-1 runs 200 jobs at once (up to 500 with a quota increase), and CreateJob accepts 5 requests a second with a burst of 100.
• ›Throughput in jobs a second is concurrent jobs divided by job duration. At one minute a job, 200 concurrent jobs finish about 3.3 a second.
• ›Limits come from AWS documentation (default quotas in us-east-1). Where AWS publishes no figure, such as how many requests a container or database handles, the lab states its assumption on the component.

What you'll build

1. An upload-to-transcode pipeline for 2 videos a second
2. Pre-signed uploads that bypass your servers
3. Bursts absorbed by a queue
4. A worker pool that survives a lost worker