Medium · 8 steps · ~35 min

Design a video upload pipeline on AWS

Take uploads instantly and transcode them in the background without losing any.

Users upload videos and expect an immediate acknowledgement, but transcoding each into several formats takes minutes. You will separate the two with storage events and a queue, and see which limits decide how many videos you can finish per second.

Same problem as Video sharingPodcast hostingOnline courses

This lab uses real AWS services. Limits are AWS's documented default quotas; figures AWS does not publish are marked as assumptions on each component. See every source and assumption.

How it works

• Uploads go to S3, which scales per key prefix (at least 3,500 writes a second per prefix). An S3 event goes to an SQS queue so processing is decoupled from upload.
• AWS Elemental MediaConvert runs transcoding jobs. A new on-demand queue in us-east-1 runs 200 jobs at once (up to 500 with a quota increase), and CreateJob accepts 5 requests a second with a burst of 100.
• Throughput in jobs a second is concurrent jobs divided by job duration. At one minute a job, 200 concurrent jobs finish about 3.3 a second.
• Limits come from AWS documentation (default quotas in us-east-1). Where AWS publishes no figure, such as how many requests a container or database handles, the lab states its assumption on the component.

What you'll build

1. An upload-to-transcode pipeline for 2 videos a second
2. Pre-signed uploads that bypass your servers
3. Bursts absorbed by a queue
4. A worker pool that survives a lost worker

Steps

1. 1Process 2 uploads/s
2. 2Starting the work
3. 3Where should the bytes go?
4. 4Absorbing a burst
5. 5Lose a worker
6. 6Climb to 3 uploads/s
7. 7A job that outlives the message
8. 8When videos get longer