BeginnerAmazon SQS
Long video jobs join a queue built for images
A queue tuned for 5-second image jobs is about to carry 8-minute video jobs.
The setup today
Uploads put a job on a standard SQS queue, and a pool of workers resizes images. Everything runs on the defaults, and jobs take about 5 seconds.
Current configuration
queue: media-jobs type: standard
VisibilityTimeout: 30 # default
ReceiveMessageWaitTimeSeconds: 0
RedrivePolicy: noneWhat you know
- Video transcoding takes 2 to 8 minutes per file.
- A small number of uploaded files are corrupt and always fail.
The new requirement
The product now accepts video uploads, and the plan is to put transcoding jobs on the same queue and workers.
Make the call
2 questions · first-try answers earn the most XP
What a strong answer also covers
- Turn on long polling (WaitTimeSeconds 20) to cut empty receives and cost.
- Give the dead-letter queue a longer retention than the source; standard queues keep the original enqueue time.
- Make transcoding idempotent (write output to a key derived from the job ID), since duplicates can still happen.
- Scale video workers on the age of the oldest message, not just the queue length.