Amazon SQS in Depth

What SQS is, in one minute

Amazon SQS is a fully managed queue: no brokers to run, patch or scale. Producers send messages, consumers poll for them, process them and delete them. You pay per request, and a standard queue scales to very high throughput without any configuration.

The model is simple, but three things shape every design: messages are delivered at least once, a received message is only hidden (not removed) until you delete it, and there is no replay once a message is deleted.

Standard or FIFO?

	Standard	FIFO
Ordering	best effort	strict, within each message group
Delivery	at least once (duplicates possible)	exactly-once processing, with deduplication over a 5-minute window
Throughput	nearly unlimited	300 API calls per second per action, or 3,000 messages with batches of 10; high-throughput mode goes much higher
Name	anything	must end in .fifo

Standard is the default choice. FIFO is for when order matters per entity (one customer's ledger, one order's status changes) or duplicates are very costly.

A message's life

The settings that matter

Setting	Default	Range or limit
Visibility timeout	30 seconds	up to 12 hours
Message retention	4 days	1 minute to 14 days
Receive wait time (long polling)	0 (short polling)	up to 20 seconds
Delivery delay	0	up to 15 minutes
Maximum message size	1 MiB	larger payloads go to S3, with a pointer in the message
Batch size	1	up to 10 messages per send, receive or delete
In-flight messages (standard)		about 120,000 received but not deleted

The visibility timeout

When a consumer receives a message, SQS hides it from everyone else for the visibility timeout. The consumer must delete it before the timeout ends. If it does not (it crashed, or it is just slow), the message becomes visible again and another consumer gets it.

So the timeout is a bet on how long processing takes:

• Too short: slow messages are processed twice, sometimes by several workers at once.
• Too long: when a worker crashes, its message sits hidden until the timeout ends, delaying the retry.

For work of varying length, set a reasonable timeout and extend it while working with ChangeMessageVisibility (a "heartbeat"). Either way, duplicates can happen, so consumers must be idempotent.

Short and long polling

By default, ReceiveMessage uses short polling: it checks a subset of SQS's servers and answers at once, so it can return nothing even when messages exist. Long polling (WaitTimeSeconds up to 20) checks all servers and waits until a message arrives or the time runs out. It means fewer empty responses, fewer requests and a lower bill. There is rarely a reason not to use it.

FIFO: message groups and deduplication

• MessageGroupId defines the unit of ordering. Messages in one group are delivered strictly in order, and SQS will not hand out a group's next message while an earlier one is in flight. Different groups are processed in parallel. So parallelism equals the number of active groups: key by customer or order, never by one constant.
• Deduplication: a message with the same MessageDeduplicationId (or the same body, with content-based deduplication) sent within 5 minutes of the first is accepted but not delivered again. That covers producer retries, not consumer duplicates hours later.
• Head-of-line blocking: a message that keeps failing holds up its whole group until it is deleted or moved to a dead-letter queue.

Dead-letter queues

A redrive policy on the source queue names a DLQ and a maxReceiveCount. Once a message has been received that many times without being deleted, SQS moves it to the DLQ instead of offering it again. Without a DLQ, a poison message is retried until retention expires, wasting work all along.

Two details:

• A FIFO queue's DLQ must also be FIFO, and moving a message out of its group breaks that group's order, which you should accept knowingly.
• For standard queues, a message keeps its original enqueue time in the DLQ, so retention there counts from when it was first sent. Give the DLQ a longer retention than the source. (FIFO queues reset the timestamp on move.)

When the bug is fixed, DLQ redrive sends messages back to the source queue for another try.

Patterns you will use

• Fan-out: SQS has no routing or topics. Publish to SNS (or EventBridge) and subscribe one SQS queue per consumer, so each service gets its own copy and its own pace.
• Lambda consumers: Lambda polls the queue for you and calls your function with batches. Set the queue's visibility timeout to at least six times the function timeout, and enable partial batch responses (ReportBatchItemFailures), so one bad message does not send the whole batch back.
• Large payloads: store the body in S3 and send a reference; extended client libraries do this for you.
• Delayed work: per-message delays up to 15 minutes; longer schedules belong in a scheduler.

Going deeper: cost and throughput tuning

You pay per request, and a request can carry up to 10 messages. Batching sends, receives and deletes, plus long polling, can cut request counts by close to ten times. For FIFO, throughput limits are per API action, so batching also multiplies throughput: 300 calls a second becomes 3,000 messages. High-throughput FIFO mode raises the limits further (the exact figure varies by region) and works best when traffic is spread across many message groups.

Watch ApproximateNumberOfMessagesVisible (backlog) and ApproximateAgeOfOldestMessage (how long the worst one has waited). Scaling consumers on age rather than count catches "few messages, all stuck" as well as "many messages."

How it fails

• Duplicate processing: visibility timeout shorter than processing time.
• Silent expiry: messages older than the retention period are deleted, processed or not.
• Poison messages: retried until retention ends when there is no DLQ.
• Blocked FIFO groups: one failing message holds its group; one hot group caps throughput.
• Too many in flight: past about 120,000 received-but-undeleted messages on a standard queue, receives fail with OverLimit until some are deleted.

SQS or something else?

Choose SQS for decoupled work queues on AWS with no infrastructure to run. Choose Kafka (or Kinesis) when consumers must replay history or many consumers need the same stream. Choose RabbitMQ for rich routing and protocol flexibility outside a single cloud. SQS plus SNS covers a large share of everyday event-driven designs on AWS.

What to say in a design review

• "A standard queue with long polling, a visibility timeout of twice our p99 processing time, and a heartbeat for slow jobs."
• "A DLQ after 5 receives, with longer retention than the source and an alarm on its depth."
• "Order updates use FIFO with the order ID as the message group, so each order is in order and different orders run in parallel."
• "Consumers are idempotent, because even FIFO's deduplication only covers 5 minutes on the send side."

Remember this

• At least once by default; a received message is only hidden until you delete it.
• The visibility timeout (default 30 seconds) must exceed processing time, or work runs twice.
• Use long polling (WaitTimeSeconds up to 20).
• FIFO orders per message group; parallelism equals the number of active groups.
• Always set a DLQ, with longer retention than the source.