SproutStackSproutStack home
SeniorAmazon SQS

One bad record keeps retrying a whole batch

Lambda reads SQS in batches of ten. A rate-limited API now rejects single records.

The setup today

A Lambda function reads a standard SQS queue and pushes each record to a partner API. If any record fails, the function throws, so the whole batch is retried.

Current configuration

event source: batch size 10
Lambda timeout: 60 s
queue VisibilityTimeout: 60 s
RedrivePolicy: none   MessageRetentionPeriod: 4 days

What you know

  • The partner API now rate-limits and rejects about 2% of records with a retryable error, and 0.1% permanently.
  • Successful records are being sent to the partner several times.

The new requirement

The partner complains about duplicates. Successful records must not be re-sent, retryable failures must be retried, and permanent failures must be kept for review.

Make the call

2 questions · first-try answers earn the most XP

1. Which configuration meets all three requirements?
2. The DLQ keeps messages for 4 days, the same as the source. A record fails for 3 days, then moves. How long is it kept?
What a strong answer also covers
  • Limit concurrency (maximum concurrency on the event source) so retries do not hammer a partner that is already rate-limiting.
  • Send an idempotency key to the partner, because at-least-once delivery will still repeat the occasional record.
  • Separate retryable from permanent errors in code: fail retryable ones, send permanent ones straight to a review store.
  • Alarm on DLQ depth and on ApproximateAgeOfOldestMessage of the source queue.

The background

More drills