SproutStackSproutStack home
BeginnerRabbitMQ

Emails must not vanish when a worker crashes

One email worker with automatic acks is about to become four.

The setup today

A single worker sends transactional emails (receipts, password resets) from a durable RabbitMQ queue. It was written quickly: messages are acknowledged automatically on delivery and there is no prefetch limit.

Current configuration

queue: emails   type: classic   durable: true
consumer: autoAck=true
basic.qos prefetch: 0   # no limit

What you know

  • When the worker was restarted during a deploy, about 300 password-reset emails were never sent.
  • Sending one email takes 100 to 300 ms.

The new requirement

Emails must not be lost when a worker crashes or is redeployed, and the team is adding three more workers to keep up with growth.

Make the call

2 questions · first-try answers earn the most XP

1. What do you change?
2. The email provider rejects a message permanently (invalid address). What should the worker do?
What a strong answer also covers
  • Manual acks mean at-least-once delivery: a crash after sending but before the ack sends the email twice. Use an idempotency key with the provider where possible.
  • Turn on publisher confirms in the services that publish, so a lost message on the way in is noticed too.
  • Consider a quorum queue so the emails survive losing a broker node, not just a worker.
  • Alert on dead-letter queue depth and on the age of the oldest email waiting.

The background

More drills