Kafka vs RabbitMQ vs SQS

The one-line verdict

• Need many services to read the same events, and replay history? Kafka: a durable, partitioned log.
• Need flexible routing of tasks to the right workers? RabbitMQ: exchanges route each message to the queues that want it.
• On AWS and want a queue with nothing to run? Amazon SQS: fully managed, and it scales without you thinking about brokers.

All three move messages between services. The difference is what happens to a message after someone reads it, and who runs the machines.

The core difference: a log or a queue

A queue hands each message to a consumer, and once the consumer confirms it's done, the message is gone. RabbitMQ's queues and SQS work this way. That suits jobs: "resize this image", "send this email". Each job should be done once and then forgotten.

A log keeps every event in order, and consumers move through it at their own pace, each remembering its own position. Kafka works this way: its documentation says plainly that events are not deleted after consumption, and a topic keeps them for as long as its retention setting says. That suits facts: "order 1042 was placed". Many services might care about that fact, including services that don't exist yet.

Ask yourself: is this a job to be done, or a fact that happened? That one question settles most decisions.

Side by side

	Kafka	RabbitMQ	Amazon SQS
Model	Partitioned, replicated log	Broker that routes messages to queues	Managed queue
After a message is read	Stays until retention removes it	Removed once acknowledged (Streams keep it)	Removed when the consumer deletes it
Replay old messages	Yes, by offset	Only with RabbitMQ Streams	No
Ordering	Within a partition	Within a queue	FIFO queues: within a message group; Standard: best effort
Delivery	Configurable; duplicates possible	At least once with acknowledgements	Standard: at least once; FIFO: exactly-once processing
Routing	Topic and partition key	Exchanges: direct, fanout, topic, headers	One queue per purpose
Who runs it	You, or a managed service	You, or a managed service	AWS

Kafka: the shared history of your system

Kafka splits each topic into partitions spread across brokers. Events with the same key, such as one customer's orders, go to the same partition, and readers of that partition always see its events in the order they were written. Production clusters usually keep three copies of every partition, so losing a broker loses no data.

Readers join a consumer group. In a classic group, Kafka assigns each partition to exactly one consumer in the group, so the partition count caps how many consumers can work in parallel. Extra consumers sit idle. Different groups each get the whole stream, which is how the email service, the analytics service and the fraud check can all read the same orders independently.

Kafka 4.x adds share groups ("Queues for Kafka", KIP-932): several consumers can share one partition and acknowledge records one by one, queue-style. They were a preview in 4.1 and are described as production-ready in 4.2, so treat them as new and check your platform's support before relying on them.

The cost is operational. Kafka is a distributed system with brokers, partitions, replication and retention to plan, whether you run it yourself or pay a managed provider to.

RabbitMQ: the smart post office

RabbitMQ speaks AMQP 0-9-1. Producers never send to a queue directly: they publish to an exchange, and bindings decide which queues receive the message.

• Direct: delivers by exact routing key, such as invoice.pdf.
• Fanout: copies to every bound queue.
• Topic: matches patterns, so a binding on order.*.india catches order.created.india and order.cancelled.india.
• Headers: routes on message header attributes instead of the key.

A message leaves its queue only after the consumer acknowledges it, so a worker that crashes mid-job doesn't lose the message; it's delivered again. That makes RabbitMQ a strong fit for task distribution with rich routing, priorities and retry patterns.

RabbitMQ also offers Streams: append-only logs that consumers can read repeatedly and attach to at any point. The docs present them as a complement to queues, not a replacement, for large fan-outs and replay.

Amazon SQS: the queue you don't run

SQS has no brokers to size. You create a queue and send messages. Its limits are worth knowing:

• Two queue types. Standard has nearly unlimited throughput, at-least-once delivery and best-effort ordering, so design consumers for duplicates and out-of-order messages. FIFO keeps order within each message group and offers exactly-once processing through deduplication.
• FIFO throughput: 300 API calls per second per action without batching, or up to 3,000 messages per second when each call carries a batch of 10. High-throughput mode raises this further, varying by region.
• Retention: 4 days by default, 14 days at most.
• Message size: up to 1 MiB. Larger payloads go to S3, with the message carrying a reference to them.
• Visibility timeout: a received message is hidden from other consumers for 30 seconds by default, up to 12 hours. If it isn't deleted in time, it becomes visible again and is retried.

A minimal worker in Python:

import boto3

sqs = boto3.client("sqs")
queue_url = "https://sqs.us-east-1.amazonaws.com/123456789012/orders"

while True:
    resp = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10, WaitTimeSeconds=20)
    for msg in resp.get("Messages", []):
        handle_order(msg["Body"])  # must be safe to run twice
        sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=msg["ReceiptHandle"])

WaitTimeSeconds=20 is long polling: the call waits for messages instead of returning empty, which saves requests and money. Deleting only after handling is what makes a crash safe: the message comes back.

A worked example: one online store, three needs

1. "Order placed" events must reach email, inventory, analytics and fraud detection, and next quarter's recommendation service will want the history too. Many readers, plus replay, points to Kafka.
2. Background jobs like "generate the invoice PDF" or "resize product photos", on an AWS stack with a small team. These are jobs that should run once, with no brokers to babysit, which points to SQS.
3. Support tickets routed by region and priority to different worker pools, with retries. Routing rules are the hard part, which points to RabbitMQ with a topic exchange.

Mixing them is normal: Kafka for the event backbone, and a queue for the jobs those events trigger.

Five questions that decide it

1. Job or fact? Jobs fit a queue (SQS, RabbitMQ). Facts that many services read fit a log (Kafka).
2. Do you need to replay history? If yes: Kafka, or RabbitMQ Streams.
3. How complex is routing? Patterns, fan-out and priorities favor RabbitMQ.
4. Who will operate it? No platform team and already on AWS favors SQS.
5. Ordering needs? Kafka orders within a partition, SQS FIFO within a message group, RabbitMQ within a queue. Choose the key that defines "order" for your data, such as the customer ID.

Honest limits

• Every option can deliver a message twice under the right failure. Idempotent consumers are not optional in any of them.
• Kafka's power comes with operational weight. For a handful of background jobs it's usually overkill.
• SQS ties you to AWS and has no replay; if you need history later, you'll add a log anyway.
• Throughput comparisons online are rarely like-for-like. Message size, batching, acknowledgements and replication change the numbers dramatically, so benchmark your own workload.

Remember this

• Queue = jobs, done once and removed. Log = facts, kept and replayable.
• Kafka parallelism is capped by partitions in a classic consumer group.
• RabbitMQ routes through exchanges; SQS gives you no servers to run.
• Design every consumer to be safe to run twice.