Kafka in Depth

What this page adds

The Kafka Intro covers topics, partitions, offsets and consumer groups. This page is about what happens underneath: how a message survives a broker dying, how consumers keep their place, what a rebalance is, and which defaults catch teams out. It is written for the moment Kafka is already in your design and someone asks "what happens if…?".

How a write is kept safe

Each partition has one leader broker and several followers that copy it. Followers that are fully caught up form the in-sync replicas (ISR). A follower that falls more than replica.lag.time.max.ms behind (30 seconds by default) drops out of the ISR until it catches up.

If the leader dies, a follower from the ISR becomes leader. Because it had every acknowledged message, nothing confirmed is lost.

The durability settings

Setting	Default	What it decides
replication.factor (per topic)	broker's default.replication.factor, which is 1	how many copies of each partition exist
min.insync.replicas	1	how many in-sync copies a write needs when acks=all
acks (producer)	all	when the producer counts a write as done
enable.idempotence (producer)	true	whether retries can create duplicates
unclean.leader.election.enable	false	whether an out-of-date replica may become leader

The standard recipe for data you cannot lose is replication factor 3, min.insync.replicas=2, acks=all. Any one broker can fail with no data loss and no interruption. If two fail, writes to the affected partitions are rejected rather than accepted on a single copy: Kafka chooses safety over availability.

Note the defaults: a replication factor of 1 means one copy. Fine on a laptop, dangerous in production, so set it explicitly on every topic (or change the broker default).

Producers: batching, retries and ordering

• Producers group messages into batches per partition. linger.ms (5 ms by default since Kafka 4.0) waits briefly to fill a batch; batch.size (16 KB) caps it. Bigger batches mean higher throughput at a small latency cost.
• Failed sends are retried until delivery.timeout.ms (2 minutes) runs out. With idempotence on (the default), the broker discards duplicate retries, and ordering within a partition holds even with up to 5 requests in flight.
• Messages are capped at about 1 MB by default. Large payloads belong in object storage, with a pointer in the message.

Consumers: offsets and the defaults that surprise

A consumer's progress is its committed offset per partition, stored in Kafka itself. Three defaults cause most of the surprises:

• enable.auto.commit=true: offsets are committed on a timer (every 5 seconds by default) when your code calls poll(). That is safe only if each batch is fully processed before the next poll. Hand the work to another thread or queue and a crash can skip messages that were committed but never processed. Many teams turn auto-commit off and commit explicitly after processing, accepting the occasional duplicate (so handlers must be idempotent).
• auto.offset.reset=latest: a brand-new group starts at the end of the topic and ignores history. Use earliest when a new consumer must process what is already there.
• max.poll.interval.ms=300000 (5 minutes): if your code does not call poll() again within this time, the consumer is assumed stuck and removed from the group. Slow batches cause duplicate processing and a loop of rebalances. max.poll.records (500) controls how much work each poll hands you.

Rebalances

When consumers join, leave or are presumed dead, the group rebalances: partitions are reassigned. Under the classic protocol this can pause the whole group briefly, and frequent rebalances (from deploys, autoscaling or slow consumers) show up as lag spikes and duplicates.

Two improvements: the cooperative assignor moves only the partitions that need to move, and the new consumer group protocol (KIP-848, generally available in Kafka 4.0) moves coordination to the broker and makes rebalances incremental. Clients opt in with group.protocol=consumer; the default is still classic.

Retention, compaction and partition count

• Retention: messages are kept for 7 days by default (retention.ms), regardless of whether anyone read them. Consumers that fall further behind than retention lose data.
• Compaction (cleanup.policy=compact) keeps only the latest message for each key: perfect for "current state of every account" topics. A message with a null value (a tombstone) deletes the key.
• Partition count caps consumer parallelism and can only go up. Adding partitions changes which partition a key maps to, so per-key ordering breaks across the change. Choose the count for expected peak consumers, with headroom, at creation time.

Exactly-once, honestly

Kafka transactions let a consumer read, process and write results to Kafka atomically, together with its offset commit. Readers set isolation.level=read_committed (the default is read_uncommitted) to skip aborted writes. This gives exactly-once within Kafka. Anything outside it (a database write, an email, a payment call) can still happen twice and needs its own idempotency.

Going deeper: unclean leader election and what "acknowledged" means

If every in-sync replica is gone, the partition is offline. unclean.leader.election.enable=true would let an out-of-date replica take over: the partition comes back, but messages it never received are gone for good, even though producers were told they succeeded. The default (false) keeps the partition down instead. Turning it on is a conscious choice of availability over durability, sometimes made for metrics or logs, almost never for orders or payments.

Also note acks=1: the leader confirms as soon as it has written the message, before any follower copies it. If the leader dies in that window, the "acknowledged" message is lost.

Going deeper: Kafka 4.x changes worth knowing

• ZooKeeper is gone. Kafka 4.0 runs only in KRaft mode, where the brokers manage metadata themselves with a Raft-based quorum. Older clusters must migrate to KRaft before upgrading.
• Queues for Kafka (KIP-932): share groups, production-ready in Kafka 4.2, let many consumers work on the same partition with per-message acknowledgement and delivery counts. That covers job-queue use cases without needing one partition per worker. The trade-off is that per-partition ordering no longer applies.
• No built-in dead-letter topic for classic consumer groups. A message your code cannot process blocks its partition until you skip it, so teams write failures to a separate "dead-letter" topic themselves.

How it fails

• Consumer lag grows: consumers are slower than producers. Watch lag per partition; add consumers (up to the partition count) or speed up processing.
• Rebalance storms: slow batches or flapping pods keep reshuffling partitions. Check max.poll.interval.ms and processing time.
• A poison message: one record that always throws stops progress on its partition. Catch, record and skip to a dead-letter topic.
• Under-replicated partitions: followers falling out of the ISR. One more broker failure and writes stop (with min.insync.replicas=2), so alert on it.
• Hot partitions: one key with most of the traffic overloads one partition, however many there are.

What to say in a design review

• "Order events use replication factor 3, min.insync.replicas=2 and acks=all, so losing one broker loses nothing."
• "Messages are keyed by order ID, so each order's events stay in order. We picked 24 partitions for peak consumer count, since we cannot reduce them later."
• "Consumers commit after processing and handlers are idempotent, because delivery is at least once."
• "Exactly-once covers our Kafka-to-Kafka steps; the payment call uses an idempotency key."

Remember this

• RF 3 + min.insync.replicas=2 + acks=all survives one broker with no loss; two down means rejected writes, not lost ones.
• The default replication factor is 1. Set it explicitly.
• New groups start at the latest offset by default; slow processing past 5 minutes triggers rebalances and duplicates.
• Partitions only go up, and adding them changes key placement.
• Exactly-once stops at Kafka's edge. Side effects need idempotency.