BeginnerKafka
A new service needs last week's orders
An analytics service joins an orders topic and finds it only sees new messages.
The setup today
Orders are published to a Kafka topic that the billing service already consumes. Topic settings were left at their defaults apart from the partition count.
Current configuration
topic: orders partitions: 12 retention.ms: 604800000 # 7 days
billing group.id=billing
analytics group.id=analytics # new, all other settings defaultWhat you know
- The topic holds about 4 million orders from the last seven days.
- Analytics started yesterday and has only processed orders created since then.
The new requirement
The analytics team needs every order from the past week, and from now on wants 30 days of history available in Kafka for future backfills.
Make the call
2 questions · first-try answers earn the most XP
What a strong answer also covers
- Estimate disk before raising retention: messages per day × average size × 30 × replication factor.
- Tiered storage, where available, keeps old segments in object storage instead of on broker disks.
- A new group reading from the beginning puts extra read load on brokers; schedule large backfills off-peak.
- Make the analytics consumer idempotent, since a reset or a crash will reprocess some orders.