SeniorKafka
A slow fraud check joins the consumer
Each record now takes up to 800 ms, and every flagged order must be written exactly once.
The setup today
An order-processing consumer group reads a 12-partition topic with six consumers. Each record is enriched and written to Postgres. Processing is fast today, so the consumer settings were left at their defaults.
Current configuration
group.id=order-processing consumers: 6 partitions: 12
max.poll.records=500 # default
max.poll.interval.ms=300000 # default, 5 minutes
enable.auto.commit=true # defaultWhat you know
- Records arrive at about 300 a second at peak.
- The new fraud check calls an external API that takes 200 to 800 ms per record.
The new requirement
Compliance requires a fraud check on every order before it is processed, and every flagged order must be recorded in Postgres exactly once, even across crashes and rebalances.
Make the call
2 questions · first-try answers earn the most XP
What a strong answer also covers
- Throughput math: 300 records/s × 0.5 s average needs about 150 concurrent checks, far more than 12 consumers processing one record at a time. Process records concurrently within a partition's batch, or add partitions.
- Store the offset in Postgres in the same transaction as the result if you want exactly-once processing of the Kafka records themselves.
- Put a timeout and a circuit breaker around the fraud API, with a decision for what to do when it is down.
- The new consumer group protocol (KIP-848) and cooperative assignment reduce the cost of each rebalance.