Kafka Interview Questions cheat sheet
Partitions, consumer groups, delivery semantics, ISR, and the tuning details senior streaming interviews dig into.
Core architecture
What is a topic, a partition, and an offset?- A topic is a named log; a partition is one ordered, append-only shard of that log stored as segment files on a broker; an offset is a monotonically increasing position within a partition. Ordering is guaranteed only within a partition, never across a topic — which is why the partition key is the most consequential design decision in a Kafka system.
How does a producer choose a partition?- With a key, the default partitioner hashes it (murmur2) modulo the partition count, so the same key always lands on the same partition. Without a key, modern clients use sticky partitioning, filling one batch before rotating, which improves batching over the old round-robin. Adding partitions later rehashes keys, breaking per-key ordering across the change.
What replaced ZooKeeper, and why did it matter?- KRaft mode, where a quorum of controller nodes stores metadata in an internal Kafka log using Raft. ZooKeeper was removed entirely in Kafka 4.0 in 2025, so new clusters are KRaft only. Benefits are one system to operate, far faster controller failover and metadata propagation, and support for millions of partitions rather than tens of thousands.
Explain the ISR and how it interacts with acks.- The in-sync replica set is the leader plus followers that have caught up within replica.lag.time.max.ms (default 30 seconds). acks=all means the leader waits for all in-sync replicas, so durability depends on min.insync.replicas — with replication factor 3 and min.insync.replicas 2 you can lose one broker and still write, and a write fails rather than silently losing data if two are down.
What is the difference between the log end offset, the high watermark, and the last stable offset?- The log end offset is the next offset the leader will write; the high watermark is the highest offset replicated to all in-sync replicas, and consumers can only read up to it; the last stable offset applies with transactions and marks where no open transaction remains, which is the read limit under read_committed isolation.
How does Kafka achieve such high throughput?- Sequential disk writes to an append-only log, the operating system page cache instead of an application cache, zero-copy transfer with sendfile for consumers reading recent data, and batching plus compression amortizing per-message overhead. The consequence for tuning is that spare RAM for page cache matters more than a large JVM heap on a broker.
Producers and delivery semantics
What do acks=0, acks=1, and acks=all mean in practice?- acks=0 is fire and forget with possible silent loss; acks=1 waits for the leader only, so an unreplicated write is lost if that leader fails immediately; acks=all waits for the in-sync set and is the only durable setting. Since Kafka 3.0 the client defaults are acks=all with enable.idempotence=true, which is the right production baseline.
How does the idempotent producer prevent duplicates?- Each producer gets a producer ID and attaches a monotonic sequence number per partition; the broker deduplicates retries of the same sequence. It removes duplicates caused by producer retries within a session, but not duplicates from application-level reprocessing or from a producer restart with a new ID, which is what transactions address.
What do Kafka transactions actually guarantee?- Atomic writes across multiple partitions plus the consumer offset commit, so a consume-transform-produce loop is exactly-once within Kafka when consumers read with isolation.level=read_committed. They do not extend to an external database — writing to Postgres and Kafka atomically needs the outbox pattern or two-phase commit, and interviewers ask this deliberately.
Which producer settings control latency versus throughput?- linger.ms (default 0) makes the producer wait to fill a batch, batch.size (default 16 KB) caps it, and compression.type (lz4, snappy, or zstd) shrinks what goes over the wire. Setting linger.ms to 5-50 ms typically multiplies throughput for a barely perceptible latency cost, because larger batches compress better.
Can messages be reordered, and how do you prevent it?- Yes, if max.in.flight.requests.per.connection is above 1 and a retry succeeds after a later batch. With enable.idempotence=true the broker enforces sequence ordering and you can safely keep up to 5 in flight. Without idempotence, strict ordering requires max.in.flight=1, which severely limits throughput.
Consumers and consumer groups
How do consumer groups distribute work?- Each partition is assigned to exactly one consumer within a group, so parallelism is capped by partition count and extra consumers sit idle. Multiple groups each get a full independent copy of the stream. The follow-up is scaling: you cannot exceed partition count, and adding partitions is one-way and breaks key ordering.
What triggers a rebalance and how do you reduce the pain?- A member joining or leaving, a session timeout, or a topic metadata change. The old eager protocol stopped all consumption; cooperative sticky assignment, the default since 3.0, moves only the partitions that must move. Also set group.instance.id for static membership so a rolling restart within session.timeout.ms causes no rebalance at all.
Explain the difference between session.timeout.ms and max.poll.interval.ms.- session.timeout.ms (default 45 seconds) governs the heartbeat thread and detects a dead process. max.poll.interval.ms (default 5 minutes) bounds how long your code may take between poll calls before the consumer is considered stuck and removed. Slow message processing trips the second, and the fix is smaller max.poll.records or offloading work, not a longer timeout.
How does offset committing affect delivery semantics?- Committing before processing gives at-most-once; committing after gives at-least-once, which is the normal choice, so consumers must be idempotent. enable.auto.commit is true by default with a 5 second interval, and it commits offsets for records merely returned by poll, so a crash mid-batch loses them — disable it and commit explicitly when correctness matters.
What is consumer lag and what does rising lag tell you?- Lag is the difference between the partition's latest offset and the consumer's committed offset, measured per partition. Rising lag on all partitions means insufficient consumer throughput; rising lag on one partition means a hot key or a stuck consumer. It is the single most valuable alerting metric because it predicts data loss once retention expires.
How do you reprocess a topic from the beginning?- Reset offsets with kafka-consumer-groups using --to-earliest, --to-datetime, or --shift-by, with the group stopped. Alternatively start a new group ID with auto.offset.reset=earliest. Be explicit about downstream side effects: reprocessing re-emits every record, so sinks must be idempotent or you need a separate output target.
Retention, storage, and compaction
What is the difference between delete and compact cleanup policies?- delete removes whole segments once they exceed retention.ms (default 7 days) or retention.bytes. compact keeps at least the latest value for every key indefinitely, turning the topic into a durable changelog you can replay into a full state snapshot. A common setting is 'compact,delete' for a changelog you still want bounded in size.
How do tombstones work in a compacted topic?- A record with a non-null key and null value marks the key deleted; compaction keeps it for delete.retention.ms (default 24 hours) so consumers have time to observe the deletion, then removes it. Consumers must handle a null value explicitly — treating it as a parse error is a common bug in change data capture pipelines.
Why does compaction never remove the active segment?- Compaction operates on closed segments only, so the most recent writes always remain uncompacted and duplicate keys are visible at the tail. Consumers therefore cannot assume one record per key; they must apply last-write-wins themselves. Tuning segment.ms or min.cleanable.dirty.ratio (default 0.5) controls how aggressively cleaning happens.
What is tiered storage and when does it help?- Tiered storage, generally available since Kafka 3.9, offloads older log segments to object storage while brokers keep recent data locally. It decouples retention from broker disk, so you can hold months of history cheaply and rebalance brokers quickly because less local data must move. Historical reads are slower and metered by the object store.
How do you size partitions for a topic?- Start from target throughput divided by the per-partition throughput a single consumer can sustain, add headroom for growth, and remember partitions are the unit of parallelism and cannot be reduced. Too many partitions raises controller metadata load, end-to-end latency from more requests, and rebalance time — a few thousand per broker is a practical ceiling.
Ecosystem and integration
What is Kafka Connect and when do you use it over custom code?- A framework of reusable source and sink connectors with distributed workers handling offset tracking, retries, scaling, and dead letter queues. Use it for standard integrations such as Debezium change data capture or an S3 sink, because reimplementing exactly-once offset handling per pipeline is wasted effort. Write custom consumers when the transformation logic is the point.
What does the Schema Registry give you?- A versioned store of Avro, Protobuf, or JSON Schema definitions; producers register a schema and embed only its ID, so payloads shrink and consumers can resolve them. Its real value is compatibility enforcement — BACKWARD, the default, lets new consumers read old data, so a producer cannot deploy a breaking field removal.
Compare Kafka Streams and Flink.- Kafka Streams is a library embedded in your application with no separate cluster, state in RocksDB backed by changelog topics, and Kafka as its only source and sink. Flink is a distributed engine with richer windowing and event-time semantics, its own checkpointing, and many connectors. Choose Streams for Kafka-to-Kafka microservices, Flink for complex multi-source stateful processing.
How do you handle a poison message that keeps failing?- Never retry forever in place — that blocks the partition and lag grows without bound. Catch the failure, publish the record with its error context to a dead letter topic, commit the offset, and continue. Then alert on dead letter volume and provide a replay path, since a silently filling dead letter queue is data loss with extra steps.
How do you monitor a Kafka cluster?- Consumer lag per group and partition, under-replicated and offline partition counts, ISR shrink and expand rate, request handler idle ratio, and produce and fetch latency percentiles. Under-replicated partitions above zero for a sustained period is the clearest sign of broker trouble, and offline partitions means active unavailability.
From DataLane — tutorials at/blog, practice SQL live in theplayground.