Kafka

Solution

A partition has a single offset per group. Throughput scales with partitions, not with extra consumers parked on the same partition. To add parallelism, add partitions (and accept the rebalance) or split topics by domain.

Solution

Kafka: you own brokers, retention, and ops (or pay for a managed cluster). Kinesis: less ops, shard math, AWS lock-in, 24h–365d retention knobs. Pick from ops budget and existing cloud, not a blog benchmark.

Solution

Most pipelines are at-least-once plus an idempotent sink (MERGE on event_id, or Kafka transactions into a compacted topic). Exactly-once is a property of a specific producer–broker–consumer setup, not a checkbox on the resume. Say what happens on a retry.

Solution

Is production up (messages/sec)? Is the consumer crashed or stuck in a rebalance? Is a hot partition (key skew) starving others? Is the sink slow (warehouse credits, rate limits)? Lag alone is not a root cause. Show consumer-lag by partition, not a single number.

Solution

Compacted topics keep the latest value per key (CDC changelog, config, ktables). Time retention drops old messages regardless of key (raw events). Using compaction on a clickstream deletes history you still need for sessionization.

Solution

A record that always fails deserialization or sink validation. A naive retry loop blocks that partition forever. Dead-letter the record (with the offset), alert, keep consuming. Know the difference between retryable sink errors and bad payloads.

Solution

If processing exceeds the interval, the member is kicked and the group rebalances. Shrink the work per poll, speed the sink, or raise the interval with eyes open.

Solution

A null value for that key on a compacted topic. Downstream must apply deletes. A random new key is not a delete.

Solution

Latency for throughput. A firehose with linger 0 wastes RPCs. Correctness is acks/ISR, not linger.

Solution

A producer adding a required field will kill old consumers. Fail the PR on a BACKWARD/FORWARD break you did not mean.