Kafka

Solution

A partition has a single offset per group. Throughput scales with partitions, not with extra consumers parked on the same partition. To add parallelism, add partitions (and accept the rebalance) or split topics by domain.

Solution

Kafka: you own brokers, retention, and ops (or pay for a managed cluster). Kinesis: less ops, shard math, AWS lock-in, 24h–365d retention knobs. Pick from ops budget and existing cloud, not a blog benchmark.

Solution

Most pipelines are at-least-once plus an idempotent sink (MERGE on event_id, or Kafka transactions into a compacted topic). Exactly-once is a property of a specific producer–broker–consumer setup, not a checkbox on the resume. Say what happens on a retry.

Solution

Is production up (messages/sec)? Is the consumer crashed or stuck in a rebalance? Is a hot partition (key skew) starving others? Is the sink slow (warehouse credits, rate limits)? Lag alone is not a root cause. Show consumer-lag by partition, not a single number.

Solution

Compacted topics keep the latest value per key (CDC changelog, config, ktables). Time retention drops old messages regardless of key (raw events). Using compaction on a clickstream deletes history you still need for sessionization.

Solution

A record that always fails deserialization or sink validation. A naive retry loop blocks that partition forever. Dead-letter the record (with the offset), alert, keep consuming. Know the difference between retryable sink errors and bad payloads.

Solution

If processing exceeds the interval, the member is kicked and the group rebalances. Shrink the work per poll, speed the sink, or raise the interval with eyes open.

Solution

A null value for that key on a compacted topic. Downstream must apply deletes. A random new key is not a delete.

Solution

Latency for throughput. A firehose with linger 0 wastes RPCs. Correctness is acks/ISR, not linger.

Solution

A producer adding a required field will kill old consumers. Fail the PR on a BACKWARD/FORWARD break you did not mean.

Solution

Retries after a timeout when the broker actually committed. Idempotent producer (enable.idempotence) plus a key. acks=all is durability, not exactly-once at the consumer.

Solution

Compaction is not instantaneous; duplicates exist until the cleaner runs. Compacted topics are a cache of latest, not a SQL unique constraint. Downstream still upserts.

Solution

Two consumer groups (or Connect sink connectors). One consumer dual-writing is a partial-failure nightmare. Each sink owns its offsets and DLQ.

Solution

Hash a finer key (user_id + shard) or isolate hot keys to a dedicated topic. Adding consumers cannot split a partition. Measure key cardinality first.

Solution

BACKWARD: consumers with the new schema can read old data — typically add optional fields. FULL needs both directions. A required field without a default breaks consumers. CI should check compatibility, not Slack.

Solution

When the transform is light (rename, flatten) and you want offset management for free. Flink when you need event-time windows, exactly-once into Iceberg, or serious state. Do not write a Connect SMT that is actually a stream join.