All stacks · Streaming & CDC
Event streaming, topics, Kafka Connect, and real-time pipelines.
Kafka🟡IntermediateThe architectural differences that actually change operations — segment storage versus partition-as-log, tiered offload, and tenant isolation — plus an honest account of when leaving a healthy Kafka cluster is a mistake.
Kafka🟡IntermediateWhen to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
Kafka🟡IntermediateThe mental model every data engineer needs before touching Kafka: how topics, partitions, offsets, and consumer groups actually work together.
Kafka🟡IntermediateThe Connect settings that decide whether your connectors survive contact with real data: converter choice, error tolerance with a DLQ, SMT limits, and how to size and scale workers.
Kafka🟡IntermediateAvro versus Protobuf, what each compatibility mode really allows, and the CI gate that stops a breaking schema change before it reaches a topic your consumers cannot read.
Kafka🟡IntermediateIdempotent producers, transactions, and read_committed give you exactly-once inside Kafka. The moment you write to a database, you still need an idempotent sink. Here is where the boundary sits.
Kafka🟡IntermediateRebalances are the most common cause of Kafka consumer lag spikes. Here is the protocol, the four timeouts that matter, and the cooperative sticky config I ship by default.
Docker🟢BeginnerA Friday CI green on python:3.11-slim:latest became a Saturday outage when prod pulled a new digest. Pin the bytes, promote the same image, and stop treating Compose as Kafka.
Apache NiFi🟢BeginnerFlowFiles pile up when a downstream processor stalls. Provenance has its own disk. Set queue limits, watch both repositories, and do not use NiFi as a Kafka bus.
Apache Flink🟢BeginnerProcessing time lies after a restart. Watermarks drop late events on purpose. Checkpoints recover; savepoints migrate. Exactly-once to Kafka does not upsert Postgres.
Apache Pulsar🟢BeginnerBrokers are stateless. BookKeeper holds the log. Pulsar wins on tenant isolation and geo. A healthy Kafka cluster is not a fashion problem.
Debezium🟡IntermediateA production walkthrough of Postgres CDC with Debezium: how the replication slot fills your disk, how to snapshot a large table without locking it, and how to apply changes so replays are safe.
Kafka is the backbone of event-driven data platforms: a distributed, replayable log that decouples producers from consumers and makes real-time pipelines possible. It is also a system where defaults are rarely what production needs — acks, retention, partitioning, and consumer group behavior all reward understanding.
The guides here cover Kafka from a data engineer's seat: designing topics and keys, getting exactly-once guarantees where they exist (and not pretending they exist where they do not), running Kafka Connect for ingestion, and landing streams into warehouses and lakehouses without duplicates.
When multiple consumers need the same events, when you need replay, or when sub-minute latency matters. If one system just needs rows moved to a warehouse hourly, batch ingestion is simpler and cheaper. Kafka pays off as an architecture, not as a fancy queue.
Enough for your target consumer parallelism with headroom — partitions are the unit of parallelism, and you cannot reduce them later without pain. For most application topics, 6–12 is a sane start; very high-throughput topics go higher deliberately, not by default.
Only within a partition. Messages with the same key land in the same partition and stay ordered; there is no global order across partitions. Design keys around the entity whose order matters — user_id, order_id — and this is enough.
Within the Kafka ecosystem, yes: idempotent producers plus transactions give exactly-once from producer through Kafka Streams. End-to-end into external systems, it becomes exactly-once processing with at-least-once delivery — your sink still needs idempotent writes or dedup.
Kinesis if you are all-in on AWS and want zero cluster operations at modest scale. Kafka for ecosystem breadth (Connect, Streams, exactly-once), multi-cloud portability, and high-throughput economics. Managed Kafka (MSK, Confluent) covers the middle ground.
