Kafka12 min read
Kafka vs Amazon Kinesis: Control vs Less Ops
When to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
- kafka
- aws
- streaming
Topic
#kafka
7 posts ·view all posts
Kafka
Kafka11 min read
Kafka Fundamentals: Topics, Partitions, and Consumer Groups Explained
The mental model every data engineer needs before touching Kafka: how topics, partitions, offsets, and consumer groups actually work together.
- kafka
- streaming
- tutorials
Debezium14 min read
Debezium on Postgres: Replication Slots, Snapshots, and Applying Changes Idempotently
A production walkthrough of Postgres CDC with Debezium: how the replication slot fills your disk, how to snapshot a large table without locking it, and how to apply changes so replays are safe.
- debezium
- kafka
- cdc
Kafka12 min read
Kafka Connect in Production: Converters, Dead Letter Queues, and Worker Sizing
The Connect settings that decide whether your connectors survive contact with real data: converter choice, error tolerance with a DLQ, SMT limits, and how to size and scale workers.
- kafka
- streaming
Kafka13 min read
Kafka Schema Registry: Evolving Event Schemas Without Breaking Consumers
Avro versus Protobuf, what each compatibility mode really allows, and the CI gate that stops a breaking schema change before it reaches a topic your consumers cannot read.
- kafka
- streaming
Kafka12 min read
Kafka Exactly-Once Semantics: What Transactions Actually Guarantee
Idempotent producers, transactions, and read_committed give you exactly-once inside Kafka. The moment you write to a database, you still need an idempotent sink. Here is where the boundary sits.
- kafka
- streaming
Kafka12 min read
Kafka Consumer Group Rebalancing: Why Your Consumers Stall and How to Stop It
Rebalances are the most common cause of Kafka consumer lag spikes. Here is the protocol, the four timeouts that matter, and the cooperative sticky config I ship by default.
- kafka
- streaming