Topic
14 posts ·view all posts
Apache Flink🟡IntermediateA production comparison of Flink and Spark Structured Streaming: how each one holds state, what their latency floors really are, and why the honest deciding factor is usually your existing batch stack.
Kafka🟡IntermediateThe architectural differences that actually change operations — segment storage versus partition-as-log, tiered offload, and tenant isolation — plus an honest account of when leaving a healthy Kafka cluster is a mistake.
Apache Flink🟢BeginnerProcessing time lies after a restart. Watermarks drop late events on purpose. Checkpoints recover; savepoints migrate. Exactly-once to Kafka does not upsert Postgres.
Apache Pulsar🟢BeginnerBrokers are stateless. BookKeeper holds the log. Pulsar wins on tenant isolation and geo. A healthy Kafka cluster is not a fashion problem.
Kafka🟡IntermediateWhen to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
Kafka🟡IntermediateThe mental model every data engineer needs before touching Kafka: how topics, partitions, offsets, and consumer groups actually work together.
Snowflake🟡IntermediateA practical comparison of Snowpipe file ingestion and Snowpipe Streaming, covering real latency ranges, the credit math per million rows, and how to pick without over-engineering.
Debezium🟡IntermediateA production walkthrough of Postgres CDC with Debezium: how the replication slot fills your disk, how to snapshot a large table without locking it, and how to apply changes so replays are safe.
Kafka🟡IntermediateThe Connect settings that decide whether your connectors survive contact with real data: converter choice, error tolerance with a DLQ, SMT limits, and how to size and scale workers.
Apache Spark🟡IntermediateThe four things that decide whether a Spark stream survives a year in production: trigger choice, checkpoint discipline, watermark math, and what exactly-once really guarantees.
Kafka🟡IntermediateAvro versus Protobuf, what each compatibility mode really allows, and the CI gate that stops a breaking schema change before it reaches a topic your consumers cannot read.
Kafka🟡IntermediateIdempotent producers, transactions, and read_committed give you exactly-once inside Kafka. The moment you write to a database, you still need an idempotent sink. Here is where the boundary sits.
Kafka🟡IntermediateRebalances are the most common cause of Kafka consumer lag spikes. Here is the protocol, the four timeouts that matter, and the cooperative sticky config I ship by default.
AWS🟡IntermediateWhen Kinesis Data Streams earns its shards, when Firehose is enough, and the buffering and partitioning patterns that keep S3 landings queryable.
