Streaming Architectures cheat sheet

Lambda versus Kappa, delivery semantics, windowing, watermarks, and the sink patterns that keep streams correct.

ArchitectureAdvanced6 sections

Topologies

Kappa: one streaming path, replay for reprocessing: The modern default. One codebase, and history comes from replaying the log rather than a parallel batch pipeline.
Lambda: batch layer plus speed layer: Two implementations of the same logic that must agree forever. Choose it only when a legacy batch system cannot be retired.
Medallion streaming: bronze append, silver dedup, gold aggregate: Keeps raw events immutable so a logic bug is fixable by rebuilding silver instead of re-ingesting from the source.
Micro-batch as a middle ground: Five-minute batches deliver most of the business value of streaming at a fraction of the operational cost. Ask before building real-time.

At-most-once: Fire and forget. Acceptable only for metrics where a gap is invisible; never for financial events.
At-least-once: The realistic default. Duplicates will happen, so every sink needs an idempotent write or a dedup key.
Exactly-once processing: Achievable inside one engine via transactions and checkpointed offsets. End-to-end it still requires an idempotent sink.
Idempotent sink: MERGE on event_id: The pattern that makes at-least-once safe. Cheaper and more debuggable than chasing true exactly-once delivery.

Event time versus processing time: Always aggregate on event time. Processing time produces numbers that change when your consumer lags, which is indefensible.
Tumbling window: fixed, non-overlapping: Hourly counts. The simplest window and the right default for reporting aggregates.
Sliding window: fixed size, overlapping: Five-minute average computed every minute. Each event lands in multiple windows, so state grows accordingly.
Session window: gap-based: Groups activity until an inactivity gap. State is unbounded per key until the gap closes, so cap it.
withWatermark("event_ts", "10 minutes"): The lateness contract. Events later than the watermark are dropped, which is what bounds state size.

Keyed state partitioned by the aggregation key: Parallelism is capped by key cardinality and partition count. A skewed key becomes a single hot task.
RocksDB state backend with incremental checkpoints: Required once state exceeds memory. Full checkpoints of large state will not finish inside a checkpoint interval.
Checkpoint interval versus recovery time: Shorter checkpoints mean faster recovery and more overhead. Thirty seconds to a few minutes suits most pipelines.
Consumer lag as the primary SLO: Alert on lag, not on CPU. Lag is the only metric that maps directly to the freshness promise you made.

Schema Registry with backward compatibility: Producers can add optional fields; consumers keep working. The default that prevents most breaking deploys.
Envelope with event_id, event_ts, source, version: A consistent envelope makes dedup, ordering, and debugging uniform across every topic.
Dead letter queue for unparseable events: Without a DLQ, one bad message halts the consumer or gets silently skipped. Both outcomes cause incidents.
Never reuse a field name with a new meaning: Add a new field and deprecate the old one. Redefining a field breaks every historical replay.

Snowpipe Streaming / Storage Write API: Row-level ingestion into a warehouse with seconds of latency and no file staging.
Delta or Iceberg with a compaction job: Streaming writes create small files. Schedule compaction as part of the pipeline, not as an afterthought.
Kafka Connect sink with a DLQ topic: Less code than a custom consumer, and the DLQ plus retry behavior is already built and tested.
Materialize aggregates, keep raw events: Serve dashboards from pre-aggregated tables but retain raw events so definitions can change retroactively.

From DataLane — tutorials at/blog, practice SQL live in theplayground.