Flink vs Spark Structured Streaming: State, Latency, and the Team You Actually Have

Most Flink-versus-Spark posts compare latency numbers and stop there. That is the least useful part of the decision. I have run both in production, and the thing that decided each project was never the benchmark — it was how much state the job had to hold per key, and whether the team already operated Spark for batch.

The short version: Spark Structured Streaming is a very good incremental batch engine wearing a streaming API. Flink is an actual stream processor. Both statements are compliments. They just point at different jobs.

flowchart LR
  src[Kafka topic] --> spark["Spark: micro-batch<br/>trigger every N seconds"]
  src --> flink["Flink: event at a time<br/>keyed state + timers"]
  spark --> sink1[Sink writes per batch]
  flink --> sink2[Sink writes continuously]

The execution model is the difference. Everything operational follows from it.

The execution model, and what it costs you

Spark Structured Streaming collects records into a micro-batch, runs a job over that batch, commits offsets, and repeats. Even with Trigger.ProcessingTime("1 second") you are running a tiny Spark job every second, with the scheduling and task setup that implies. The latency floor is your trigger interval plus job overhead. Continuous processing mode exists and I have never seen a team run it in production with a straight face.

Flink runs operators as long-lived tasks. A record arrives, flows through the operator chain, and updates state. There is no batch boundary to wait for. That is why Flink can hold sub-second end-to-end latency and why its state backend is a first-class piece of the system rather than a checkpoint file.

The practical consequence is not “Flink is faster”. It is that in Spark, every stateful operation has to be expressible in the micro-batch model. Deduplication, joins, and aggregations are supported. Arbitrary per-key logic is flatMapGroupsWithState, and that API is where Spark streaming projects go to become unmaintainable.

State is the real dividing line

Ask one question: does each key need its own state machine with its own timers?

If the answer is no — you are windowing, aggregating, deduplicating, or enriching — Spark handles it fine. Watermarks work, state is checkpointed, and you can reason about it. I covered the mechanics of watermarks and checkpoints in the Structured Streaming production guide.

If the answer is yes, Flink is built for exactly that. Keyed state with ValueState, MapState, and registered event-time timers is the core abstraction, not an escape hatch. A fraud rule that says “if three failed logins from one account within ten minutes, and no successful login since, emit an alert” is thirty lines of Flink and a genuine engineering project in Spark.

// Flink: per-key state and an event-time timer, which is the thing
// Spark makes you fight for.
public class FailedLoginRule
    extends KeyedProcessFunction<String, LoginEvent, Alert> {

  private transient ListState<Long> failures;

  @Override
  public void open(Configuration cfg) {
    failures = getRuntimeContext().getListState(
        new ListStateDescriptor<>("failures", Long.class));
  }

  @Override
  public void processElement(LoginEvent e, Context ctx, Collector<Alert> out)
      throws Exception {
    if (e.successful()) {
      failures.clear();
      return;
    }
    failures.add(e.timestamp());
    ctx.timerService().registerEventTimeTimer(e.timestamp() + 600_000L);

    long recent = 0;
    for (long t : failures.get()) {
      if (e.timestamp() - t <= 600_000L) recent++;
    }
    if (recent >= 3) out.collect(new Alert(ctx.getCurrentKey(), recent));
  }

  @Override
  public void onTimer(long ts, OnTimerContext ctx, Collector<Alert> out)
      throws Exception {
    // expire the window without a full state scan
    failures.clear();
  }
}

The Spark equivalent means holding a state object per group, managing your own timeout via GroupState.setTimeoutTimestamp, and accepting that the timeout only fires when a batch runs. It works. It is not the same tool.

Watermarks and late data behave differently enough to matter

Both engines use watermarks, and both drop late events on purpose. The difference is control.

Flink lets you set allowed lateness per window and route dropped events to a side output. You can build a “late arrivals” topic and actually see what you are losing. Spark gives you a watermark and a state timeout, and late records past the watermark are silently discarded. Nobody sets an alert on records Spark did not tell them about.

If your data has genuinely messy event time — mobile clients, IoT devices, anything with offline buffering — that side output is not a nice-to-have. I have spent more time debugging “where did those rows go” on Spark streams than on Flink ones, and the side-output ability is most of the reason. The event time and watermark post goes deeper on the semantics.

Exactly-once: the claim both engines make and neither fully keeps

Flink gives you exactly-once state via distributed snapshots, and end-to-end exactly-once to sinks that support two-phase commit — Kafka, and file sinks with atomic rename.

Spark gives you exactly-once via checkpointed offsets plus idempotent or transactional sinks, which in practice means Delta, Iceberg, or foreachBatch with your own logic.

Neither of them makes a MERGE into Postgres exactly-once. The moment the sink is an external database, you own idempotency yourself — a natural key, a MERGE, or a dedupe on a content hash. This is the same boundary I described in the Kafka exactly-once post, and it does not move because you changed engines.

Operations, which is where the decision usually lands

Here is the part vendor comparisons skip.

Spark means one engine, one skill set, one deployment story. If your team runs Databricks or EMR for batch, a streaming job is a different trigger on tooling that already exists. Autoscaling, logging, and the Spark UI are the same. On Databricks it is a job with a checkpoint location, and the operational surface is close to zero new things.

Flink means a second cluster, a second set of failure modes, and a savepoint discipline your team has to learn. Job upgrades go through savepoints, not restarts. State backend sizing (RocksDB tuning, incremental checkpoints, local disk) becomes real work once state passes a few hundred gigabytes. Managed options — Confluent, Ververica, Kinesis Data Analytics — reduce this but do not remove the concepts.

I have never regretted choosing Spark for a job that fit the micro-batch model. I have regretted choosing Spark for a job that needed per-key timers, and I have regretted introducing Flink for a job whose real requirement turned out to be five-minute freshness.

flowchart TD
  q1{Sub-second latency required?} -->|yes| flink[Flink]
  q1 -->|no| q2{Per-key state machines or timers?}
  q2 -->|yes| flink
  q2 -->|no| q3{Already running Spark for batch?}
  q3 -->|yes| spark[Spark Structured Streaming]
  q3 -->|no| q4{Is streaming even the requirement?}
  q4 -->|no| batch["Incremental batch + dbt"]
  q4 -->|yes| flink

Two of the four paths end somewhere other than a new streaming cluster.

Pitfalls

Choosing streaming when the SLA is 15 minutes. An incremental batch job on a five-minute schedule is cheaper, easier to backfill, and easier to hire for. Streaming is an operational commitment, not a performance upgrade.

Unbounded state in either engine. A stream-stream join without a watermark, or a keyed state with no TTL, grows until the job dies. Set state retention explicitly. Both engines will happily let you not do this.

Assuming Flink’s latency automatically reaches your users. If the sink is a warehouse that batches commits, or a BI tool with a 15-minute cache, your millisecond pipeline delivers 15-minute data. Measure end-to-end freshness, not operator latency.

Running Flink SQL and thinking you have avoided the operations. Flink SQL is excellent and it still produces a stateful job with checkpoints, savepoints, and a state backend to size.

Restarting a Flink job instead of taking a savepoint. You will lose state and rebuild from the retention window, which is a different dataset than the one you had.

Micro-batch tuning by shrinking the trigger. A one-second trigger on a job whose batch takes four seconds gives you a permanent backlog, not fresher data.

FAQ

Is Flink always lower latency than Spark? For the same job, yes, structurally — Flink has no batch boundary. Whether that difference reaches anyone depends on your sink and your consumers. Sub-second matters for alerting and serving; it rarely matters for a dashboard.

Can Spark do everything Flink does? Almost, with more code. flatMapGroupsWithState covers arbitrary keyed logic and Spark 3.5+ improved the API considerably. The question is whether you want to maintain that code, not whether it compiles.

What about Kafka Streams? A real option when the job is Kafka-to-Kafka and fits in a JVM service you already deploy. No cluster, no scheduler, scales by running more instances. If you do not need Flink’s operator graph or Spark’s ecosystem, it is often the least ceremony — see the Kafka Connect post for where connectors end and processing begins.

We are on Databricks. Is Flink even worth evaluating? Only for genuine event-driven state machines or sub-second SLAs. If you are windowing and aggregating, Structured Streaming on the platform you already pay for wins on total cost, including the cost of your attention.

How much state is too much for Spark? There is no clean number, but once keyed state passes low hundreds of gigabytes and you care about recovery time, Flink’s incremental RocksDB checkpointing is a materially better story.

What this means for data engineers

Decide on state and SLA, in that order, before anyone opens a benchmark. Per-key state machines and sub-second reaction are Flink problems. Windowed aggregation on a stack that already runs Spark is a Spark problem, and pretending otherwise costs you a cluster and a skill set you have to keep staffed.

Then check whether the requirement is streaming at all. A large share of “real-time” requests I have taken are satisfied by a five-minute incremental batch, and that pipeline is one you can backfill, test, and hand to someone else on a Friday.