Dataflow vs Dataproc: The Beam Model, the Cost Profiles, and Which One Your Team Can Actually Run

I have watched three teams pick Dataflow because it was “the Google-native option” and regret it, and one team pick Dataproc for a streaming workload and regret that much harder. The choice is routinely made on the wrong axis.

The framing that leads people wrong is treating these as two managed Spark-equivalents at different price points. They are not. Dataproc is managed Hadoop and Spark: you get a cluster, you own its configuration, and everything you know about Spark applies. Dataflow is a managed runtime for Apache Beam specifically, and Beam is a different programming model with a genuinely different mental picture of what a pipeline is.

That means the decision has a hard prerequisite most comparisons skip: can your team write and debug Beam? Not “can they learn it” — can they debug a stuck watermark at 2 a.m. six months from now? If the answer is no, the cost tables do not matter, because you will end up rewriting the pipeline in Spark anyway.

Here is what each one actually is, what the bills look like with real numbers, and the three questions I ask before recommending either.

Two different bets

Dataproc gives you a cluster of Compute Engine VMs with Hadoop, Spark, Hive, Flink, and Presto installed, plus autoscaling and a serverless mode. You submit Spark jobs. You control the Spark version, the JARs, the machine types, the initialization actions. It is the GCP equivalent of EMR, and the same operational trade-offs I wrote up in Glue vs EMR transfer almost line for line.

Dataflow runs Beam pipelines. There is no cluster you configure; you submit a pipeline and Dataflow provisions workers, autoscales them, and handles shuffle and streaming state as managed services. You do not pick a Spark version because there is no Spark. You write Beam in Java, Python, or Go, and Dataflow is the runner.

flowchart TD
  work["Pipeline to build"] --> mode{"Batch or streaming?"}
  mode -->|"streaming, event-time windows"| beam["Beam on Dataflow"]
  mode -->|"batch only"| team{"Does the team write Spark today?"}
  team -->|"yes"| proc["Dataproc, or Dataproc Serverless"]
  team -->|"no, and no streaming"| sql{"Can it be SQL?"}
  sql -->|"yes"| bq["BigQuery scheduled queries or dbt"]
  sql -->|"no"| either["Either. Pick on ops appetite."]
  beam --> engine["Enable Streaming Engine"]
  proc --> spot["Preemptible secondary workers"]

Streaming with event-time semantics is the case Beam was built for. Batch-only work rarely justifies the model change.

What the Beam model actually gives you

Beam’s core idea is that batch and streaming are the same computation over different data completeness assumptions. A PCollection is a distributed dataset that may be bounded or unbounded, and the same transforms apply to both. In principle you write once and run in batch or streaming by changing the source.

In practice, the “unified” promise is oversold and the thing that genuinely matters is narrower and more valuable: Beam has the best event-time semantics of any pipeline framework I have used. Windows, watermarks, triggers, and accumulation modes are first-class, explicit, and correct.

"""Streaming pipeline with event-time windowing and explicit late
data handling. This is the code shape that justifies Beam."""

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.transforms import window, trigger

options = PipelineOptions(streaming=True, save_main_session=True)

with beam.Pipeline(options=options) as p:
    (p
        | "ReadPubSub" >> beam.io.ReadFromPubSub(
              subscription="projects/acme/subscriptions/orders-sub",
              with_attributes=False,
          )
        | "Parse" >> beam.Map(parse_order)
        # Event time comes from the payload, not arrival time. This is
        # the whole point -- a message delayed 40 minutes still lands
        # in the window it belongs to.
        | "Timestamp" >> beam.Map(
              lambda o: beam.window.TimestampedValue(o, o["event_epoch"])
          )
        | "Window" >> beam.WindowInto(
              window.FixedWindows(60),           # 1-minute event-time windows
              # Emit an early speculative result every 15s of processing
              # time, then a final result on watermark, then updates for
              # late data arriving within the 30-minute allowance.
              trigger=trigger.AfterWatermark(
                  early=trigger.AfterProcessingTime(15),
                  late=trigger.AfterCount(1),
              ),
              allowed_lateness=1800,
              accumulation_mode=trigger.AccumulationMode.ACCUMULATING,
          )
        | "SumByCountry" >> beam.CombinePerKey(sum)
        | "ToBQ" >> beam.io.WriteToBigQuery(
              table="acme:analytics.revenue_per_minute",
              write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
              method="STORAGE_WRITE_API",
          ))

Reproducing that correctly in Structured Streaming is possible and noticeably more awkward, particularly the combination of early triggers, allowed lateness, and accumulating mode. If your business logic depends on “what was true as of 14:03 event time, corrected as late data arrives,” Beam is the right tool and Dataflow is the best runner for it.

If your pipeline is “read yesterday’s files, join, aggregate, write,” none of that machinery does anything for you. You are paying Beam’s learning curve, its more limited connector ecosystem, and its harder debugging story for a capability you are not using.

Dataflow’s cost profile

Dataflow bills per vCPU-hour, per GB of memory-hour, and per GB of persistent disk, all metered per second while workers exist. On top of that sit two separately billed managed services that most production pipelines should enable.

Dataflow Shuffle (batch) moves shuffle out of worker disks into a managed service, billed per GB processed. It makes autoscaling actually work for batch, because workers become stateless. Without it, scaling down means moving shuffle data.

Streaming Engine does the same for streaming state and timers, billed per GB of streaming data processed. It cuts worker CPU and disk needs substantially and is required for several autoscaling behaviors.

The mistake in every Dataflow cost estimate I have reviewed is budgeting only the worker line. On a shuffle-heavy batch job, the Shuffle charge has been 25 to 40 percent of the total for me. On a busy streaming pipeline, Streaming Engine has been a third. They are still worth enabling — the alternative is more workers and worse autoscaling — but they belong in the forecast.

# Launch a Flex Template with the settings I actually use in prod.
gcloud dataflow flex-template run "orders-enrich-$(date +%Y%m%d-%H%M)" \
  --template-file-gcs-location=gs://acme-templates/orders-enrich.json \
  --region=us-central1 \
  --parameters=input=gs://acme-lake/orders/dt=2026-05-15/ \
  --parameters=output=acme:analytics.fct_orders \
  --max-workers=25 \
  --worker-machine-type=n2-standard-4 \
  --additional-experiments=enable_streaming_engine \
  --additional-experiments=shuffle_mode=service \
  --staging-location=gs://acme-dataflow/staging \
  --temp-location=gs://acme-dataflow/temp \
  --service-account-email=dataflow-runner@acme.iam.gserviceaccount.com \
  --disable-public-ips

Dataflow supports spot VMs for batch through Flexible Resource Scheduling, which trades scheduling latency (up to six hours) for a substantial discount. It is excellent for nightly batch with a loose deadline and useless for anything time-sensitive. Streaming pipelines cannot use spot, which is the structural reason Dataflow streaming is expensive relative to a preemptible Dataproc cluster.

Dataproc’s cost profile

Dataproc charges for the Compute Engine instances plus a small per-vCPU Dataproc premium. The instances are the number, and they can be preemptible, which is the single largest cost lever either platform offers.

The standard production shape: on-demand primary and a small set of on-demand primary workers to hold HDFS and shuffle data, plus preemptible secondary workers for the bulk of compute. Secondary workers can vanish with 30 seconds notice, so they must not hold data you cannot lose.

gcloud dataproc clusters create nightly-batch \
  --region=us-central1 \
  --image-version=2.2-debian12 \
  --master-machine-type=n2-standard-4 \
  --num-workers=2 \
  --worker-machine-type=n2-highmem-8 \
  --num-secondary-workers=12 \
  --secondary-worker-type=spot \
  --enable-component-gateway \
  --autoscaling-policy=batch-scale-policy \
  --max-idle=30m \
  --properties=^#^spark:spark.sql.adaptive.enabled=true\
#spark:spark.sql.adaptive.coalescePartitions.enabled=true\
#spark:spark.shuffle.service.enabled=true \
  --enable-kerberos=false \
  --no-address

Two settings there do most of the work. --max-idle=30m terminates the cluster when nobody is using it, which is the difference between a $300 month and a $4,000 month. And enabling adaptive query execution lets Spark fix partition skew at runtime, which is the single most effective Spark tuning change available and costs nothing.

Dataproc Serverless deserves separate mention because it changes the comparison meaningfully. You submit a Spark batch or interactive session, Google provisions and tears down capacity, and you are billed per Data Compute Unit-hour. No cluster to manage, real Spark, your own dependencies via a custom container image.

gcloud dataproc batches submit pyspark \
  gs://acme-jobs/orders_enrich.py \
  --region=us-central1 \
  --version=2.2 \
  --container-image=us-central1-docker.pkg.dev/acme/spark/orders:1.14.0 \
  --properties=\
spark.dynamicAllocation.enabled=true,\
spark.dynamicAllocation.minExecutors=2,\
spark.dynamicAllocation.maxExecutors=40,\
spark.sql.adaptive.enabled=true \
  --deps-bucket=gs://acme-dataproc-deps \
  -- --run-date=2026-05-15

For batch Spark on GCP, this is my default recommendation now. It keeps the Spark skills your team has, removes the cluster you did not want to operate, and prices between a preemptible cluster and Dataflow. The custom container image solves the dependency problem that makes Glue painful on the AWS side.

Comparing the bills on one workload

A concrete case: a pipeline reading about 500 GB of GCS Parquet nightly, enriching against two BigQuery dimensions, writing to BigQuery.

On Dataproc with a transient cluster — two on-demand n2-highmem-8 workers plus twelve spot workers, running 35 minutes including provisioning — the effective cost landed around $2.10 per run, so roughly $63 a month.

On Dataproc Serverless, the same job at about 40 DCU-hours came to roughly $3.30 per run, about $99 a month, with no cluster to maintain.

On Dataflow with 20 n2-standard-4 workers for 28 minutes plus about 600 GB through Dataflow Shuffle, the total was roughly $5.40 per run, about $162 a month.

So Dataflow was around 2.5x the transient preemptible cluster and 1.6x serverless Spark for this batch job. That is the typical spread, and it is not large enough to decide anything by itself at this scale. At ten times the volume it becomes a real number worth a migration; at this scale, the deciding factor is which one your team can operate without incident.

Streaming inverts the comparison. A steady Pub/Sub pipeline on Dataflow with Streaming Engine and autoscaling has consistently cost me less in total than a Dataproc cluster running Spark Structured Streaming, because the Dataproc cluster must be sized for peak and runs 24 hours a day, while Dataflow scales workers with backlog. Add the engineering time of managing checkpoints and restarts on a long-running Spark cluster and it is not close.

Which one fits your team

Three questions, in order.

Do you have real streaming requirements with event-time semantics? If yes, Dataflow. Beam’s windowing model and Dataflow’s managed state are the strongest combination available on GCP, and the correctness you get on late data is worth the learning curve.

Does your team already write Spark? If yes and you are batch-only, Dataproc Serverless. The value of existing Spark fluency — knowing how to read a stage, diagnose skew, tune shuffle partitions — is larger than the cost difference. Beam’s debugging story is genuinely harder, and a team that cannot read a Dataflow execution graph will be stuck on problems that a Spark team solves in twenty minutes.

Could this be SQL in BigQuery instead? Ask this before either. A large share of the Dataflow and Dataproc batch jobs I have reviewed were joins and aggregations that BigQuery does better, cheaper, and with no pipeline code to maintain. Get the pruning right and a scheduled query beats both — the reasoning is in the BigQuery cost post and the partitioning guide. Reach for a compute framework when you need something SQL cannot express, not by default.

Where teams get this wrong

Choosing Dataflow for batch because it is “more Google-native.” Native is not a technical property. For batch Spark you get better economics, better debugging, and better dependency control from Dataproc Serverless.

Running Spark Structured Streaming on a persistent Dataproc cluster. You pay for peak capacity constantly, you own checkpoint recovery, and you get a pager. This is the case Dataflow exists to solve.

Budgeting Dataflow without Shuffle and Streaming Engine. They are separate line items and routinely 25 to 40 percent of the total. Enable them anyway — the alternative is worse — but forecast them.

No --max-idle on a Dataproc cluster. The most expensive cluster on GCP is the one a contractor spun up in March. Set idle termination on every cluster, including the ones you promise to delete manually.

Assuming Beam pipelines port between runners. They do in principle. In practice, connector behavior, state handling, and performance differ enough that “we can move to Flink later” is not a plan you should price into the decision.

Ignoring Dataflow’s update semantics. Updating a running streaming pipeline requires compatible state, and an incompatible transform change means draining and restarting. Design your transform names and state shapes with that in mind from day one, because renaming a step can break the update path.

FAQ

Is Dataflow just managed Beam, or is there more to it?

It is the reference runner for Beam plus managed shuffle and streaming state, autoscaling, and observability. The Beam SDK is open source and runs on Flink or Spark, but Dataflow’s Streaming Engine and dynamic work rebalancing are proprietary and are a large part of why it performs well.

Can I run Spark on Dataflow?

No. Dataflow runs Beam pipelines only. You can run Beam on Spark via the Spark runner on a Dataproc cluster, which is a configuration almost nobody should choose — you get Beam’s abstraction with a cluster to operate and none of Dataflow’s managed services.

Does Dataproc Serverless replace regular Dataproc?

For batch, mostly yes, and it is my default. Persistent clusters still make sense when you need Hive, HBase, Presto, or long-lived notebook sessions against a shared metastore, or when preemptible economics at large scale justify the operational cost.

How does Dataflow compare to Cloud Run jobs for small pipelines?

For a job processing tens of gigabytes, a Cloud Run job with plain Python or DuckDB is dramatically cheaper and simpler than either option here. Neither Dataflow nor Dataproc earns its overhead until the work genuinely needs distribution — see DuckDB in production for where that line sits.

What is the hardest part of operating Dataflow?

Diagnosing a stalled pipeline. When the watermark stops advancing, the cause can be a slow external call inside a DoFn, a hot key, a source that stopped reporting, or state that grew unbounded. The observability is decent, but the mental model required is deeper than reading a Spark stage list, and this is where teams without Beam experience lose days.

Which one do I use with Airflow?

Both have first-class operators, and orchestration is not a differentiator here. Use deferrable operators so a two-hour job does not occupy a worker slot the whole time; the reasoning is in the deferrable operators guide.

What this means for your pipelines

Sort your GCP compute work into three buckets before choosing anything. Streaming with event-time correctness requirements goes to Dataflow, and that is the one case where I consider Beam’s learning curve clearly worth paying. Batch Spark goes to Dataproc Serverless, because your team’s existing Spark fluency is an asset you should not discard for a 40 percent infrastructure difference. And everything that is really a join and an aggregation goes to BigQuery, where it should have been in the first place.

That third bucket is usually the largest and always the most surprising. Every GCP platform review I have done has found batch pipelines running expensive distributed compute to produce a result a scheduled query could have written for a few dollars a month. The pipeline existed because someone chose a framework before characterizing the workload, and then the framework chose the architecture.

Do the characterization first. Write down the data volume, the freshness requirement, whether event-time correctness matters, and what your team can debug under pressure. Those four facts determine the answer, and none of them appear in a feature comparison table. The cost difference between these platforms is real but second order; the cost of running a pipeline your team cannot debug is not.