Beam Runners and Dataflow Cost: Portability vs Workers That Never Scale to Zero

The streaming Dataflow job processed card authorizations. Daytime it needed four n2-standard-4 workers. At 20:00 traffic fell off a cliff. Autoscaling’s floor was one worker — Dataflow streaming will not go to zero — and the job sat there until 06:00 doing almost nothing. Finance forwarded a $6,200 month that was mostly nights, weekends, and a Streaming Engine line nobody had put in the design doc.

The pipeline was “portable.” We never ran it on Flink. We never ran it on Spark. We ran it on Dataflow because that is what the quickstart used, and we paid Dataflow’s always-on floor for a workload that was idle sixteen hours a day.

flowchart TD
  code[Beam pipeline] --> run{Runner}
  run -->|Dataflow streaming| df[Min 1 worker, 24x7]
  run -->|Dataflow batch| batch[Workers die at job end]
  run -->|Flink / Spark| other[You own the cluster]
  df --> bill[vCPU + RAM + Shuffle + Streaming Engine]
  batch --> bill
  df --> idle{Night traffic ~0?}
  idle -->|yes| waste[Pay the floor anyway]
  idle -->|no| ok[Streaming is the shape]

Portability chooses a runner. The bill is the runner’s. Streaming Dataflow has a floor of one worker.

The portable model is not the invoice

Beam’s promise is one pipeline, many runners. The transforms are portable. The cost model is not. Dataflow bills worker vCPU-hours, memory, Persistent Disk, and — if you enable them — Dataflow Shuffle and Streaming Engine as separate SKUs. Flink on a GKE cluster bills the nodes you forgot to scale in. Spark on Dataproc bills the cluster you left up.

I treat “we might run this on Flink later” as a lie until there is a second runner in CI. If the only runner is Dataflow, write the pipeline as a Dataflow job and own that bill. The longer comparison of Dataflow versus a Spark cluster is Dataflow vs Dataproc. This post is the Beam-shaped ways that bill goes wrong.

import apache_beam as beam
from apache_beam.options.pipeline_options import (
    GoogleCloudOptions,
    PipelineOptions,
    StandardOptions,
    WorkerOptions,
)
from apache_beam.transforms.window import FixedWindows
from apache_beam.transforms.trigger import AfterWatermark, AccumulationMode


def pipeline_options() -> PipelineOptions:
    opts = PipelineOptions(save_main_session=True)
    std = opts.view_as(StandardOptions)
    std.runner = "DataflowRunner"
    std.streaming = True  # this is the 24x7 floor

    gcp = opts.view_as(GoogleCloudOptions)
    gcp.project = "acme-data"
    gcp.region = "us-central1"
    gcp.temp_location = "gs://acme-dataflow/tmp"
    gcp.staging_location = "gs://acme-dataflow/stg"

    workers = opts.view_as(WorkerOptions)
    workers.max_num_workers = 8
    workers.num_workers = 1
    # Autoscaling will not go below 1 in streaming.
    return opts


class ToKv(beam.DoFn):
    def process(self, row: dict):
        yield row["merchant_id"], int(row["amount_cents"])


def build(p: beam.Pipeline) -> None:
    (
        p
        | "read" >> beam.io.ReadFromPubSub(topic="projects/acme/topics/auth")
        | "kv" >> beam.ParDo(ToKv())
        | "win" >> beam.WindowInto(
            FixedWindows(60),
            trigger=AfterWatermark(),
            accumulation_mode=AccumulationMode.DISCARDING,
        )
        | "sum" >> beam.CombinePerKey(sum)
        | "write" >> beam.io.WriteToBigQuery(
            table="acme:marts.auth_1m",
            write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
        )
    )

WRITE_APPEND plus a job restart is how you double a minute. BigQuery does not participate in Dataflow’s exactly-once story the way you want. Land with an insert id or a merge key, or write to a partitioned table you can overwrite for the closed windows. Streaming Engine will not save you from that.

Workers that never scale to zero

Streaming jobs keep at least one worker. That is the product: the pipeline is a long-lived process. If your traffic is a daytime hump, you are buying 24×7 compute for an 8-hour job.

The fix I actually shipped for that $6,200 job: drain the streaming job at 20:30, run a Dataflow batch job every five minutes overnight on the same subscription (or a BigQuery load from the hour’s files), and bring streaming back at 06:00. Ugly. Thirty percent of the bill. The elegant version was “set min workers to zero,” which Dataflow streaming will not do.

If the overnight SLA is 15 minutes, you did not need streaming at all. A scheduled batch Beam job — or Spark AvailableNow, see Structured Streaming — dies when the data ends. That is the scale-to-zero you wanted.

Beam versus Spark

Beam earns its complexity when you need event-time windowing on a runner you do not want to operate, and the team can debug a stuck watermark at 2 a.m. For batch transforms a Spark team already knows, Beam is a tax: new API, new failure messages, same shuffle.

I will not retrain a Spark bench onto Beam to save a theoretical runner move. I will write Beam when Dataflow streaming is the chosen runtime and the windows are the reason. I will write Spark on Dataproc when the team is a Spark team. The infrastructure delta is smaller than the rewrite.

Portability as a resume line is how you get one runner, two abstractions, and a bill you cannot explain.

Failure modes

Streaming for a batch SLA. You pay the night.

max_num_workers without a budget alert on the job id. A watermark stall scales out and stays there.

Shuffle / Streaming Engine enabled by default, never costed. They are usually worth it. They are never free.

WRITE_APPEND on replay. Duplicates. Use a key.

“We’ll run Flink later.” Later never comes. CI the second runner or drop the story.

What to do Monday

Label every Beam job batch or streaming. If it is streaming, price one worker × 730 hours and put that in the design doc before the first deploy. Turn on billing export for dataflow.googleapis.com and alert on the job name. If overnight volume is near zero, drain and batch. If the team writes Spark, write Spark. Portability is a test you run, not a flag you set.