AWS Glue vs EMR for Spark: The Cost Model, the Cold Starts, and the Migration Point

Glue and EMR both run Spark. That is where the overlap ends. Glue is a job runner: you hand it a script, it hands you a bill per DPU-hour. EMR is a cluster: you own the instances, the scaling, the Spark config, and everything that can go wrong between them.

I have run both in production, usually at the same time, because most teams end up with a split: Glue for the wide shallow layer of ingestion jobs, EMR for the three jobs that actually hurt. This post is the cost math, the cold-start reality, and the specific point where I move a job from one to the other.

If you want the wider context of where Spark sits in the AWS stack, start with the AWS data engineering stack guide. This post assumes you already know you need Spark and are deciding who runs it.

flowchart LR
  script[Spark job] --> decide{Who runs it?}
  decide -->|"Glue: serverless, DPU-hours"| glue[Glue job run]
  decide -->|"EMR: instances, your cluster"| emr[EMR cluster]
  glue --> s3[S3 output]
  emr --> s3

Same Spark, same S3. The difference is who owns the machines and the bill.

The cost model, without the marketing

Glue charges per DPU-hour, billed per second with a one-minute minimum per run. A standard DPU is 4 vCPU and 16 GB. A typical mid-size job at 10 DPUs for 20 minutes is roughly 3.3 DPU-hours. At the on-demand rate that is small money — until you multiply by 400 runs a day.

EMR charges for the EC2 instances plus an EMR uplift per instance. The uplift is small. The instances are the real number, and spot pricing applies to them, which is the single biggest lever EMR has over Glue. Glue has no spot. You pay the DPU rate whether AWS had spare capacity or not.

The crossover math I actually use: take your total Glue DPU-hours per day. Price the same vCPU and memory as EMR on spot with managed scaling. In my experience the EMR number lands at 30–50% of the Glue number for sustained workloads — before you count the engineering time EMR costs you. That engineering time is not zero. A cluster is a pet until you make it not one.

The honest summary: Glue wins when jobs are short, bursty, and numerous. EMR wins when jobs are long, heavy, and few. The uncomfortable middle is where most teams live.

Cold starts are a feature of the pricing model

A Glue job run does not start on your code. It starts on AWS provisioning capacity for your DPUs. In my logs that is 30–90 seconds on Glue 4.0, occasionally worse. For a nightly batch job nobody cares. For a job that runs every five minutes, you are paying a minute of latency tax on a five-minute cycle, and your freshness SLA is quietly eating it.

EMR has no per-job cold start if the cluster is already up — which is the point, and also the trap. A long-running EMR cluster is a machine you pay for at 3 a.m. EMR Serverless splits the difference with pre-initialized capacity you can keep warm, but now you are back to paying for idle, just with a nicer API.

My rule: if a pipeline’s cycle time is under 15 minutes, Glue’s cold start is a real fraction of the budget and I look at a long-running EMR cluster or move the workload to streaming entirely — see Kinesis patterns for when micro-batch Spark is the wrong shape.

Library control: the actual migration trigger

Cost gets the attention. Dependencies force the move.

On Glue you get the runtime AWS built. You can add pure-Python packages with --additional-python-modules, and JARs with --extra-jars. What you cannot do: pick your Spark version independently of the Glue version, install native libraries that need root, or patch the JVM. If your job needs a specific pyarrow that fights Glue’s pinned one, you will find out in the worst way — at runtime, in the third environment you deploy to.

aws glue create-job \
  --name orders-enrichment \
  --role arn:aws:iam::123456789012:role/glue-etl \
  --glue-version "4.0" \
  --number-of-workers 10 \
  --worker-type G.1X \
  --command '{
    "Name": "glueetl",
    "ScriptLocation": "s3://acme-etl/scripts/orders_enrichment.py",
    "PythonVersion": "3"
  }' \
  --default-arguments '{
    "--additional-python-modules": "great_expectations==0.18.12,phonenumbers",
    "--extra-jars": "s3://acme-etl/jars/spark-xml_2.12-0.17.0.jar",
    "--enable-metrics": "true",
    "--enable-spark-ui": "true",
    "--spark-event-logs-path": "s3://acme-etl/spark-logs/",
    "--job-bookmark-option": "job-bookmark-enable"
  }'

On EMR you own the AMI, the bootstrap actions, and the spark-defaults. Anything that installs on Amazon Linux installs on your cluster. That is the whole pitch. The day a data scientist hands you a job that needs a CUDA build, a geospatial stack with GDAL, or a Spark patch that has not shipped in Glue yet, the Glue conversation is over.

The reverse trigger exists too. I have moved jobs from EMR back to Glue when the cluster existed only to run two small hourly jobs and the maintenance burden — AMI patching, Spark upgrades, the occasional 4 a.m. spot reclaim — cost more than the DPU premium.

What EMR looks like when it is done properly

EMR in production means: instance fleets with spot, managed scaling, and clusters that are cattle. Here is the shape of a transient cluster I run per nightly batch, submitted from Airflow:

{
  "Name": "nightly-batch-orders",
  "ReleaseLabel": "emr-7.1.0",
  "Applications": [{ "Name": "Spark" }],
  "Instances": {
    "InstanceFleets": [
      {
        "InstanceFleetType": "MASTER",
        "TargetOnDemandCapacity": 1,
        "InstanceTypeConfigs": [{ "InstanceType": "m6g.xlarge" }]
      },
      {
        "InstanceFleetType": "CORE",
        "TargetSpotCapacity": 8,
        "TargetOnDemandCapacity": 2,
        "InstanceTypeConfigs": [
          { "InstanceType": "r6g.2xlarge", "WeightedCapacity": 2 },
          { "InstanceType": "r6gd.2xlarge", "WeightedCapacity": 2 },
          { "InstanceType": "r5.2xlarge", "WeightedCapacity": 2 }
        ]
      }
    ],
    "KeepJobFlowAliveWhenNoSteps": false
  },
  "ManagedScalingPolicy": {
    "ComputeLimits": {
      "UnitType": "InstanceFleetUnits",
      "MinimumCapacityUnits": 4,
      "MaximumCapacityUnits": 24
    }
  },
  "AutoTerminationPolicy": { "IdleTimeout": 900 },
  "LogUri": "s3://acme-etl/emr-logs/"
}

Three deliberate choices in there. Multiple instance types in the spot fleet, because a single-type spot fleet is a reclaim outage waiting to happen. A couple of on-demand core nodes, so HDFS does not lose blocks when spot vanishes mid-shuffle. And KeepJobFlowAliveWhenNoSteps: false plus an idle timeout, because the most expensive EMR cluster is the one everyone forgot.

If you cannot commit to that operational posture, you are not choosing EMR, you are choosing an incident. Stay on Glue.

The migration point most teams hit

There is a predictable arc. Team starts on Glue because there is nothing to operate. The catalog, crawlers, and bookmarks make the first ten pipelines fast. Then two things grow: the DPU-hours line on the invoice, and the number of --additional-python-modules hacks in the job arguments.

The migration point is when either of these is true:

• One job (or job family) accounts for the majority of DPU-hours and runs more than ~4 hours of DPU time per day, every day.
• A dependency cannot be expressed as a pip install into Glue’s pinned runtime.

Move those jobs. Only those. The long tail of small ingestion jobs almost never earns a cluster. The steady state I keep landing on: Glue runs 80% of the jobs and 30% of the spend, EMR runs 20% of the jobs and the heavy compute, and the S3 layout is identical for both so a job can move without touching the data.

flowchart TD
  start[New Spark job] --> glue[Run it on Glue]
  glue --> watch{"DPU-hours > 4/day or native deps?"}
  watch -->|no| stay[Stay on Glue]
  watch -->|yes| emr[Move to EMR fleet]
  emr --> spot["Spot + managed scaling + auto-terminate"]

Default to Glue. Promote individual jobs, not the whole platform.

Pitfalls

Sizing Glue by adding workers. Ten G.1X workers on a job that is skew-bound does nothing but multiply the bill. Read the Spark UI (enable it — it is one argument) before you scale.

Long-running Glue jobs as a cost strategy. A Glue job that runs six hours nightly is EMR-shaped. The DPU premium on long jobs is exactly the money spot instances would save you.

Single instance type in an EMR spot fleet. One capacity event in that pool and your cluster halves mid-job. Always diversify types and let weighted capacity absorb it.

Trusting job bookmarks blindly. Glue bookmarks track processed files, but a rewritten upstream file with the same key will be skipped. Idempotent partition overwrites are safer than bookmark faith.

Forgetting the cold start in SLA math. A 5-minute Glue schedule with a 90-second cold start is a 30% latency tax. Measure end-to-end freshness, not job runtime.

Keeping the EMR cluster warm “just in case.” Auto-termination exists. An idle EMR cluster is the AWS equivalent of a Snowflake warehouse that never suspends — same disease, see the Snowflake cost guide for the same lecture in a different accent.

FAQ

Is Glue just managed Spark? Mostly, plus the Data Catalog, crawlers, and bookmarks. The Spark is real Spark, but pinned: Glue version decides Spark version. You trade version control for zero operations.

What about EMR Serverless — does it make this whole debate moot? It narrows it. EMR Serverless gives you per-job billing with more Spark control than Glue and custom images. But pre-initialized capacity reintroduces idle cost, and you still cannot beat a well-run spot fleet on price for sustained load. I treat it as Glue’s ceiling, not EMR’s replacement.

Do I still need the Glue Data Catalog if I run EMR? Yes, keep it. The catalog is the shared metastore both engines (and Athena) read. Moving compute between Glue and EMR should never require touching table definitions.

Can I run the same PySpark script on both? Almost. Strip the GlueContext and Job.init wrapper, keep everything in plain SparkSession code, and inject paths as arguments. I write jobs Spark-first and Glue-wraps-them, so the migration is a submit command, not a rewrite.

When is neither the right answer? When the data fits on one machine. A surprising number of “Spark jobs” are 20 GB of Parquet that DuckDB handles in a container for a fraction of the cost and none of the cluster ceremony.

What this means for data engineers

Default new Spark work to Glue and instrument DPU-hours from the first run. The bill and the dependency list will tell you which jobs have outgrown it — you do not need to guess up front.

When a job crosses the line — sustained hours daily, or a dependency Glue cannot hold — move that job to an EMR instance fleet with spot, managed scaling, and auto-termination. Keep the catalog and the S3 layout shared so the move is a submit command.

The platforms are not rivals. They are two price points for the same engine. Pay the serverless premium where it buys you operations you do not want, and stop paying it where it does not.