Step Functions vs Airflow for Data Orchestration: Cost, Observability, and the Point Where You Outgrow One

Every team I have joined in the last five years has had this argument, and it almost always goes badly because the two sides are describing different problems. The platform engineer says Step Functions, because it is serverless, retries natively, and integrates with everything in the account. The analytics engineer says Airflow, because they need to backfill March and rerun the downstream models when a source changes.

Both are right. They are also not talking about the same thing. Step Functions is a workflow engine for AWS API calls. Airflow is a scheduler for a graph of data dependencies. The overlap is real but partial, and the mistake is treating the overlap as the whole picture.

I have run both in production, at the same time, deliberately. This post is the cost math in units you can actually compare, what each one shows you at 3 a.m., and the specific signals that mean you have outgrown the one you are on.

If you are choosing between Airflow and its competitors rather than against Step Functions, Airflow vs Dagster vs Prefect is the post for that. This one assumes you are inside AWS and wondering whether you need Airflow at all.

What each one actually models

Step Functions models a state machine. States are Task, Choice, Map, Parallel, Wait, Pass, Succeed, Fail. Each state gets its input as JSON, produces JSON, and passes it on. Retries, catches, and timeouts are declarative properties of a state, not code you write. The workflow definition is Amazon States Language, which is JSON, which is both its greatest strength (it is data, so you can generate it) and its most annoying property (it is JSON, so you write it in a JSON).

Airflow models a DAG of tasks over time. The unit is not “a workflow ran,” it is “the 2026-05-12 logical date of this DAG ran.” That temporal dimension is the entire reason Airflow exists. Backfills, catchup, data intervals, sensors that wait for yesterday’s partition — all of it flows from the idea that a pipeline is a function of a date, not an event.

That distinction predicts almost every difference downstream. Step Functions has no concept of “run this for last March.” Airflow has no native concept of “this ran because a file landed” that is as clean as an EventBridge rule.

flowchart TD
  ev["S3 file lands"] --> eb[EventBridge rule]
  eb --> sfn["Step Functions: validate, convert, register"]
  sfn --> mark["Write partition marker"]
  cron["Airflow schedule 06:00"] --> sensor["Sensor: partition marker present?"]
  sensor --> dbt["dbt build: staging to marts"]
  dbt --> tests["Data quality tests"]
  tests --> pub["Publish to BI"]

Event-driven choreography on the left, date-driven data graph on the right. They meet at a marker in S3.

The cost models, converted into comparable units

These are priced so differently that people compare them wrong. Here is how I normalize.

Step Functions Standard bills per state transition, roughly $0.025 per 1,000, plus whatever the underlying services cost. Executions can run for up to a year, history is retained, and execution is exactly-once. Express workflows bill per request plus GB-seconds of duration, cap at five minutes, and are at-least-once with no built-in execution history beyond what you log.

MWAA (Managed Workflows for Apache Airflow) bills an environment hour, plus worker hours, plus a small storage charge. A small environment runs roughly $0.49 per hour for the environment plus workers, which puts the practical floor around $350 to $400 a month with one worker, before you run anything. Self-managed Airflow on ECS or EKS is cheaper in dollars and more expensive in your time; I have done both and the break-even is somewhere around “do you have a platform team.”

Now convert. Step Functions costs nothing at rest and scales linearly with transitions. Airflow costs a fixed monthly floor and then almost nothing per task. So:

• A handful of event-driven workflows, a few thousand transitions a day: Step Functions costs a few dollars a month. Airflow costs $400. Not close.
• Two hundred DAGs, 5,000 tasks a day, every day: Airflow costs $400 to $800. The same work in Step Functions Standard is maybe 150,000 transitions a day, about $110 a month in transitions plus the operational cost of having no backfill story. Now it is close on price and Airflow wins on capability.
• A Map state iterating 50,000 files with five states each: that is 250,000 transitions per run. Run it daily and you have found the pricing cliff nobody warns you about.

That last case is worth internalizing. Step Functions pricing punishes fine-grained iteration. If your instinct is to loop per-record, either use Distributed Map with Express child workflows (much cheaper per iteration) or batch the work into a single Glue job and let Step Functions orchestrate one call instead of fifty thousand.

Observability: two genuinely different experiences

This is where Step Functions is better than people expect and where Airflow is better than people admit.

Step Functions gives you a visual execution history per run. Every state, its input, its output, the exact error, how long it took. When a workflow fails, you click the red box and see the JSON that went in and the exception that came out. There is nothing to correlate; the execution is the trace. For debugging a single failed run, this is the best experience in AWS.

Airflow gives you a grid across time. Every DAG run, every task, colored by state, going back weeks. When someone asks “has this been failing all week or just today,” Airflow answers in one glance and Step Functions cannot answer at all without querying CloudWatch. Airflow also gives you task logs in the UI, retry history, and the ability to clear a task and everything downstream of it.

The practical difference: Step Functions tells you what happened in one execution. Airflow tells you what has been happening. For data pipelines, where the question is usually about a pattern across days, that temporal view is worth a lot.

One caveat on Step Functions history: Standard workflows cap execution history at 25,000 events. A long-running workflow with a big Map state will hit that and the execution fails. It is a surprisingly common way to discover that your loop is too fine-grained.

What a real Step Functions data workflow looks like

Here is the shape I use for event-driven ingestion. Note the retry and catch blocks — this is the part you would otherwise write badly by hand in a Lambda.

{
  "Comment": "Land, convert, validate, register a new source file",
  "StartAt": "ValidateHeader",
  "States": {
    "ValidateHeader": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "validate-header",
        "Payload.$": "$"
      },
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
          "IntervalSeconds": 2,
          "MaxAttempts": 4,
          "BackoffRate": 2.0
        }
      ],
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "Next": "Quarantine" }
      ],
      "ResultPath": "$.validation",
      "Next": "ConvertToParquet"
    },
    "ConvertToParquet": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "raw-to-parquet",
        "Arguments": {
          "--source_key.$": "$.detail.object.key",
          "--target_date.$": "$.detail.partition_date"
        }
      },
      "TimeoutSeconds": 3600,
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "Quarantine" }],
      "Next": "WriteMarker"
    },
    "WriteMarker": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:s3:putObject",
      "Parameters": {
        "Bucket": "acme-lake-curated",
        "Key.$": "States.Format('_markers/orders/{}/_SUCCESS', $.detail.partition_date)",
        "Body": "ok"
      },
      "End": true
    },
    "Quarantine": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "arn:aws:sns:us-east-1:123456789012:data-alerts",
        "Message.$": "$"
      },
      "Next": "Failed"
    },
    "Failed": { "Type": "Fail", "Error": "IngestFailed" }
  }
}

Three things worth stealing. startJobRun.sync waits for the Glue job to finish and surfaces its failure as a state error — no polling loop. The Catch on every task routes to one quarantine path rather than dying silently. And the last state writes a marker file, which is how this hands off to Airflow without either system knowing about the other.

What the Airflow side looks like

from datetime import datetime, timedelta
from airflow.decorators import dag, task
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator

DEFAULTS = {
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
    "retry_exponential_backoff": True,
}


@dag(
    dag_id="orders_marts",
    schedule="0 6 * * *",
    start_date=datetime(2026, 1, 1),
    catchup=True,                 # the whole reason we are on Airflow
    max_active_runs=3,            # bound the backfill blast radius
    default_args=DEFAULTS,
    tags=["orders", "marts"],
)
def orders_marts():
    # Deferrable sensor: releases the worker slot while waiting.
    wait_raw = S3KeySensor(
        task_id="wait_for_raw_marker",
        bucket_name="acme-lake-curated",
        bucket_key="_markers/orders/{{ ds }}/_SUCCESS",
        deferrable=True,
        timeout=60 * 60 * 4,
        poke_interval=60,
    )

    build = GlueJobOperator(
        task_id="build_orders_fact",
        job_name="orders-fact",
        script_args={"--run_date": "{{ ds }}"},
    )

    @task
    def assert_row_count(run_date: str) -> int:
        """Fail loudly if the fact table is suspiciously small."""
        rows = query_scalar(
            "SELECT count(*) FROM marts.fact_orders WHERE order_date = %s",
            (run_date,),
        )
        if rows < 1_000:
            raise ValueError(f"fact_orders has only {rows} rows for {run_date}")
        return rows

    wait_raw >> build >> assert_row_count("{{ ds }}")


orders_marts()

The deferrable=True on that sensor is not a detail. A non-deferrable sensor holds a worker slot for four hours doing nothing, and three of those will wedge a small environment. I wrote about why in deferrable operators.

The point where you outgrow Step Functions

Five signals. Any two of these and I start planning an Airflow environment.

1. You need a backfill. “Rerun the last 60 days in dependency order, three at a time” is a one-line Airflow command and a custom application in Step Functions.
2. Tasks depend on other pipelines, not on events. When the trigger for pipeline B is “pipeline A finished for the same logical date,” you want a data-aware scheduler, not a chain of EventBridge rules.
3. The definition is generated JSON nobody reads. ASL is fine for 15 states. At 60 states with nested Maps, a Python DAG is dramatically more maintainable.
4. Analytics engineers need to own pipelines. dbt developers will write Python DAGs. They will not write ASL, and asking them to means every model change routes through you.
5. You need lineage and cross-DAG dependencies. Airflow assets (formerly datasets) give you “run when this table updates” across teams. Step Functions has no equivalent.

And the reverse — when I would move off Airflow:

• The only workflows are event-driven, sub-minute, and AWS-native.
• Total task volume does not justify the fixed monthly floor.
• Nobody on the team wants to own a scheduler, and MWAA upgrades keep landing on the same person.

Running both without creating a mess

The split that has worked for me every time: Step Functions owns everything between “an event happened” and “data is in the lake.” Airflow owns everything from “data is in the lake” to “the warehouse and BI layer are correct.”

They communicate through markers, not through APIs. Step Functions writes _SUCCESS to a partition prefix; Airflow’s deferrable sensor waits for it. No IAM cross-wiring, no coupling, and either side can be replaced without touching the other.

The anti-pattern is Airflow triggering Step Functions and then polling it, or Step Functions calling the Airflow REST API. Both work and both create a distributed handoff with two sources of truth about whether the work succeeded. Markers in object storage are boring and they survive both systems being down.

For the transformation half of that split, keep the logic in dbt rather than in operators. Then the orchestrator is scheduling a build command, and swapping orchestrators later costs you a day instead of a quarter.

Pitfalls

Fine-grained Map states. A Map over 50,000 items with five states each is 250,000 transitions per run and will also blow the 25,000-event history limit. Use Distributed Map with Express children, or batch the work.

Airflow tasks that do the work. An operator that pulls a dataframe into the scheduler’s worker is a scaling bug with a deadline. Orchestrators submit work; they do not perform it.

Assuming Express workflows are just cheaper Standard. Express is at-least-once with a five-minute cap and no durable execution history. Using it for a payment-adjacent or non-idempotent step is a data correctness bug waiting to happen.

MWAA sized for the steady state. Backfills are the peak. Environment class and worker count that handle Tuesday will fall over when someone reruns a quarter. Set max_active_runs and pool limits before you need them.

Cross-account IAM discovered late. Step Functions calling services in another account requires role assumptions that security will want to review. Find that out in week one, not the week of cutover.

No alerting on the orchestrator itself. Both systems can be down while individual tasks look fine. Alarm on “expected DAG runs in the last 24 hours” and on Step Functions ExecutionsFailed, not just on task errors.

FAQ

Can Step Functions replace Airflow entirely for a small team?

If your pipelines are event-driven, AWS-native, and you never backfill, yes, and you will save the monthly floor. The moment someone asks to reprocess last month with correct dependency order, you will be writing that feature yourself. Ask whether that day is coming before you commit.

Is MWAA worth it over self-managed Airflow?

For most teams, yes. You are paying a premium for someone else to handle the metadata database, scheduler HA, and version upgrades. Self-managed on EKS is cheaper per month and costs a meaningful fraction of an engineer. If you already run Kubernetes well, that math can flip.

What about Glue Workflows or EventBridge Pipes?

Glue Workflows only orchestrate Glue, which is too narrow for anything real. EventBridge Pipes is point-to-point plumbing with filtering and enrichment, not orchestration. Neither is a substitute for either tool in this post.

How do I test a Step Functions state machine?

Locally with the Step Functions Local container for the state machine logic, and with mocked service integrations for the tasks. It is genuinely worse than testing a Python DAG, where you can unit test task functions directly. Factor in that testing gap when you compare.

Does Airflow 3 change this comparison?

It sharpens Airflow’s side of it. Asset-driven scheduling, a proper task execution API, and better event triggering close some of the gap on event-driven work. It does not change the cost floor, and Step Functions is still the better fit for pure AWS service choreography.

Which one do I start with on a greenfield project?

Step Functions, almost always. It has no fixed cost, it forces you to think about retries explicitly, and if the project dies you have spent nothing. Add Airflow when the second signal from the list above shows up, and by then you will know exactly which DAGs belong in it.

What this means for your pipelines

Choose by the shape of your triggers. If work starts because something happened in AWS, Step Functions is the right home: declarative retries, native service integrations, a per-execution trace that makes debugging trivial, and no bill when nothing is running. If work starts because a date arrived and a graph of tables needs rebuilding in order, Airflow is the right home, and the monthly floor buys you backfills, catchup, and a view across time that Step Functions structurally cannot provide.

Most mature AWS data platforms end up with both, and that is not a failure of decision-making. Draw the boundary at the lake: events and ingestion on one side, scheduled transformation on the other, with a marker file as the only contract between them. That boundary is cheap to maintain and it means neither choice is permanent.

The failure mode to avoid is picking one and then bending it into the other’s shape. Airflow used as an event router with sensors polling every ten seconds is expensive and fragile. Step Functions used as a data scheduler with a hand-rolled backfill application is a product you now maintain. Both mistakes are recoverable, but both cost a quarter, and the signals that predict them are visible months in advance if you are looking.