Airflow 3 for Pipeline Authors: What Actually Changed and What to Fix First

The Airflow 3 upgrade I ran took nine days of engineering time. Seven of them went to two things: a custom operator that had been reading the metadata database directly since 2021, and about forty DAGs that formatted an S3 path out of execution_date.

The UI rewrite, which is the thing everyone talks about first, cost me roughly an hour of relearning where the log button lives.

That ratio is the most useful thing I can tell you. Airflow 3 is not a rewrite of how you author pipelines. It is a tightening of the boundary between your task code and Airflow’s internals, plus the removal of a decade of deprecated surface area. The parts that break are the parts where you were reaching through that boundary, and most teams have three or four of those without knowing it.

Here is what changed that affects how you write DAGs, in the order I would fix it.

The Task SDK boundary is the real change

In Airflow 2, a task ran inside a process that had a database connection to the metadata DB. Nothing stopped you from importing airflow.settings.Session and querying task_instance directly. Lots of code did — homegrown operators that looked up prior run states, custom sensors that polled DAG run tables, cleanup scripts that ran as tasks.

In Airflow 3, task execution goes through an API. The worker talks to a Task Execution API server; it does not hold a DB session. Your task code imports from airflow.sdk, and the things it can ask for are the things the API exposes.

flowchart TD
  subgraph Airflow2["Airflow 2"]
    t2["Task process"] --> db2[("Metadata DB")]
    sch2["Scheduler"] --> db2
  end
  subgraph Airflow3["Airflow 3"]
    t3["Task process (Task SDK)"] --> api["Task Execution API"]
    api --> db3[("Metadata DB")]
    sch3["Scheduler"] --> db3
  end

The arrow that disappeared is the one your custom operator was using.

This is a good change. It is what makes remote and multi-language execution possible, and it means a badly written task can no longer lock a table the scheduler needs. It is also the change most likely to break something you own.

Finding the damage takes one grep:

# Anything here is a candidate for rewrite, not a find-and-replace.
grep -rn "create_session\|airflow.settings\|provide_session\|@provide_session" dags/ plugins/
grep -rn "from airflow.models import" dags/ plugins/

Most of what these operators wanted is available another way. Prior run state, XComs from other DAGs, and connection lookups all have supported paths. A handful genuinely wanted arbitrary DB access, and those become a REST API call against the Airflow API server or, more often, a realization that the logic belonged outside Airflow anyway.

logical_date is optional now, and your paths depend on it

In Airflow 2, execution_date (later logical_date) always existed, even for a manual run, where it was a slightly fictional value that confused every new engineer for their first month.

In Airflow 3, a run triggered manually or by an asset update can have logical_date of None. Scheduled runs still have one. This is honest — an asset-triggered run genuinely has no logical interval — and it will break any DAG that unconditionally formats a date from it.

from airflow.sdk import dag, task
import pendulum

@dag(
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
)
def orders_daily():

    @task
    def build_path(**context) -> str:
        # Airflow 3: logical_date is None for manual and asset-triggered runs.
        logical = context.get("logical_date")
        if logical is None:
            # Fall back to the run's actual start. Decide this once,
            # per DAG, and write down why.
            logical = context["dag_run"].run_after
        return f"s3://lake/orders/dt={logical.strftime('%Y-%m-%d')}/"

    @task
    def load(path: str) -> None:
        # Idempotent write for that partition.
        ...

    load(build_path())

orders_daily()

Do not paper over this with a global default. The right fallback depends on what the DAG means. A daily aggregation that someone manually reruns almost always wants the same logical date as the scheduled run it replaces, which means the operator should pass it explicitly at trigger time. An asset-triggered ingest wants the timestamp of the data that arrived, which is in the asset event, not in a date field.

While you are in there: execution_date is gone entirely, along with next_ds, prev_ds, tomorrow_ds, yesterday_ds, and friends. ds survives. Template strings referencing removed keys fail at render time, which means they fail in production on a Tuesday rather than at parse time in CI, so grep for them explicitly.

grep -rn "execution_date\|next_ds\|prev_ds\|tomorrow_ds\|yesterday_ds" dags/

Assets replaced datasets, and scheduling got a real vocabulary

Dataset is now Asset. It is more than a rename: assets have watchers, aliases, and the ability to express conditional scheduling across multiple producers. The mechanics are worth their own treatment, and I wrote them up in dataset-driven scheduling across teams.

For upgrade purposes, the relevant fact is that the import path and class name changed, and old dataset-scheduled DAGs need updating:

# Airflow 2
# from airflow.datasets import Dataset
# orders = Dataset("s3://lake/orders/")

# Airflow 3
from airflow.sdk import Asset

orders = Asset("s3://lake/orders/")

@dag(schedule=[orders], start_date=pendulum.datetime(2026, 1, 1, tz="UTC"))
def downstream_marts():
    ...

Also gone: schedule_interval and the separate timetable argument. Everything goes through schedule, which accepts a cron string, a timedelta, a list of assets, a timetable object, or None. That consolidation is one of the nicer parts of the release — there is exactly one place to look to know when a DAG runs.

DAG versioning ends the frankenrun

This is the change I appreciate most in operations and that nobody mentions in release summaries.

In Airflow 2, a DAG run executed whatever code was on disk at the moment each task started. Deploy in the middle of a long run and tasks one through five ran the old graph, tasks six onward ran the new one. If you renamed a task, the run had an orphan. If you removed one, the run just skipped it. Debugging a failure from that morning meant guessing which version of the code was live at 04:17.

Airflow 3 tracks DAG versions. A run is associated with the version it started under, the UI shows you which version a given run used, and task instances execute against that version’s structure. The practical effect is that a mid-run deploy no longer silently produces a hybrid, and post-incident analysis stops with “here is the code that ran” instead of a git log archaeology session.

You do not have to do anything to get this. It is worth knowing because it changes how you reason about deploys: pushing during the nightly window is much less dangerous than it used to be, though I still avoid it.

Backfills are scheduler-managed

airflow dags backfill used to spawn a local process on whatever machine ran the command, which meant a laptop closing could kill a three-day backfill, and there was no visibility into it from the UI.

In Airflow 3 backfills are created as a first-class object handled by the scheduler. You can trigger one from the UI or the API, watch it, pause it, and cancel it. It respects max_active_runs and pools like any other run.

# Still a CLI, but the scheduler owns the execution now.
airflow backfill create \
  --dag-id orders_daily \
  --from-date 2026-04-01 \
  --to-date 2026-04-30 \
  --max-active-runs 3 \
  --reprocess-behavior failed

The --max-active-runs flag matters more than it looks. A month-long daily backfill with unlimited concurrency will happily submit thirty simultaneous warehouse queries and turn a cost conversation into a cost incident.

Defaults that changed underneath you

Three of these will alter behavior without an error message, which makes them worse than the breaking changes.

catchup_by_default is now False. If a DAG did not set catchup explicitly and relied on the old default of True, it will stop backfilling on unpause. This is the correct default — accidental thousand-run catchups have burned every team once — but set catchup explicitly on every DAG so the behavior does not depend on a config value someone can change.

SubDAGs are gone. They were deprecated for years and were always a deadlock generator. Replace with task groups, or with dynamic task mapping if the SubDAG was doing fan-out, which is covered in dynamic task mapping patterns.

SLAs were removed. The old SLA mechanism was unreliable enough that most teams had already replaced it with external monitoring. If you were depending on sla_miss_callback, you need a replacement before you upgrade, not after.

Standard operators moved to a provider. PythonOperator, BashOperator, and the standard sensors now live in airflow.providers.standard. Old import paths were kept working for a while through shims, but the shims are the sort of thing that disappears in a minor release, so fix the imports during the upgrade rather than after.

# Fix these during the migration, not later.
from airflow.providers.standard.operators.bash import BashOperator
from airflow.providers.standard.operators.python import PythonOperator
from airflow.providers.standard.sensors.filesystem import FileSensor

from airflow.sdk import Asset, dag, task, task_group

The pre-upgrade checklist I actually use

Do these in order. The first four are cheap and find most of the work.

# .github/workflows/airflow3-readiness.yml
name: airflow-3-readiness
on: [pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install ruff
        run: pip install ruff
      - name: Airflow 3 lint rules
        # AIR3xx rules flag removed imports, renamed args, and
        # deprecated context keys. Start with warnings, then enforce.
        run: ruff check --select AIR3 dags/ plugins/
      - name: No direct metadata DB access from task code
        run: |
          ! grep -rn "create_session\|provide_session" dags/ plugins/
      - name: No removed context keys
        run: |
          ! grep -rEn "execution_date|next_ds|prev_ds|tomorrow_ds|yesterday_ds" dags/

1. Get on the latest Airflow 2 minor first, with zero deprecation warnings in the scheduler log. Every warning you ignore is a breaking change in 3.
2. Run the ruff AIR3 rules across dags/ and plugins/. Most of the diff is mechanical import and argument renames.
3. Grep for metadata DB access. Triage each hit as rewrite, delete, or move-out-of-Airflow. Budget days, not hours.
4. Grep for removed context keys and for logical_date used without a None check.
5. Set catchup explicitly on every DAG.
6. Replace SubDAGs with task groups or mapped tasks.
7. Replace sla_miss_callback with real monitoring.
8. Upgrade a staging environment, run every DAG for one logical date with dag.test(), and compare outputs to production for the same date. The local testing setup is what makes step eight take an afternoon instead of a week.
9. Confirm the triggerer is running before you cut over, because deferrable sensors stall silently without it — see deferrable operators.

Where teams get this wrong

Treating it as a version bump. pip install "apache-airflow==3.*" in a requirements file, merged on a Friday. The DAGs parse. The custom operator that reads the metadata DB fails at 2 a.m. on a task nobody watches closely.

Assuming the shims are permanent. Compatibility layers for old import paths exist so you can upgrade incrementally, not so you can skip the work. Fix the imports in the same PR as the upgrade.

Not testing manual triggers. Every scheduled run has a logical_date, so a staging environment that only runs on schedule will never hit the None path. Manually trigger every DAG once.

Upgrading the database without a restore-tested backup. The migration is one-way. A metadata DB migration that half-completes on a large instance is a very long evening.

Forgetting the DAG processor. In Airflow 3 the DAG file processor is a separate component. If your deployment manifest does not run it, DAGs simply do not appear, and the error is a quiet absence rather than a crash.

Leaving catchup implicit. It changed. If you have not written it down per DAG, you are relying on a default that already changed once.

FAQ

Do I have to rewrite my DAGs for the Task SDK?

No, for ordinary DAGs. @dag, @task, and standard operators work the same way with updated imports. The rewrite is only for code that reached into Airflow internals, which is usually a small number of custom operators and plugins.

Can Airflow 2 and Airflow 3 DAGs coexist during migration?

Not in one deployment. What works is running a second Airflow 3 environment against a copy of your DAG repo, migrating DAGs in batches, and cutting over per DAG by pausing on the old instance and unpausing on the new. Plan for the two instances to overlap for a few weeks.

Is the new UI actually better?

For finding a failed task’s log, yes, noticeably. For anything involving many DAG runs at once, it is different rather than strictly better, and there are views the old UI had that took a release or two to reappear. Do not upgrade for the UI; do not avoid upgrading because of it either.

What happens to my existing XComs and history?

The migration preserves DAG run history and XComs. Very old records from removed features (SubDAG runs, for instance) are cleaned up. Take the backup anyway.

Should I move to assets during the upgrade?

No. Do the mechanical migration first, get a stable Airflow 3, then convert cron-scheduled DAGs to asset scheduling as a separate project. Combining them means any scheduling weirdness has two possible causes.

Is it worth upgrading if my DAGs work fine on 2?

Yes, but not urgently unless you need something specific. Airflow 2 security patches will not last forever, DAG versioning is a real operational improvement, and every month you wait adds DAGs written against the old context keys. If you are choosing an orchestrator rather than upgrading one, Airflow vs Dagster vs Prefect is the better starting point.

What this means for your pipelines

Airflow 3’s theme is a hard boundary between your task code and Airflow’s internals. Everything else — the UI, the scheduler-managed backfills, DAG versioning — follows from being able to draw that line cleanly. If your DAGs stayed on the public API, the upgrade is a day of import renames and a careful pass over logical_date. If they did not, the boundary is where your week goes.

The two changes that will actually alter your on-call life are DAG versioning and scheduler-managed backfills. Knowing which code a run executed removes an entire category of incident postmortem guesswork, and a backfill that survives a laptop lid closing removes another. Neither shows up in a feature comparison, and both are worth the migration on their own.

Plan it as a project with a staging environment and a per-DAG cutover list, not as a dependency bump. Get to zero deprecation warnings on Airflow 2 first — that single step converts most of the upgrade from discovery into mechanical work, and mechanical work is the kind you can estimate.