Airflow

Solution

It backfills every interval since start_date. A two-year daily start_date creates hundreds of runs the moment you unpause. Use catchup=False until you mean it, then a targeted backfill.

Solution

Idempotency: delete-then-insert for the partition, or MERGE on a business key. Passing a 50 MB dataframe through XCom is the other fail — XCom is for small metadata (paths, watermarks), not data.

Solution

A poking sensor holds a worker slot. At scale use deferrable sensors / triggers, or an external sensor with mode=reschedule. “Wait for the file every minute with a Sensor” is how a 4-worker prod queue dies at 9:00.

Solution

In 2.x, a 00:00 DAG run for 2026-08-28 covers [28, 29). The partition you load is usually data_interval_start (the logical date). Using “today” or pendulum.now() inside the task is how you reload the wrong day after a retry.

Solution

TaskFlow is fine for Python that stays in the worker. Use a dedicated Operator (SnowflakeOperator, DatabricksSubmit) when the work should run in the warehouse, not on the Airflow box. The tell: if the function downloads a table into pandas, you already lost.

Solution

Give expensive warehouse tasks a small pool. Separate extract vs transform pools. Do not let 80 backfills starve the 8:00 SLA DAG. priority_weight is a hint; pools are the hard cap. Say this as an on-call story, not a docs recitation.

Solution

Map over the list of tables/dates at runtime. Do not map 50,000 tiny tasks if the warehouse can loop cheaper. Mapping is organization plus fan-out, not a new executor.

Solution

The scheduler parse loop. Keep DAG files cheap. I/O belongs in tasks.

Solution

Airflow 2.7+ teardown runs even when the work fails. Use it for ephemeral clusters and temp schemas. Hope is not a task.

Solution

A UI backfill or a DAG param. Changing start_date to now() is how you lose history.