Airflow

Solution

It backfills every interval since start_date. A two-year daily start_date creates hundreds of runs the moment you unpause. Use catchup=False until you mean it, then a targeted backfill.

Solution

Idempotency: delete-then-insert for the partition, or MERGE on a business key. Passing a 50 MB dataframe through XCom is the other fail — XCom is for small metadata (paths, watermarks), not data.

Solution

A poking sensor holds a worker slot. At scale use deferrable sensors / triggers, or an external sensor with mode=reschedule. “Wait for the file every minute with a Sensor” is how a 4-worker prod queue dies at 9:00.

Solution

In 2.x, a 00:00 DAG run for 2026-08-28 covers [28, 29). The partition you load is usually data_interval_start (the logical date). Using “today” or pendulum.now() inside the task is how you reload the wrong day after a retry.

Solution

TaskFlow is fine for Python that stays in the worker. Use a dedicated Operator (SnowflakeOperator, DatabricksSubmit) when the work should run in the warehouse, not on the Airflow box. The tell: if the function downloads a table into pandas, you already lost.

Solution

Give expensive warehouse tasks a small pool. Separate extract vs transform pools. Do not let 80 backfills starve the 8:00 SLA DAG. priority_weight is a hint; pools are the hard cap. Say this as an on-call story, not a docs recitation.

Solution

Map over the list of tables/dates at runtime. Do not map 50,000 tiny tasks if the warehouse can loop cheaper. Mapping is organization plus fan-out, not a new executor.

Solution

The scheduler parse loop. Keep DAG files cheap. I/O belongs in tasks.

Solution

Airflow 2.7+ teardown runs even when the work fails. Use it for ephemeral clusters and temp schemas. Hope is not a task.

Solution

A UI backfill or a DAG param. Changing start_date to now() is how you lose history.

Solution

Batch the map (chunks of files, not one task per file), raise max_map_length only with a story, and push heavy work to Spark/dbt. Dynamic mapping is not a for-loop over S3.

Solution

XCom is for tiny handles (S3 URI, partition, row count), not dataframes. Put the frame in object storage. Postgres XCom tables are not a data lake.

Solution

Worker slots. A poking sensor holds a slot for the whole wait. Deferrable releases the worker and waits in the triggerer. Still need a timeout and an SLA miss callback.

Solution

Parse-time I/O. Top-level queries fail or hang, and every parse hits the warehouse. Build the list in a task, use a catalog, or static YAML.

Solution

Every interval between start_date and now when the DAG was paused or deployed late. That is fine for “from now on” jobs; it is a hole for daily finance. Say it in the design review.

Solution

A separate backfill DAG with a low pool/priority, partitions in batches, and a freeze on overlapping writes to gold. Same DAG with a huge start_date is how Monday pages.