Airflow Best Practices cheat sheet
DAG design, idempotency, scheduling, resource control, and monitoring practices for Airflow you can be on call for.
Idempotency first
DELETE by partition then INSERT, or MERGE by key- The golden rule. A task rerun must produce the same result, or every retry and backfill is a data risk.
Use data_interval_start, never datetime.now()- A task using the wall clock produces different output on rerun, which makes backfills meaningless.
Write to a deterministic path per interval- s3://lake/bronze/orders/dt={{ ds }}/. Reruns overwrite the same prefix instead of appending duplicates.
Make external calls safe to repeat- Idempotency keys on API writes. Retries are guaranteed to happen, so design for them from the start.
DAG design
catchup=False on every new DAG- Otherwise enabling the DAG schedules a run for every interval since start_date. The classic first-day incident.
max_active_runs=1 for non-idempotent or stateful pipelines- Prevents two runs writing the same target concurrently, which no amount of retry logic can fix afterward.
No top-level code that hits a network or database- Module top level is parsed every few seconds by the scheduler. An API call there will take the scheduler down.
Keep DAG files small; put logic in importable modules- Business logic in a package is unit-testable; the same logic inside a PythonOperator body is not.
One DAG per business outcome- A 300-task DAG is unoperable. Split by outcome and connect with Datasets or an external task sensor.
Scheduling
A run stamped Aug 1 executes after Aug 1 ends- Airflow labels runs by the interval they process. Once internalized, this is a feature rather than a surprise.
Datasets over cron guesswork- Trigger on upstream production instead of scheduling downstream fifteen minutes later and hoping.
Stagger schedules across DAGs- Everything at 0 0 * * * creates a scheduler thundering herd. Spread start times deliberately.
Avoid schedule=None DAGs that someone must remember to trigger- Undocumented manual DAGs are the ones forgotten during an incident.
Reliability
default_args with retries=2 and retry_delay- Set at DAG level, override per task. A task with no retries will page you for a transient network blip.
execution_timeout on every long task- Without it, a hung task holds a worker slot indefinitely and blocks the whole pipeline silently.
Deferrable operators instead of poking sensors- A sensor waiting six hours occupies a worker slot the whole time. Deferrable operators release it.
Pools for shared resources- Cap concurrent warehouse connections with a pool so Airflow cannot saturate the database it depends on.
sla_miss_callback or explicit freshness checks- Success is not the same as on time. Alert on lateness separately from failure.
Configuration and secrets
Connections and Variables, never hardcoded credentials- Use a secrets backend (Secrets Manager, Vault) so credentials never live in the metadata database or code.
One Variable.get per task, not per module- Top-level Variable.get runs on every parse and hammers the metadata database.
Environment-specific config via env vars, not branches- The same DAG code should run in dev and prod. Divergent branches drift and break on promotion.
Testing and monitoring
python dags/my_dag.py- The cheapest smoke test. Import errors surface immediately instead of as a missing DAG in the UI.
airflow dags test my_dag 2026-08-01- Executes a full DAG locally for one date without a scheduler. The main pre-merge check.
A CI test asserting no import errors across all DAGs- One broken import can hide a DAG from the UI entirely. Catch it in CI, not from a user report.
on_failure_callback wired to a real channel- Alerts nobody watches are not monitoring. Route to the on-call channel with the log URL included.
Monitor scheduler DAG parse time- Rising parse time is the leading indicator of a scheduler problem, long before tasks visibly queue up.
From DataLane — tutorials at/blog, practice SQL live in theplayground.