All stacks · Orchestration
Workflow orchestration: DAGs, scheduling, and operations.
AirflowHow the triggerer frees workers: why classic sensors sleep on a slot, FileSensor vs deferrable waits, and the operational habits that keep deferred tasks from sitting forever.
AirflowExtract from an API, transform with pandas, load to a warehouse — scheduled in Airflow with retries, idempotency, and a local Docker setup.
AirflowA four-layer testing setup for Airflow: parse checks, DAG structure tests, task unit tests with mocked hooks, and dag.test() end-to-end runs against local infrastructure.
AirflowPractical patterns for expand, expand_kwargs, and partial in Airflow: how to size a fan-out, batch it safely, and avoid the mapped tasks that overwhelm your scheduler and metadata database.
AirflowHow to move cross-team DAG dependencies from padded cron offsets to dataset-driven triggers, including conditional scheduling, naming rules, and the failure modes nobody warns you about.
AirflowThe Airflow 3 changes that break real DAGs: the Task SDK boundary, logical_date becoming optional, removed context keys, DAG versioning, and a pre-upgrade checklist.
DagsterSoftware-defined assets vs Airflow tasks, how partitions change backfills, asset checks versus dbt tests, and why warehouse credentials belong in resources.
SnowflakeCREATE STREAM on a table, consume it in a Task MERGE, SHOW STREAMS, stale offsets, and the cases where Airflow still orchestrates dbt and everything outside Snowflake.
ToolsAn honest comparison of the three major Python orchestrators: where each one shines, where each one hurts, and a simple decision rule.
KubernetesAirflow and Spark can spawn unbounded pods, requests and limits are the difference between busy and evicted, crash loops hide in operators, and MWAA or Databricks is sometimes the right way to not run K8s.
AWSState transitions versus scheduler hours, execution history versus task logs, and the specific signals that tell you which orchestrator your pipeline has outgrown.
Airflow remains the default orchestrator of the data world: DAGs as Python code, a mature scheduler, and an ecosystem of providers for every system you will ever touch. It is also famously easy to run badly — non-idempotent tasks, surprise backfills, and datetime.now() bugs have burned every team at least once.
The articles here focus on operating Airflow well: TaskFlow-first DAG design, the data interval model that everyone misunderstands at first, backfills that do not double-load, deferrable operators that stop burning worker slots, and the settings that matter before an on-call rotation depends on your scheduler.
Yes. Airflow has the largest install base by far, so it is the orchestrator you will most likely inherit at work. Dagster and Prefect have real design advantages — we compare them honestly — but Airflow experience remains the most transferable orchestration skill.
Airflow runs are stamped with the data interval they process, and a run executes after its interval closes. A daily run labeled Aug 1 executes at midnight on Aug 2, processing Aug 1 data. Use data_interval_start in your logic instead of the wall clock and this becomes a feature.
Idempotent tasks. Every task should produce the same result if run twice — DELETE+INSERT by partition or MERGE by key instead of blind INSERT. Once tasks are idempotent, retries, reruns, and backfills all become boring.
Managed (MWAA, Cloud Composer, Astronomer) unless you have platform engineers to spare. Self-hosting means owning the metadata database, scheduler tuning, and upgrades — undifferentiated toil for most teams.
Set catchup=False on the DAG. With catchup=True (the historical default behavior), Airflow schedules a run for every interval between start_date and now the moment you enable the DAG.
One email a week with our latest tutorials. No spam.
Newsletter signup is not live yet. Use the contact form if you want to be notified.