Topic
7 posts ·view all posts
DagsterSoftware-defined assets vs Airflow tasks, how partitions change backfills, asset checks versus dbt tests, and why warehouse credentials belong in resources.
PrefectA flow is Python. A deployment is what actually runs. Work pools are where. Prefect will call an empty extract a success unless you raise.
ToolsAn honest comparison of the three major Python orchestrators: where each one shines, where each one hurts, and a simple decision rule.
KubernetesAirflow and Spark can spawn unbounded pods, requests and limits are the difference between busy and evicted, crash loops hide in operators, and MWAA or Databricks is sometimes the right way to not run K8s.
AirflowPractical patterns for expand, expand_kwargs, and partial in Airflow: how to size a fan-out, batch it safely, and avoid the mapped tasks that overwhelm your scheduler and metadata database.
AirflowHow to move cross-team DAG dependencies from padded cron offsets to dataset-driven triggers, including conditional scheduling, naming rules, and the failure modes nobody warns you about.
AWSState transitions versus scheduler hours, execution history versus task logs, and the specific signals that tell you which orchestrator your pipeline has outgrown.