Topic
7 posts ·view all posts
Dagster🟢BeginnerSoftware-defined assets vs Airflow tasks, how partitions change backfills, asset checks versus dbt tests, and why warehouse credentials belong in resources.
Prefect🟢BeginnerA flow is Python. A deployment is what actually runs. Work pools are where. Prefect will call an empty extract a success unless you raise.
Tools🟡IntermediateAn honest comparison of the three major Python orchestrators: where each one shines, where each one hurts, and a simple decision rule.
Kubernetes🟢BeginnerAirflow and Spark can spawn unbounded pods, requests and limits are the difference between busy and evicted, crash loops hide in operators, and MWAA or Databricks is sometimes the right way to not run K8s.
Airflow🟡IntermediatePractical patterns for expand, expand_kwargs, and partial in Airflow: how to size a fan-out, batch it safely, and avoid the mapped tasks that overwhelm your scheduler and metadata database.
Airflow🟡IntermediateHow to move cross-team DAG dependencies from padded cron offsets to dataset-driven triggers, including conditional scheduling, naming rules, and the failure modes nobody warns you about.
AWS🟡IntermediateState transitions versus scheduler hours, execution history versus task logs, and the specific signals that tell you which orchestrator your pipeline has outgrown.
