Prefect Deployments: Flows, Work Pools, and the Green Run With Zero Rows

The orders flow was green for fourteen nights. Prefect Cloud showed fourteen green run rows, median duration 41 seconds, zero retries. The warehouse table for those fourteen days had zero rows. The S3 prefix had moved from s3://lake/orders/dt= to s3://lake/landing/orders/dt= on a Friday. list_objects returned []. The task returned []. The flow returned. Prefect did its job: nothing raised.

I found it because finance asked why Monday’s revenue was exactly equal to the Friday before the cutover, which is what a BI cache does when the fact table stops growing. The orchestrator never pages on “succeeded at doing nothing.”

flowchart TD
  code[Flow in git] --> dep[Deployment]
  dep --> sched[Schedule and parameters]
  dep --> pool[Work pool]
  pool --> worker[Worker in your VPC]
  worker --> run[Flow run]
  run -->|raises| red[Failed]
  run -->|returns| green[Succeeded]
  green --> rows{Row count?}
  rows -->|zero, no raise| silent[Silent hole]
  rows -->|asserted| ok[Safe to publish]

Cloud stores the run. The worker runs the code. Success is “did not raise,” not “wrote today’s partition.”

Three nouns, and only one of them runs

A flow is a Python function with @flow. It is what you pytest. It is not what production invokes.

A deployment is a versioned binding: which flow revision, which parameters, which schedule or event trigger, which work pool. Change the cron without a new deployment and you changed a laptop habit.

A work pool is the place work is allowed to run — process, Docker, Kubernetes. Workers poll the pool. No worker, no run, and the UI will sit in Scheduled until someone notices.

from datetime import date

from prefect import flow, task


@task(retries=3, retry_delay_seconds=30)
def extract_orders(day: date) -> list[dict]:
    rows = list_parquet(f"s3://lake/landing/orders/dt={day.isoformat()}/")
    return rows


@task
def load_orders(rows: list[dict], day: date) -> int:
    n = merge_day("analytics.fct_orders", rows, day)
    return n


@flow(name="orders-daily")
def orders_daily(day: date | None = None) -> int:
    day = day or date.today()
    rows = extract_orders(day)
    if len(rows) == 0:
        raise RuntimeError(f"orders extract empty for {day.isoformat()}")
    n = load_orders(rows, day)
    if n == 0:
        raise RuntimeError(f"orders merge wrote 0 rows for {day.isoformat()}")
    return n

FailedRun is the page. A comment in Slack is not. I want this raise in the flow, not in a warehouse test twenty minutes later.

Deploy it to a pool. flow.serve is for a laptop or a single always-on process. Production is a worker watching a pool:

# deploy.py — run in CI against the Prefect API
from prefect.client.schemas.schedules import CronSchedule
from prefect.runner.storage import GitRepository

if __name__ == "__main__":
    orders_daily.deploy(
        name="orders-daily-prod",
        work_pool_name="k8s-prod",
        schedules=[CronSchedule(cron="15 6 * * *", timezone="UTC")],
        parameters={},
        source=GitRepository(url="https://git.example.com/data/orders.git"),
        entrypoint="flows/orders.py:orders_daily",
    )

If the worker’s image does not pin the same Prefect and dependency versions as CI, you will debug “it works in Cloud’s syntax check” at 06:20. Pin the image. The deployment points at a git SHA, not at main.

Cloud vs self-host

Prefect Cloud is the control plane: API, UI, schedules, notifications. Workers stay in your network and pull runs. Source data does not have to leave. Flow-run metadata does — names, parameters, logs you chose to emit, state.

Self-host the Prefect server when legal or network policy forbids that metadata hop. You then own auth, upgrades, and the Postgres under the API. That is a real on-call, not a checkbox.

Do not self-host because Cloud “might get expensive.” The compute bill is the workers you already run. Cloud is the pager and the audit trail. Self-host because you must, not because a blog said OSS is free.

When Prefect is enough

Enough: a team that already writes Python, fewer than a few dozen flows, dynamic parameters, event-driven runs, no need for a first-class asset graph. The gap from script to retries is the smallest of the three orchestrators I operate.

Not enough: you need software-defined assets, you already have MWAA approved, or you are hiring a bench that only knows Airflow. Those are the reasons in Airflow vs Dagster vs Prefect, and they are still the reasons. Switching orchestrators costs more than any missing decorator.

Prefect will not notice an empty partition. dbt still might, if you run it — see dbt incremental models for the merge grain the flow should have written. The flow must raise first. The warehouse test is the second line.

Failure modes

Green empty extract. The default success condition is “no exception.” Assert row counts before you return.

No worker on the pool. Deployments schedule. Nothing runs. Alert on Scheduled older than two intervals.

Parameters only in the UI. Tuesday’s ad-hoc run used day=2026-08-01 and nobody can replay it. Put the default in git. Log the resolved parameters on every run.

One giant flow. Prefects dynamic tasks are a feature until the run graph is a novel. Split at publish boundaries.

Cloud as the runtime. The API does not execute your SQL. If the worker dies, Cloud is a scoreboard.

FAQ

Flow vs deployment? The flow is the Python. The deployment is schedule, parameters, and work pool. Without a deployment, you have a script you run by hand.

Do I need Prefect Cloud? No. Self-host the API if metadata cannot leave the network. You still need a worker next to the data.

How is this different from Airflow? Lighter Python, no DAG file parser in the same way. No asset catalog unless you add one. See Airflow vs Dagster vs Prefect. Hub: Prefect.

Why was the run green with zero rows? Nothing raised. Assert counts (and a watermark) before the flow returns.

Workers are the runtime. Cloud is the scoreboard.

I run one worker per work pool, in the VPC that can reach the warehouse and the bucket. The worker image is pinned — same rule as Docker pinned images. Cloud (or a self-hosted API) stores flow-run state. If the worker process dies, scheduled runs sit in Scheduled until someone pages. I alert on that state older than two intervals, not only on Failed.

Deploy from CI at a git SHA. A laptop prefect deploy is how Tuesday’s parameter set never made it to git. Log the resolved parameters on every run so an ad-hoc day= is replayable.

Hub: Prefect. If you need assets, read Dagster assets before you rebuild an asset catalog inside Prefect.

What to do Monday

Add the zero-row raise to every extract that can return []. Put a worker on a named pool. Deploy from CI at a SHA. Page on failed runs and on scheduled-but-not-started. If that is the whole platform, Prefect is enough. If you are about to rebuild Airflow inside Prefect because you missed assets, stop and pick the other tool on purpose.