Prefect vs Dagster: which tier does your pipeline actually need?

Most "Prefect vs Dagster" comparisons skip the first question: do you need either? The vendors' answer is yes. The honest answer, for most analytics work, is no. A daily job that runs a script, writes a Parquet file and finishes in ten minutes is well served by cron, a Makefile and a log file. That is the opposite of what an orchestrator's landing page tells you.

Chapter 8 of my book, Local-First Analytics, is built around that idea. It treats orchestration as four tiers: Make and cron, then Prefect, then Dagster, then Temporal. You move up a tier only when a specific pain forces you to. This post walks through that progression, the decision matrix for which tier a job needs, and what has changed since the chapter was written. I run Dagster in production. I have not run Prefect or Temporal in production, so those parts come from their documentation and from running Prefect locally for this post.

The four tiers, and what each one adds

Each tier adds one capability the tier below lacks:

1. Make + cron. Make runs a step only when its output is missing or older than its inputs. Cron runs it on a clock. Zero installs, plain text, version-controlled.
2. Prefect. Adds retries with backoff, result caching, parallel tasks and a UI with run history. A flow is a decorated Python function.
3. Dagster. Adds the asset: the orchestrator knows which tables and partitions exist, when each was built, and whether its checks passed.
4. Temporal. Adds durable execution: a workflow survives a crash and resumes at the step it was on, and it can wait days for a signal.

The chapter's estimate of where projects land: about 80% stay on Make and cron, 15% need Prefect, 4% Dagster and 1% Temporal. Those numbers are a judgement, not a survey. But they point the right way, and they explain why I rarely recommend starting with a server.

When cron is enough

Cron is enough when four things are true:

• the pipeline runs daily or less often;
• each step finishes in minutes, not hours;
• failures are rare and obvious when they happen;
• you are not calling a rate-limited API that fails at random.

The cron line itself is one line. The part people skip is the wrapper: set -euo pipefail, a timestamped log file per run, and a message to Slack or email on a non-zero exit. With that wrapper, cron tells you when something broke and the log tells you why.

The chapter lists five signs you have outgrown it:

1. Flaky dependencies. An API fails at random and you need retries with backoff.
2. Long pipelines. Steps take over 30 minutes and you want to resume from the failed step, not from the start.
3. Parallel work. Independent steps could run at the same time.
4. State across runs. You need to know "did this row or this date already process?"
5. Observability. tail -f logs/* no longer tells you what happened.

The chapter's own counter-claim matters here. The 80% figure holds for one person. On a team, Make's terse rules become a cost: if two engineers are writing Make recipes for each other to read, move up a tier even if nothing is broken.

Prefect vs Dagster: steps or tables

Prefect describes work. You write a flow, call tasks inside it, and Prefect records each run, retries failed tasks and caches results. Dagster describes data. You declare assets (a table, a file, a model) and the assets they depend on, and Dagster builds the graph from those dependencies.

That difference used to be clean. It is less clean now. Prefect 3 has assets through a @materialize decorator, which its docs describe as a task decorator that tracks asset creation, with lineage and asset-level health. I checked that the decorator is in Prefect 3.8.7. It is still a task that records what it wrote, rather than the unit the whole tool is built around, but "Prefect has no idea what data exists" is out of date.

The book's counter-claim is the one I'd stand behind: the choice is partly aesthetic. Do you want to describe pipelines or data products? Plenty of teams stop at Prefect and never feel the lack. If your team prototypes faster in Prefect, that is a real gain. Don't switch because a decision tree said so.

What a Prefect flow looks like when you run it

I installed Prefect 3.8.7 (released 27 September 2026, per PyPI) and DuckDB in a fresh virtual environment and ran this flow. The extract step fails twice on purpose, like a rate-limited API, and Prefect retries it:

from prefect import flow, task
import duckdb

calls = {"n": 0}

@task(retries=3, retry_delay_seconds=1)
def extract() -> list[tuple[str, int]]:
    calls["n"] += 1
    if calls["n"] < 3:  # fail twice, like a rate-limited API
        raise RuntimeError(f"HTTP 429 on attempt {calls['n']}")
    return [("2026-10-20", 41), ("2026-10-20", 7), ("2026-10-21", 12)]

@task
def summarise(rows: list[tuple[str, int]]) -> list[tuple]:
    con = duckdb.connect()
    con.execute("CREATE TABLE events(day DATE, n INT)")
    con.executemany("INSERT INTO events VALUES (?, ?)", rows)
    return con.sql("SELECT day, sum(n) FROM events GROUP BY day ORDER BY day").fetchall()

@flow(log_prints=True)
def daily_metrics():
    print(summarise(extract()))

if __name__ == "__main__":
    daily_metrics()

The output, trimmed to the log lines:

INFO | prefect - Starting temporary server on http://127.0.0.1:8399
INFO | Flow run 'unyielding-caribou' - Beginning flow run 'unyielding-caribou' for flow 'daily-metrics'
INFO | Task run 'extract-860' - Task run failed with exception: RuntimeError('HTTP 429 on attempt 1') - Retry 1/3 will start 1 second(s) from now
INFO | Task run 'extract-860' - Task run failed with exception: RuntimeError('HTTP 429 on attempt 2') - Retry 2/3 will start 1 second(s) from now
INFO | Task run 'extract-860' - Finished in state Completed()
INFO | Task run 'summarise-3ca' - Finished in state Completed()
INFO | Flow run 'unyielding-caribou' - [(datetime.date(2026, 10, 20), 48), (datetime.date(2026, 10, 21), 12)]
INFO | Flow run 'unyielding-caribou' - Finished in state Completed()
INFO | prefect - Stopping temporary server on http://127.0.0.1:8399

Three things I noticed:

• No server to set up. With no server configured, Prefect starts a temporary one, backed by SQLite, and stops it at the end. The whole run took 7.3 seconds of wall time, about 2 of them starting the server and 2 waiting between retries.
• The retry is the whole pitch. Two lines of decorator replace the backoff loop you would otherwise write in your cron script.
• The first run was noisy. On the very first run, the temporary server logged a database is locked traceback from its own telemetry heartbeat. The flow still completed. With PREFECT_SERVER_ANALYTICS_ENABLED=false the second run was clean. Harmless, but it is the kind of log line that wastes a morning if you don't know it.

To schedule it, daily_metrics.serve(name="daily", cron="0 6 * * *") keeps a process running that launches runs on that schedule. For runs on Docker or Kubernetes, you use flow.deploy() with a work pool, and a worker polls the pool for runs. Or you keep system cron and call the script: a Prefect flow is still a Python script, so cron plus Prefect's retries is a legitimate middle tier.

Where Dagster earned its place for me

I run a production Dagster pipeline alone: 112 assets, 57 schedules and 93 asset checks, self-hosted on a homelab Docker Swarm with Postgres for run history. I covered why the asset model fit, and what still hurt, in Dagster vs Airflow, so I won't repeat it.

The short version, in the chapter's terms: my pipeline crossed every line on the "outgrown cron" list. 34 daily-partitioned assets that fail one date at a time. Checks that must sit next to the tables they guard. Steps that must wait for their inputs, not for a clock. That last one cost me a week of nightly failures before I declared the real dependency instead of a schedule time. Tables were the product, so an asset-first tool fitted.

Would Prefect have worked? Probably, with more of the partition bookkeeping written by me. That is the honest form of this comparison: both tools can do the job, and the difference is which bookkeeping you get for free.

Prefect vs Temporal: when durability is the point

Temporal is a different kind of tool. It records every step of a workflow as an event, so when a worker dies, another one replays the history and carries on from the step it was on. A workflow can sleep for days, or wait for a person to approve something.

The chapter's test is blunt: if you can afford to restart from the beginning, you don't need Temporal. A daily ETL job that finishes in 15 minutes can simply run again.

Then it argues against itself, and the counter-claim is the useful part. "Restart from scratch" is cheap until each retry costs something real: a paid API call, an email already sent, a payment already started, or a source system that is only available in one batch window. Then durability matters even for a workflow that takes two minutes. If that describes your job, don't wait for hour-long pipelines before you consider Temporal.

Temporal's server and Python SDK are MIT-licensed.

The decision matrix

This is the chapter's matrix, condensed, with my corrections marked:

	Make + cron	Prefect	Dagster	Temporal
Setup	None	pip install	pip install	Server + workers
Retries with backoff	Write your own	Yes	Yes	Yes
Parallel steps	make -j only	Yes	Yes	Yes
Knows what data exists	No	Partly (@materialize, newer)	Yes, the core idea	No
Survives a crash mid-step	No	No	No	Yes
Waits days for a human	No	No	No	Yes
Web UI	No	Yes (book omits this)	Yes (book omits this)	Yes
Learning time (book's estimate)	1 day	3 days	1 week	2 weeks

And the decision tree, in words:

1. Fewer than about five dependent steps? Make and cron.
2. Do steps fail at random? If not, still Make and cron.
3. Is the product a set of tables you need tracked, checked and partitioned? If not, Prefect.
4. Must a workflow survive crashes, or wait hours or days? If not, Dagster. If so, Temporal.

The book's matrix has a "Visual UI" row ticked only for the n8n column. That undersells Prefect and Dagster: both ship a web UI with run history in the open-source package. Read that row as "visual workflow builder", which is what n8n adds.

Is Prefect free?

Yes, for the open-source package. Prefect is licensed under Apache 2.0, the same as Dagster. The self-hosted server includes the UI and uses SQLite by default; its docs recommend PostgreSQL for production. Most automations work self-hosted; the email-notification and declare-incident actions are Prefect Cloud only.

Prefect Cloud, checked at prefect.io/pricing on 2 October 2026:

• Hobby: free. 2 users, 1 workspace, 5 deployments, 7 days of run retention.
• Starter: $100 a month. 3 users, 20 deployments, 7 days of retention.
• Team: $100 per user a month, for 4 to 8 users. 100 deployments, 14 days of retention.
• Enterprise: custom pricing.

For comparison, the Dagster+ prices I checked on 1 October 2026 were $10 a month for Solo and $100 a month for Starter, each plus per-credit charges. The pricing shapes differ: Prefect charges by seats and deployments, Dagster+ by seats plus materializations. Count your own runs before you compare.

What has changed since the chapter was written

The chapter assumed Prefect 3.7, Dagster 1.13 and Temporal's Python SDK 1.27, as of May 2026. Checked again on 2 October 2026:

• Prefect is at 3.8.7. 3.8.0 (July 2026) made the new V2 UI the default for self-hosted servers. The chapter's 3.x patterns, flow.serve(), workers and work pools, and the INPUTS cache policy, all still work; I checked each one in 3.8.7.
• Prefect Cloud's pricing page now lists prices. The chapter says it shows no dollar amounts and lists a Pro tier. Today it shows Hobby, Starter, Team and Enterprise, with the prices above.
• Prefect has assets. The chapter's "Prefect thinks in tasks, Dagster thinks in assets" is still true at the core, but @materialize narrows it.
• prefect-duckdb is not worth installing. The chapter's install line includes it. On PyPI its only release is an alpha, 0.1.0a1, from May 2024. You don't need it: call duckdb inside a task, as the example above does.
• Dagster is at 1.13.25; Temporal's Python SDK is at 1.34.0 (30 September 2026). Neither changes the decision.

A checklist before you install anything

1. Write down the last three times your pipeline failed and what fixed each one.
2. If the fix was "run it again", add retries to your cron wrapper first.
3. If the fix was "rerun one date", you want partitions: look at Dagster.
4. If the fix was "find out which step failed", you want a run history: Prefect is the smallest step.
5. If a rerun sent something twice or paid for something twice, look at Temporal.
6. If you can't name three failures, stay on cron.

So: Prefect or Dagster?

For most analytics work, neither yet. Cron, a Makefile and a wrapper that logs and alerts will carry a daily pipeline further than the vendors suggest. When steps start failing at random, Prefect is the smallest move up: decorators on the functions you already have. When the product is a set of tables with partitions and checks, Dagster is the better fit, and it is what I run. Temporal is for work where a retry has a real-world cost.

Pick the tier for the pain you have today. Moving up later is cheaper than running a tier you don't need.

This post is from chapter 8 of Local-First Analytics, "Orchestration Progression": the same Makefile pipeline built three ways (Make, Prefect and Dagster), the Temporal and n8n tier, and the full decision matrix. Chapter 1 is free to read; the rest is on Amazon. Related reading: Dagster vs Airflow, Dagster and dbt, and is Dagster free?.

If you are choosing an orchestrator and want a second opinion on your workload, a Data Platform Audit is a week and a written roadmap you keep. If your pipeline runs on cron and works, you probably don't need me, and I'd rather say so here.