Dagster and dbt: do you need both?

"Why use dbt if I have Dagster?" is the number-two Google result for this topic, and it's a Reddit thread. That tells you the question is real. People see two tools that both draw a lineage graph, both run on a schedule, and both run tests, and they wonder if they are paying twice.

They aren't, quite. dbt is a SQL transformation framework. Dagster is an orchestrator. They overlap in the middle, and the overlap is where the confusion lives. My own production pipeline runs on Dagster with no dbt at all, so I'll also say when skipping dbt is the right call.

What each tool actually does

dbt takes a folder of SELECT statements with Jinja templating. It works out their order from ref() calls, compiles them for your warehouse, runs them as tables or views, and runs tests (not_null, unique, custom SQL) against the results. It does not schedule itself. Something has to call dbt build.

Dagster decides when things run, records what each run produced, retries failures, tracks partitions, and runs arbitrary Python. It has its own data-quality checks (asset checks) and its own lineage graph, built from the dependencies you declare.

The overlap: both have a lineage graph and both have tests. The difference: dbt's graph covers SQL models in one warehouse. Dagster's graph covers everything, including the Python that loads data before dbt runs and the models you train after it.

Why use dbt if you have Dagster

You still want dbt when:

• Most of your transformations are SQL in one warehouse. dbt's templating, ref(), incremental materializations and the docs site are a decade of work you would otherwise rebuild badly.
• Analysts write the models. dbt is the tool analysts already know. Asking them to write Dagster assets in Python is a bigger change than it sounds.
• You want the project to be portable. A dbt project runs under Dagster, Airflow, Prefect, cron or dbt Cloud. Your SQL is not locked to the orchestrator.

You can skip dbt when the transformations are mostly Python, or when one engineer writes everything and is happy to keep the SQL inside Python assets. That is my situation, below.

How dagster-dbt runs dbt

The dagster-dbt integration reads dbt's manifest.json and creates one Dagster asset per dbt model. The core of it, from the dagster-dbt reference:

from pathlib import Path
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, DbtProject, dbt_assets

my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project"))
my_dbt_project.prepare_if_dev()  # runs `dbt parse` locally to build the manifest

@dbt_assets(manifest=my_dbt_project.manifest_path)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()

Three things to notice:

1. dbt still does the work. Dagster calls the dbt CLI with build. Your models run exactly as they would from a terminal.
2. The manifest is the contract. Dagster builds its graph from the manifest at load time. If the manifest is stale, the graph is stale. In production you build it during the image build, not at runtime.
3. .stream() reports per model. Dagster records each model as its own materialization, so you see which model failed, not just "the dbt task failed".

Newer Dagster versions also offer a DbtProjectComponent that you scaffold with dg scaffold defs dagster_dbt.DbtProjectComponent. It generates the same wiring from a YAML file. Use whichever your Dagster version and team prefer.

Because dbt models are now assets, a Python asset can depend on a dbt model and a dbt model's sources can map to upstream Python assets. That single graph across SQL and Python is the main reason to run dbt under Dagster rather than under cron.

Dagster dbt partitions

Partitioned dbt models work by passing partition bounds as dbt variables. Adapted from the same reference (imports of json and DailyPartitionsDefinition left out):

@dbt_assets(
    manifest=my_dbt_project.manifest_path,
    partitions_def=DailyPartitionsDefinition(start_date="2023-01-01"),
)
def daily_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    start, end = context.partition_time_window
    dbt_vars = {"min_date": start.isoformat(), "max_date": end.isoformat()}
    yield from dbt.cli(["build", "--vars", json.dumps(dbt_vars)], context=context).stream()

Your incremental models then filter on var('min_date') and var('max_date'). For dbt's microbatch strategy, the Dagster docs say to pass --event-time-start and --event-time-end instead.

Before you partition, check that you need to. In my pipeline, one sector-statistics asset rebuilt 1,882 dates, 109,152 rows, in about 45 seconds as a single unpartitioned asset. Doing it per partition would have meant over 2,000 partitions and 5,000 to 20,000 queries. Partitions are the right tool when one date is expensive or fails on its own. When the whole history is cheap to rebuild, one asset is simpler, and the same is true of a dbt model.

dbt tests as asset checks

Since dagster-dbt 0.23.0, dbt tests become Dagster asset checks by default. A not_null test on customers shows up as a check on the customers asset. You can turn it off with DagsterDbtTranslatorSettings(enable_asset_checks=False).

That is useful, but know the order of events. dbt build builds a model, then tests it. A failing test stops downstream models, but the table it tested is already in the warehouse. Anyone querying it directly sees the bad rows.

I hit the same shape of bug in pure Dagster. A blocking=True asset check failed the run, but only after the asset had already written its rows to ClickHouse. "Blocking" stopped the downstream assets. It did not stop the write. The fix was to have the asset return its data frame and let the IO manager do the insert, so the checks run before anything lands. With dbt, the equivalent options are building into a staging schema and swapping, or putting the critical tests on the sources so bad data fails before the model runs.

My pipeline uses Dagster without dbt

My production pipeline has 112 assets and 93 asset checks, and no dbt. dbt is not in its dependencies, and none of its architecture decision records (they run to ADR-0067) mentions it. So I can't point you to a written decision. What I can show you is what the pipeline does instead:

• Transformations are Python. Assets use Polars (and some pandas), with DuckDB in four places to consolidate Parquet files.
• Storage goes through IO managers. 44 of its 49 resources are ClickHouse IO managers. The asset returns a frame; the IO manager writes it.
• Checks are asset checks. The 93 checks are written in Python against the same frames.

Most of this work is reading scraped files, cleaning them, computing features and training models. Very little of it is "SELECT from one warehouse table into another", which is dbt's home ground. For that shape, dbt would add a second graph and a second test system without removing anything.

If your shape is the opposite, mostly SQL in a warehouse with a few Python steps at the edges, dbt under Dagster is a sound design. If you want dbt without a warehouse bill, dbt with DuckDB is a cheap way to run it.

Prefect or Dagster for dbt

Both have maintained dbt integrations. I have used neither Prefect nor its dbt integration in production, so this is from the docs.

prefect-dbt offers a PrefectDbtRunner, which runs dbt build and observes each node as a Prefect task, and a PrefectDbtOrchestrator, where Prefect drives each node for per-node retries and concurrency. It can also trigger dbt Cloud jobs.

The difference is in the model, as in the Dagster vs Airflow comparison. Dagster turns dbt models into assets, so they sit in the same graph as your other tables. Prefect treats dbt nodes as tasks in a flow. If you think in tables, Dagster fits dbt more closely. If you think in flows, Prefect is a reasonable fit. Try both on your own project before deciding.

A short decision checklist

1. Are most transformations SQL in one warehouse? Use dbt, orchestrated by Dagster or anything else.
2. Do analysts write the models? Use dbt.
3. Are most transformations Python, written by engineers? Dagster alone is enough.
4. Do you need partitions? dagster-dbt supports them through --vars, but check whether a full rebuild is cheap first.
5. Is a bad table worse than a late table? Build into a staging schema or test sources, because a failing test does not undo a write.
6. Do you only need dbt on a schedule? You may not need an orchestrator yet. dbt Core vs dbt Cloud covers the scheduler that comes with the managed product.

So: do you need both?

You need both when most of your logic is SQL and you also need Python around it. dagster-dbt makes that a single graph, with dbt tests as checks. You need only Dagster when most of your logic is Python, as mine is. You may need only dbt and a scheduler when everything is SQL and nothing else runs.

If you are also deciding on the SQL layer itself, SQLMesh vs dbt covers the alternative.

Local-First Analytics covers the storage and query side of this stack: DuckDB, Parquet and Arrow on hardware you already have. Chapter 1 is free to read, the rest is on Amazon. Related reading: Dagster vs Airflow, dbt with DuckDB, and dbt Core vs dbt Cloud.

If you are deciding how dbt and an orchestrator should fit together on your platform, a Data Platform Audit is a week and a written roadmap you keep. If your dbt project already runs on a schedule and nobody complains, you probably don't need me.