Is DuckDB faster than Spark?

On data that fits one machine, usually yes, because DuckDB runs inside your process with no cluster to schedule, no executors to start and no shuffle over a network. Spark spends part of every job coordinating work across machines, and that cost buys nothing when one machine could hold the whole working set. Once the data is far larger than one machine can hold, the comparison stops making sense: DuckDB does not spread a query across machines at all, so Spark is the one that can run the job. Measure on your own data before you decide; published benchmarks rarely match your query shape.

Can DuckDB replace Spark?

For batch jobs whose working set fits one large machine, often yes. Most Spark jobs read Parquet, filter, join, aggregate and write Parquet, and DuckDB does all of that in SQL over the same files. It cannot replace Spark for work that is genuinely larger than one machine, for long jobs that need to survive a lost worker, or for Structured Streaming. DuckDB also has a PySpark-style Python API, but its own documentation calls it experimental with features still missing, so plan on rewriting jobs in SQL rather than running PySpark code unchanged.

When do you need Spark instead of DuckDB?

When the bytes one job must hold at once outgrow the biggest machine you can rent, when a job runs for hours and must survive a worker dying halfway, or when many teams already share a Spark platform with its catalog and permissions. Total data size is not the test. A 50 TB lake whose jobs each touch one day of data is a single-node workload. A 2 TB table that must be joined to itself on a high-cardinality key may not be.

Which workloads suit DuckDB over PySpark?

Batch transforms over Parquet where one job reads gigabytes to a few hundred gigabytes after pruning; scheduled aggregations and reports; CI tests that need the real engine in seconds; ad-hoc analysis on a laptop or one server; and jobs that a team runs on a cluster only because the cluster already existed. Aggregation-heavy jobs fit best. DuckDB's tuning guide budgets about 1 to 2 GB of memory per thread for aggregations and 3 to 4 GB for joins, so join-heavy jobs need a bigger box for the same data.

Should I use DuckDB or Spark for small data?

DuckDB. Under a few hundred gigabytes of working set, a cluster adds start-up time, a driver, executors, network shuffles and a bill, and buys nothing, because one machine could have held the data. Spark's own local mode runs on one machine, but it still carries the JVM and the scheduler. DuckDB is a pip install, reads Parquet directly and runs in the process you already have. If the data grows past one machine later, Parquet files move to Spark unchanged.

DuckDB vs Spark: which workloads actually need a cluster

Most "DuckDB vs Spark" questions are really a sizing question. Spark is a distributed engine: it splits a job across many machines and pays to coordinate them. DuckDB is an in-process engine: it runs the whole job on one machine and pays nothing to coordinate. Which one is faster depends almost entirely on whether one machine can hold the work.

So the useful question is not which engine is better. It is: how many bytes does your job actually touch, and does that fit on one box? Here is the arithmetic, where a cluster earns its cost, and what one node has handled for me. I have not run Spark in production; this is the single-node side of the line, measured, plus what Spark's own documentation says about the other side.

DuckDB vs Spark: what each one is built for

Spark spreads work over a cluster. A driver plans the job, executors on many machines each process part of the data, and a shuffle moves rows between machines when a join or group-by needs them together. If a machine dies, Spark rebuilds the lost piece. Its documentation puts it plainly: if any partition of a cached dataset is lost, "it will automatically be recomputed using the transformations that originally created it." That is what you rent a cluster for: data larger than any one machine, and jobs that keep going when one machine fails.

DuckDB is a columnar SQL engine that runs inside your Python process, a CLI or a service. No driver, no executors, no shuffle over a network. It reads Parquet and CSV in place, uses every core, and spills to disk when a join or sort outgrows memory. The tuning guide lists GROUP BY, JOIN, ORDER BY and window functions as operators that can work larger than memory.

Both read the same Parquet. That matters more than any benchmark: the choice is reversible, because the files don't care which engine opens them.

The sizing math that decides it

The number that matters is the working set: the bytes a job has to process after partition pruning and column selection. It is not the size of the lake.

Three steps get you there.

1. Start from what the job reads, not what exists. A 20 TB events table, partitioned by day, queried for the last 30 days, touching 6 of 60 columns, reads roughly 30/365 × 6/60 of the table. That is about 160 GB on disk before any filter on row values. Parquet compresses, so memory use runs higher than disk size. But most of that work streams through and never has to sit in memory at once.

2. Budget memory by operation. DuckDB's environment guide gives the rule of thumb: about 1–2 GB of memory per thread for aggregation-heavy work and 3–4 GB per thread for join-heavy work. So a 32-thread job needs roughly 32–64 GB for aggregations, or 96–128 GB for joins. That is one ordinary server, not a cluster.

3. Compare with the biggest box you can rent. Single machines are large now. AWS lists an x2iedn.metal with 128 vCPUs and 4,096 GiB of memory. You don't need that. The point is that the ceiling of "one machine" sits far above where most teams assume it is.

Local-First Analytics draws the same line. Chapter 5 suggests DuckDB for up to about 1 TB of partitioned Parquet for aggregation-heavy work, and less for join-heavy work. It also warns that many-to-many joins blow up intermediate state whatever the input size. Its rule of thumb for starting local-first is a working set that fits in 32 GB of RAM and fewer than ten people running queries. Both bars are much lower than where a single node actually gives out.

If your working set after this arithmetic is in the gigabytes or low hundreds of gigabytes, you are running Spark for the cluster's sake, not the data's.

What one node has handled for me

I run a self-hosted Dagster pipeline for Indian market data, written up in full here. It has grown since that post: 112 assets today. The code location that runs every asset is one container limited to 4 CPUs and 16 GB of RAM. The warehouse is a single ClickHouse server, and DuckDB runs in-process inside the assets. There is no compute cluster anywhere in it.

Two numbers from it:

A full rebuild beats a partition fan-out. One sector-statistics table in ClickHouse, rebuilt as a single unpartitioned job, processes 1,882 trading dates into 109,152 rows in about 45 seconds. Done the "distributed-shaped" way, one partition per date, the same work meant over 2,000 partitions and 5,000–20,000 small queries. When I moved ten similar tables to single full rebuilds, nine of them materialised in about six minutes on the first pass.
DuckDB consolidation is seconds, not minutes. Four assets use duckdb.connect(":memory:") with read_parquet(...) to merge hundreds of raw Parquet files into one table. The commit notes record one of them doing a single-pass FULL OUTER JOIN over about 650,000 rows in about 7 seconds.

The lesson from the first one applies directly to Spark. Splitting work into many small units has a cost. On a cluster, the units are tasks and the cost is scheduling and shuffle. On my pipeline, the units were partitions and the cost was thousands of small queries. If the whole job fits in one machine's memory, the cheapest plan is usually one pass on that machine.

My data is small by Spark standards. That is the point, not a disclaimer. Most teams' data is small by Spark standards too.

Where the cluster earns its cost

Spark is the right tool when one of these is true. Not "might be true in two years". True now.

The working set is larger than one big machine. After pruning, a single job must hold or shuffle terabytes at once, for example a self-join on a high-cardinality key across years of events. DuckDB will spill, but spilling a few terabytes through one machine's disk is slow, and some aggregates can't spill at all. The tuning guide says list() and string_agg() do not offload to disk.
Jobs run for hours and must survive a lost machine. Spark recomputes a lost partition and carries on. A DuckDB job that dies at hour three starts again from zero. If your batch is short and re-runnable, that doesn't matter. If it runs all night, it might.
You need streaming. Spark has Structured Streaming. DuckDB is batch.
A shared platform already exists. If twenty teams use one Spark platform with a shared catalog, permissions and lineage, the platform is the product. Pulling one small job out of it saves compute and costs you governance. That trade is sometimes worth it, but it is a governance decision, not a speed decision.

The reverse is also true. If none of these holds, the cluster is overhead: start-up time on every job, a driver and executors to size, a network shuffle for joins one machine could do in memory, and a bill that runs whether the job is busy or idle.

The middle ground people skip

Two options sit between "laptop" and "cluster".

Spark on one machine. Spark's local mode runs on a single machine: local[*] uses "as many worker threads as logical cores on your machine." Teams use it for tests. If you are already running Spark on one machine in production, you are paying for the JVM and the scheduler and getting none of the distribution. That job is the easiest DuckDB candidate you have.

DuckDB's PySpark-style API. DuckDB ships a Spark-compatible Python API under duckdb.experimental.spark. The documentation says it "is currently experimental and features are still missing." It is useful for trying a job on DuckDB without rewriting it first. I would not run production jobs through it. Rewrite the job in SQL instead, which is usually shorter than the PySpark it replaces:

-- One PySpark batch job, as DuckDB SQL over the same Parquet
COPY (
  SELECT
    customer_id,
    date_trunc('month', order_ts) AS month,
    count(*)                      AS orders,
    sum(amount)                   AS revenue
  FROM read_parquet('lake/orders/dt=2026-09-*/*.parquet', hive_partitioning = true)
  WHERE status = 'completed'
  GROUP BY ALL
) TO 'lake/marts/customer_month' (FORMAT PARQUET, PARTITION_BY (month));

Run that with explicit limits, so a big job spills instead of being killed:

SET memory_limit = '48GB';               -- below the machine's RAM
SET temp_directory = '/mnt/nvme/duckdb';  -- real disk for spills
SET threads = 16;

The other settings that matter on one node are in running DuckDB on your own infrastructure.

A sizing checklist before you keep (or buy) a cluster

Measure the working set of your three biggest jobs. Bytes read after pruning, from the query plan or storage metrics, not the table size.
Classify each as aggregation-heavy or join-heavy. Multiply threads by 1–2 GB or 3–4 GB. Compare with a machine you can rent.
Check job duration. Under an hour and safe to re-run: a failure costs a re-run. Hours long: Spark's recovery is worth something.
List anything streaming. That stays on Spark or moves to a streaming tool, not to DuckDB.
Count who shares the platform. One team: the cluster is a cost. Twenty teams: the cluster is a platform, and the decision is about governance.
Move one job and time it. Same Parquet, same output, compared row for row. One real measurement beats every benchmark post, including this one.

If steps 1 and 2 put every job on one machine, and steps 3 to 5 come back empty, you don't need the cluster for those jobs.

So: DuckDB or Spark?

For batch work that fits one machine, and that is most batch work, DuckDB is the simpler and usually faster choice. There is less to start, less to size and less to break.

Spark earns its cost when the data really is bigger than one machine, when long jobs must survive failures, when you need streaming, or when a shared platform is what you are buying. Those are real needs. Check that you have one before you pay for it.

Because both engines read the same Parquet, you are not locked in either way. Start on one node. Move the jobs that outgrow it.

Local-First Analytics covers single-node sizing, Parquet layout and the DuckDB-plus-Polars stack with runnable code. Chapter 1 is free to read, the rest is on Amazon. Related: when local-first runs out of road, DuckDB vs Snowflake, DuckDB vs Polars, and what DuckDB is actually good at in production.

If you want someone to size your actual jobs and tell you which ones still need the cluster, a Data Platform Audit is a week and a written roadmap you keep. The scoping call below is free.