DuckDB in production: what it's actually good at (and what it isn't)

DuckDB is having a moment, and like every tool having a moment, the hype runs ahead of the nuance. "Just use DuckDB" is now a reflex answer to questions it doesn't actually fit. So let me give you the honest version, from someone who ships it.

What DuckDB is

DuckDB is an in-process analytical (OLAP) database — think "SQLite for analytics." It runs a columnar, vectorized query engine inside your application process. No server, no network hop, no cluster. You point it at Parquet, CSV, or its own format and run real SQL — window functions, joins, the works — at speeds that embarrass a round-trip to a cloud warehouse for the same-size data.

Is DuckDB production-ready? Do people actually run it?

Yes — with a shape attached, and the shape is the entire answer.

"Production-ready" doesn't mean "safe to drop in anywhere." It means the engine is stable, the file format is stable, and the failure modes are known and boring. DuckDB clears that bar. It's past 1.0, it's MIT-licensed, and there's a foundation and a company behind it rather than one maintainer with a day job. In the systems I run it has never lost data or quietly returned a wrong answer. The bugs have all been in my SQL.

What people are usually asking is the less polite version: do serious teams run this, or am I the guinea pig? They run it — mostly in a shape you can't see from outside. DuckDB rarely appears on an architecture diagram as "our warehouse." It appears inside a batch job, a service, a CLI, a notebook that runs in CI. It's an engine, not a platform, so it hides inside things, which is why real usage is far more common than the blog posts about it.

The caveat worth internalising: production-readiness here is a property of the system you build around it, not of the database. A warehouse hands you concurrency, access control and a serving layer in the box. DuckDB hands you a query engine and expects you to bring the rest. Bring it, and it's production-ready. Expect it included, and you'll conclude the tool is immature when what's missing is your service layer.

Where it shines in production

Transform-heavy batch jobs. Reading Parquet, joining, aggregating, and writing Parquet back out. DuckDB will saturate your cores and finish before a warehouse has authenticated your session.
Embedded analytics in an app or service. Ship query capability inside your service instead of calling out to a warehouse. Lower latency, no per-query bill, fewer moving parts.
Local and CI data work. Notebooks, ad-hoc analysis, and test fixtures that run identically on a laptop and in CI — no shared environment to provision or pollute.
The "interactive" layer over a lake. Parquet in object storage as the source of truth, DuckDB as the fast query engine over it. You get warehouse-ish ergonomics without the warehouse.

Can DuckDB be the warehouse for a small SaaS?

Yes — up to a specific shape. Worth knowing the shape before you commit rather than after.

The shape that works:

One writer. A single process — your ETL job, your API — holds write access. Everything else reads.
It sits behind your own service. Your users never connect to DuckDB. They hit your API, which queries DuckDB in-process and returns JSON. Your API is the concurrency layer, the auth layer and the rate limiter.
Parquet in object storage is the source of truth. The .duckdb file is a materialisation you can rebuild from scratch, not the only copy of the company's data.
The working set fits on one machine — gigabytes to low terabytes, with the hot tables comfortably inside RAM and disk.
Freshness is measured in minutes or hours, not seconds.

Inside that shape it's genuinely excellent, and it's dramatically less infrastructure than the alternative — no warehouse to keep warm, no per-query bill, no credentials rotation, and a stack a single engineer can hold in their head. The local-first walkthrough is the concrete version of this pattern in code: Parquet on disk or in a bucket, SQL straight over it, nothing in between.

Outside that shape, the failures are predictable rather than mysterious. The day a second team wants its own access. The day someone asks for row-level permissions per customer. The day the working set stops fitting on the box. The day "how fresh is this?" needs to be answered in seconds. Those are the signals I wrote up in when local-first runs out of road. Short version: one or two of them means fix your setup, not your architecture. All of them means graduate, and graduate deliberately.

DuckDB's limitations, honestly

Being honest about the edges is what makes a tool trustworthy. These are the ones that actually bite.

Single-process write concurrency. One process holds the database read-write at a time. Several processes can attach the same file read-only, and multiple threads inside a single process handle concurrent reads and writes fine — but there is no shared server that fifty clients connect to. If your design needs that, you are designing a service in front of DuckDB whether you intended to or not. It is not a multi-user warehouse and it is not trying to be.

No governance layer. No cross-team roles, no first-class row-level policies, no audit trail, no data sharing, no catalog everyone points at. The file is the permission boundary — whoever can read the file can read everything in it. For one team owning one dataset, fine. As "the one warehouse everyone trusts," not fine, and no amount of tooling around it fixes that cleanly.

Memory and spill behaviour. Larger-than-memory sorts, joins and aggregations spill to disk. That works, and it's a genuine strength compared to tools that simply die — but spilling is slower, it needs a temp directory with real room on it, and the memory limit is a setting you should choose deliberately rather than discover. The failure I see most often in practice: a container with a modest memory limit, a join that fans out further than expected, and an OOM kill that takes down the whole service — because DuckDB is running inside your process. In-process is a feature and a blast radius at the same time. Set DuckDB's memory limit below the container's, always.

Heavy concurrent writes and OLTP are the wrong job. It's an analytical engine. Don't make it your application's primary write store; that's what Postgres is for.

You build the serving layer yourself. Auth, connection lifecycle, request concurrency, caching, rate limiting, retries, metrics. None of that ships with the database, and all of it is work you'd otherwise have bought. Budget for it honestly — for most teams it's the largest hidden line item in "we'll just use DuckDB."

One machine. You scale up, not out. That's usually further than people expect on modern hardware, but it's a hard ceiling rather than a soft one.

Thin operational tooling by warehouse standards. No query-history console, no cost dashboard, no built-in scheduler, no usage attribution. Everything a warehouse UI would have given you, you either build or go without.

Is it safe to bet on?

Three different questions hide inside that one, and they have different answers.

Will my data survive? Transactions are ACID and the storage engine is crash-safe, so the database behaves. But the durable answer is architectural: keep Parquet in object storage as the source of truth and treat the DuckDB file as a derived artifact you can rebuild. Back up the Parquet, regenerate the rest, and "is it durable enough?" stops being a bet you're making.

Will the format still open in three years? The real format bet is Parquet and Arrow — industry-standard, multi-vendor, going nowhere. Your data isn't sitting in a proprietary blob. DuckDB's own storage format has carried backwards-compatibility commitments since 1.0, which is a meaningful signal of intent, but I'd still keep "rebuild from Parquet" a routine operation rather than a disaster-recovery drill.

Will the project still be here? MIT license, a non-profit foundation stewarding it, a company selling support around it, and an ecosystem that includes basically every data tool you'd want to interoperate with — a healthier structure than most open-source infrastructure. The more useful risk measure isn't a prediction, though, it's the exit cost. If DuckDB vanished tomorrow you'd point another engine at the same files and fix some dialect quirks. Leaving a proprietary warehouse is a migration project with a budget line. Low switching cost is the safety.

Three deployment patterns that work

1. In-process inside a service. Your API imports DuckDB and queries it directly — no network hop, no query service to operate, microseconds instead of milliseconds of overhead. One process writes, the API reads, your service handles auth and concurrency. Watch memory: DuckDB's limit goes below the container's, and a runaway query taking out the process takes out the request handler with it.

2. Containerised batch job. Ephemeral container starts, reads Parquet from object storage, transforms, writes Parquet back, exits. Nothing to keep alive between runs, retries are free, and "scaling" means a bigger box for ten minutes. Point cron, GitHub Actions, Dagster or Airflow at it and you're done. This is the lowest-risk pattern of the three and it's where most of my DuckDB usage lives.

3. Parquet lake with DuckDB readers. Object storage holds the truth; many independent read-only readers — a batch job, a notebook, a service, even the browser via duckdb-wasm — query it in parallel. Reads fan out infinitely because there's no shared server to contend on. You manage consistency by writing new partitions and swapping a prefix or manifest, rather than mutating files in place.

The anti-pattern, for completeness: several processes on a shared network filesystem all wanting write access to one DuckDB file. Don't. That's the setup that produces "DuckDB is unreliable" posts.

At a glance: where DuckDB fits

Workload shape	DuckDB?
Batch transforms over files — read Parquet, join, aggregate, write back	Yes
Embedded analytics inside one service, behind your own API	Yes
Read-only query layer over a Parquet lake	Yes
Local and CI analysis, reproducible reports, test fixtures	Yes
The warehouse for a small SaaS — single writer, working set on one box	Yes, within the shape above
Dozens of people querying one shared instance directly	No — put a service in front, or use a warehouse
Governed company-wide source of truth: roles, audit, sharing	No
High-rate concurrent writes, OLTP, application primary store	No — that's Postgres
Constant sub-second dashboard load from many users at once	No — ClickHouse territory
Streaming ingest with seconds-level freshness	No
Hot working set larger than one machine	No

The pattern that works

The mistake is treating DuckDB as a replacement for your warehouse. The win is treating it as the right-sized compute engine for the large fraction of your workload that never needed a warehouse in the first place:

Truth lives in Parquet. DuckDB is the fast, cheap, serverless way to ask questions of it. Reach for the warehouse only for the workloads that genuinely earn it.

Done this way, DuckDB isn't a toy or a hype cycle — it's the least infrastructure that actually solves the problem. Which, in my experience, is almost always the right amount.

The book: Local-First Analytics is the long version of all this — DuckDB, Parquet and Arrow in practice, with runnable code and real datasets. It's on Amazon, and if you'd like to read it before you buy it, you can request a review copy — I send those out by hand.

Trying to figure out where DuckDB fits in your stack without betting the company on it? Let's talk.