Is DuckDB safe for production? The honest limitations

"Is DuckDB safe for production?" is the right question asked slightly wrong. DuckDB is not a scaled-down toy that becomes safe once you're brave enough. It's a database with a specific concurrency and durability model, and it is completely safe inside that model and genuinely unsafe outside it.

So the useful version of the question is: safe for which workload? Here's what actually bites, in the order it bites people.

The single-writer model is the whole story

DuckDB allows one writing process at a time against a database file. Multiple threads inside that process are fine. Multiple readers are fine. Two processes both opening the file for writing are not.

This is not a bug or a temporary limitation — it's the design. DuckDB is an in-process database, like SQLite. There is no server arbitrating between clients, because there is no server.

Almost every "DuckDB isn't production-ready" story I've heard traces back to this. Someone runs the ETL job on a schedule, someone else points a dashboard at the same file, and eventually the two overlap. What you get is a lock error, or — if you were clever enough to copy the file to dodge the lock — a reader seeing a half-written state.

The design that avoids it entirely:

                    ┌─ writer job ──► Parquet partitions on S3
                    │                   (immutable, append-only)
  source systems ───┤
                    └─ readers ────► DuckDB, N processes, read-only

Writers never share a mutable file. They produce immutable Parquet partitions. Readers open those partitions read-only, as many processes as you like, with no coordination at all. The concurrency problem disappears because you removed the shared mutable state, not because you managed it better.

If you must use a .duckdb file with concurrent readers, open them explicitly read-only:

import duckdb

# Safe: many of these can run at once against the same file.
con = duckdb.connect("analytics.duckdb", read_only=True)

Read-only connections don't take the write lock, and DuckDB will refuse a write through them rather than corrupting anything.

Durability: it's ACID, but know what that covers

DuckDB is ACID-compliant with write-ahead logging. A transaction that commits is durable; a crash mid-transaction rolls back cleanly. On this axis it behaves like a real database, because it is one.

What people actually get wrong is what sits around the transaction:

• The file is a single point of failure. ACID protects you from a crash. It doesn't protect you from a deleted file, a corrupted volume, or an ephemeral container's disk vanishing at the end of the run. If your database lives on a pod's local disk, it lives exactly as long as the pod.
• A .duckdb file is not a backup format. Copying it while a writer is mid-transaction gives you a file whose contents are undefined. Snapshot by exporting (EXPORT DATABASE) or by treating Parquet as the durable layer.
• Storage format compatibility. DuckDB reached 1.0 with a stability commitment, and files are forward-compatible within that line — but a file written by a newer version isn't necessarily readable by an older one. Pin the version in production and upgrade deliberately, the same as any database.

The rule I use: Parquet is the durable artefact, the .duckdb file is a cache. If losing the file would be a data-loss incident rather than an inconvenience, the architecture is wrong, not DuckDB.

Memory is the failure mode you'll actually hit

DuckDB spills to disk when a query exceeds memory, so it doesn't simply die the way people expect. But the spill has to go somewhere, and the default temp location is often not where you want it in a container.

Two settings that belong in every production configuration:

SET memory_limit = '12GB';        -- leave headroom for the host process
SET temp_directory = '/var/tmp/duckdb';  -- a real disk with real space

Get these wrong and the classic production failure is a large join filling the container's ephemeral layer until the orchestrator kills the pod — which looks like a mysterious OOM and is actually a disk problem.

Set memory_limit well below the container limit. DuckDB accounts for its own buffers, not for the Python process holding your dataframes, the Arrow tables in flight, or the runtime itself.

Where DuckDB is genuinely the wrong choice

Be honest about these rather than working around them:

• Many concurrent writers. Covered above. If the requirement is real and can't be redesigned into partitioned writes, use a database with a server.
• Row-level security, roles, audit trails. DuckDB has no user model. It's a library reading a file — the security boundary is the process and the filesystem, not the database. In regulated environments this is usually decisive on its own.
• High-concurrency interactive serving. Fifty analysts writing exploratory SQL simultaneously is a warehouse workload. One dashboard hitting a cached result set is not. Know which you have.
• OLTP. It's a columnar analytical engine. Thousands of small point updates per second is the workload it's worst at, and Postgres is right there.
• Cross-region distributed queries. Single node means single node.

Where it's been safe, in my experience

Everything that is batch, single-writer, and reproducible:

• Transformation jobs reading Parquet and writing Parquet — the largest share of most compute bills, and structurally free of the concurrency problem.
• Embedded analytics inside an application that controls its own queries.
• CI: running the real engine against real data in seconds, so tests exercise production SQL rather than a mock.
• Local development against a copy of production Parquet, offline, with no cloud credentials.
• Anything where the alternative was a cluster provisioned for a working set that fits in RAM.

The common thread: the workload has one writer and tolerates being re-run. If a job fails, you fix it and run it again, and the output is identical. That property makes the whole class of concurrency questions moot.

A short pre-production checklist

Before putting DuckDB on the critical path, I'd want yes to all of these:

1. Exactly one process writes; every other connection is read_only=True.
2. Parquet (or another durable store) is the source of truth; the .duckdb file can be rebuilt from it.
3. memory_limit and temp_directory are set explicitly, with temp_directory on real disk.
4. The DuckDB version is pinned, and upgrades are a deliberate change with a test run.
5. Failure means "re-run the job", not "restore from backup".

That list is short because the model is simple. The danger isn't that DuckDB is fragile — it's that it's so easy to start using that people skip the design step they'd never skip for Postgres.

So: is it safe?

For batch transformation, embedded analytics, CI and development, DuckDB has been more reliable in my hands than the distributed systems it replaced, mostly because there's dramatically less to go wrong. One process, one file, no network partition, no cluster state.

For multi-writer, governed, high-concurrency serving, it isn't safe and won't become safe — that's a different tool's job.

The honest failure mode isn't corruption. It's discovering six months in that you needed the governance layer.

If you want the full argument with runnable code, that's my book, Local-First Analytics — DuckDB, Parquet and Arrow on hardware you already have. Related reading: what DuckDB is actually good at in production, when local-first runs out of road, and DuckDB vs Snowflake.

If you'd like someone to look at your specific workload and tell you honestly whether this design fits it, a Data Platform Audit is a week and a written roadmap you keep. The scoping call below is free.