Find out if DuckDB fits your workload before you bet a migration on it.

DuckDB is excellent inside one shape and wrong outside it. Answer these before any tuning — a bad answer here is an architecture decision, not a setting.

1. How many processes need to write, and do they ever overlap?

   Bad answer: More than one, at the same time — an ETL job and an ingestion service both writing the same file. DuckDB allows one writing process. The second gets a lock error, or someone copies the file to dodge the lock and a reader sees a half-written state.

2. What is the working set — the bytes a typical query actually reads?

   Bad answer: Nobody has measured it, and the machine is sized off total data volume. What decides single-node fit is what one query touches after partition pruning and column selection, and it is usually far smaller.

3. How many queries run at the same moment at peak?

   Bad answer: Dozens of people writing ad-hoc SQL at once, or a customer dashboard queried by every logged-in user with no cache in front. That is a server’s job, and DuckDB has no server.

4. How fresh does the data need to be?

   Bad answer: Seconds old, from several producers writing continuously. DuckDB’s natural write pattern is a batch job; a constant stream of small writes from many services is the workload it is worst at.

5. Does anyone need roles, row-level security or an audit trail inside the database?

   Bad answer: Yes, because a regulator or a customer contract asks for it. DuckDB has no user model — the file is the permission boundary — and that is usually decisive on its own.

ACID covers a crash mid-transaction. It does not cover a deleted file, a container disk that vanished with the pod, or a copy taken mid-write.

1. If the database file disappeared right now, how long until you are back?

   Bad answer: Nobody knows, or the answer is “restore from backup”. The design that holds keeps Parquet in object storage as the source of truth and treats the database file as a cache you can rebuild.

2. Where does the database file live?

   Bad answer: On a pod’s local disk or in a container’s writable layer. It lives exactly as long as the pod does.

3. How is it backed up?

   Bad answer: By copying the file while a writer may be mid-transaction. The copy’s contents are undefined. Use EXPORT DATABASE, or rebuild from Parquet.

4. Is the DuckDB version pinned everywhere that opens the file?

   Bad answer: The notebook, the job and the service each run whatever the package manager resolved. A file written by a newer version is not guaranteed to open in an older one.

Memory is the failure you will actually hit, and inside a container it rarely looks like a memory problem.

1. Is memory_limit set explicitly, well below the container limit?

   Bad answer: It is left at the default or set equal to the container limit. DuckDB counts its own buffers, not the Python process, the dataframes or the Arrow tables in flight, so the orchestrator kills the process before DuckDB ever decides to spill. Roughly 70% of the container limit is a sane start.

2. Where does temp_directory point, and how much room does it have?

   Bad answer: The default location in the container’s ephemeral layer. A large join spills there, fills it, and the pod is killed — which reads as an out-of-memory crash and is actually a disk problem.

3. Does the threads setting match the CPU the container is actually given?

   Bad answer: It is left at the default, which comes from the node’s core count. A 4-core limit on a 64-core host means 64 threads spending their lives being throttled.

DuckDB is an engine, not a server. Whatever sits in front of it is the part of the database you are building yourself.

1. Does any user, BI tool or dashboard connect to DuckDB directly?

   Bad answer: Yes — a BI tool pointed at a shared file. The pattern that holds is your own API in front: it owns auth, caching and rate limits, and DuckDB answers only the queries your code writes.

2. Is every connection that only reads opened with read_only=True?

   Bad answer: Readers open the file read-write “just in case”, take the lock, and block the one writer that matters.

3. In a multi-tenant product, where does the tenant ID in each query come from?

   Bad answer: A request parameter, or a file path built by string formatting. It must come from the authenticated session and be passed as a query parameter. Anything else is a cross-tenant data leak, and it will not look like SQL injection when it happens.

4. When a query is slow, who finds out first?

   Bad answer: A customer. Nothing records query timings and nothing alerts. There is no warehouse console, so the only metrics are the ones your service emits.