DuckLake vs Iceberg: which table format for a small team

DuckLake vs Iceberg is usually argued on design. Iceberg keeps table metadata as files next to the data. DuckLake keeps it as rows in a SQL database. Both camps have a diagram that proves they are right.

I wanted a number instead. So I ran the same workload on both: 50 small commits, a delete, a column rename, time travel, then cleanup. I counted every file each one left on disk and timed every commit. Below is what each format cost, and which one I would pick for a team of two to ten people.

What is DuckLake?

DuckLake is an open table format from the people behind DuckDB. The data is Parquet, the same as Iceberg. The difference is where the table's bookkeeping lives.

In Iceberg, every commit writes new metadata files: a metadata.json, a manifest list and one or more manifests. A catalog then points at the newest metadata.json. Iceberg catalogs explained covers that part.

In DuckLake, the bookkeeping is a set of tables in a database you already know how to run: Postgres, SQLite, a DuckDB file, or any ACID SQL database with primary keys, per the FAQ. A commit is one transaction against that database. There are no metadata files at all.

On status: DuckLake 1.0 shipped on 13 April 2026 with DuckDB 1.5.2. The announcement calls it "a production-ready release with guaranteed backward-compatibility". In 1.0, data inlining is on by default, and the release added sorted tables, bucket partitioning that uses the same murmur3 hash as Iceberg, GEOMETRY and VARIANT types, and experimental Iceberg v3 deletion vectors. Clients exist for DataFusion, Spark, Trino and pandas. The release calendar plans 2.0 for 21 October 2026, and says the date is tentative.

The test

Same machine, local disk, same table: id, ts, kind, amount. Fifty appends of 1,000 rows each, then the same follow-up steps on both.

• Iceberg: PyIceberg 0.12.0 with its SQL catalog on SQLite. These are the numbers from my Iceberg vs Parquet test.
• DuckLake: DuckDB 1.5.6 with the ducklake extension, run once with a SQLite catalog and once with Postgres 16 in a throwaway container.

INSTALL ducklake; LOAD ducklake;
ATTACH 'ducklake:sqlite:catalog.sqlite' AS lake (DATA_PATH 'data/');
USE lake;
CREATE TABLE events (id BIGINT, ts TIMESTAMP, kind VARCHAR, amount DOUBLE);
-- then 50 x: INSERT INTO events SELECT ... FROM range(start, start + 1000)

One caveat up front. The writers differ: PyIceberg is Python, DuckDB is C++. So commit times compare two real tools, not two formats in a vacuum. The file counts are a property of the formats.

DuckLake vs Iceberg: what 50 commits leave behind

After 50 appends	Iceberg	DuckLake (SQLite)	DuckLake (Postgres)
Data files	50, 450 KB	50, 588 KB	50, 588 KB
Metadata files	151, 1,643 KB	0	0
Catalog growth	one row updated	+16 KB	+80 KB
Time for 50 commits	0.84 s (17 ms each)	0.39 s (8 ms each)	0.26 s (5 ms each)
Snapshots readable	50	50	50

Both made one Parquet file per commit. DuckLake's files were a little bigger; I did not tune either writer's compression.

The metadata is where they part. Iceberg wrote 51 metadata.json files, 50 manifest lists and 50 manifests: 1.6 MB of metadata for 450 KB of data. DuckLake wrote nothing outside the data folder. Its SQLite catalog went from 148 KB empty to 164 KB. The catalog has 29 tables; ducklake_data_file had 50 rows and ducklake_snapshot had 52.

Then I pushed both to 500 commits of 200 rows.

500 commits	Iceberg	DuckLake (SQLite)	DuckLake (Postgres)
Commit time, median of first 10	9 ms	7.6 ms	3.9 ms
Commit time, median of last 10	202 ms	7.8 ms	3.9 ms
Metadata	1,501 files, about 106 MB	in a 316 KB catalog file	in Postgres

Iceberg slows down because each new metadata.json carries the whole snapshot history. DuckLake adds a few rows per commit, and a database does not care how many rows came before. On object storage the gap grows, because each Iceberg metadata file is another request.

Deletes, renames and time travel: a draw

The features that make a table format worth having worked on both.

• Delete. I deleted every row where kind = 'view'. Both returned 33,333 rows, which is right. PyIceberg rewrote the affected data files (50 files became 100). DuckLake left the data files alone and wrote 50 small delete files, which is merge-on-read. A plain read_parquet over DuckLake's data files still returns 50,000 rows, so never point a raw glob at a table-format folder.
• Rename. I renamed amount to amount_inr, added channel and appended 1,000 rows. Both returned 34,333 rows with all 34,333 amount_inr values. A plain Parquet glob saw only the 1,000 new ones.
• Time travel. Reading the first commit's snapshot returned exactly 1,000 rows on both. In DuckLake it is plain SQL: SELECT count(*) FROM events AT (VERSION => 2).

Small files: inlining changes the problem

Iceberg's small-file problem is a cleanup job. DuckLake avoids much of it at write time. Inserts below a threshold (10 rows by default in 1.0) go into the catalog database as rows, not into new Parquet files.

I ran 50 inserts of 5 rows each into a second table. Not one Parquet file appeared. Each commit took 7 ms on SQLite and 4 ms on Postgres. CALL ducklake_flush_inlined_data('lake') then wrote all 250 rows as one file.

That suits the pattern small teams actually have: a webhook, a trickle of corrections, a nightly batch. With Iceberg, each of those small writes is a data file plus three metadata files.

Cleanup: both need it, DuckLake's is shorter

Both formats keep old files until you say otherwise. In DuckLake it is four calls:

CALL ducklake_rewrite_data_files('lake', delete_threshold => 0.3);
CALL ducklake_merge_adjacent_files('lake');
CALL ducklake_expire_snapshots('lake', older_than => now());
CALL ducklake_cleanup_old_files('lake', cleanup_all => true);

Two things caught me out. First, merge_adjacent_files skipped every file that had a delete file attached. My first run merged nothing. Second, rewrite_data_files only rewrites files with more than 95% of rows deleted by default (docs). I had deleted a third, so I lowered the threshold. After that, the 100 data and delete files became 2 files, 483 KB, and the table still returned 34,333 rows. Every call took under a tenth of a second. DuckLake's docs recommend running CHECKPOINT, which runs all of these steps for you.

For the 500-commit table, merge_adjacent_files took 0.05 s and turned 500 files into 1. The sum query went from 15.7 ms to 4.3 ms on the Postgres catalog. In my Iceberg test, PyIceberg 0.12 had no compaction call, so I rewrote the table by hand (1.8 s), and expiring snapshots still left every old file on disk.

Where Iceberg still wins

Engine reach. Iceberg is the format other systems agree on: the big query engines, the warehouses, the managed catalogs. If Spark, Trino, Snowflake or a vendor's ingestion tool must read and write the same tables as you, Iceberg is the safe answer, and the catalog question in Iceberg catalogs explained comes with it.

DuckLake's list is shorter: DuckDB first, then the DataFusion, Spark and Trino clients and a pandas implementation named in the 1.0 post. I have not tested those clients. Check the one you need before you commit to it.

If your data already lives in Postgres, Postgres and Iceberg covers pg_lake, which writes Iceberg from Postgres and runs its compaction for you.

The choice is also less final than it looks. Since DuckLake 0.3, iceberg_to_ducklake imports an Iceberg table's metadata, snapshot history included, without copying the data. COPY FROM DATABASE deep-copies the latest snapshot either way.

What two writers did

Several writers are a reason to want a table format at all. I started two processes that each made 25 commits to one DuckLake table at the same time.

• Postgres catalog: all 50 commits landed, and no error reached my code.
• SQLite catalog: two commits failed with "Failed to commit DuckLake transaction". My loop retried them and all 5,000 rows landed.

DuckLake's docs say the same thing: SQLite works for a few processes on one machine, and Postgres is the catalog for many clients on many machines. A DuckDB-file catalog is single-client. The setup post covers the single-writer DuckDB file this replaces.

Is DuckLake production ready?

The format is, by its maintainers' statement and its backward-compatibility promise. Whether it is ready for you depends on three things:

1. Your catalog database is production-grade. With DuckLake, the catalog is the table. Lose the Postgres database and the Parquet files are an unordered pile. Back it up like the primary database it is.
2. Your readers speak DuckLake. If a BI tool or a vendor needs to read the tables directly, check that it can.
3. You will run maintenance. Inlining reduces small files. It does not remove the need for CHECKPOINT or the four calls above.

I have not run DuckLake in production. Everything here comes from the lab runs above. My production lake is plain Parquet under one writer, which is still enough for most teams, as Iceberg vs Parquet argues.

So: DuckLake or Iceberg?

For a small team that queries with DuckDB, pick DuckLake. It has fewer files and flat commit times, and the metadata lives in a database you can query with SQL. Run the catalog on Postgres, or on SQLite if one machine does all the writing.

Pick Iceberg when other engines must share the tables, or a vendor already speaks it. Plan its compaction and cleanup on day one.

Pick neither if one job writes and everything else reads. Plain Parquet does that with no catalog at all.

For a fuller treatment of DuckLake itself, O'Reilly's DuckLake: The Definitive Guide by Matt Martin and Alex Monahan is in early release, and MotherDuck offers the early chapters free.

Local-First Analytics does not teach DuckLake or Iceberg. It covers the layer underneath both: Parquet files, how to lay them out and partition them (Chapter 4), and how far plain files go before you need a table format (Chapter 3). Chapter 1 is free to read; the rest is on Amazon.

If you are choosing a table format for a real system and want a second opinion, a Data Platform Audit is a week and a written roadmap you keep. If plain Parquet is enough, that is what it will say.

More in this series: DuckLake with Postgres and S3, lakehouse vs data warehouse, Iceberg vs Parquet, Postgres and Iceberg and Iceberg catalogs explained.