How do I convert Parquet files to an Iceberg table?

Two ways. Rewrite: read the Parquet and append it to a new Iceberg table, which copies every byte but lets you fix file sizes, sorting and schema on the way. Register: call add_files (in PyIceberg or as a Spark procedure), which writes Iceberg metadata that points at the existing files and copies no data. In a test, PyIceberg registered 90 files (20 MB) in 0.24 seconds and wrote 38 KB of metadata; a rewrite of the same data took 0.83 seconds.

Does Iceberg add_files copy or move the Parquet files?

Neither. The files stay where they are and the table's manifests point at them. But the table now treats them as its own. A later compaction followed by snapshot expiry deletes them. In a test, Spark's add_files, rewrite_data_files and expire_snapshots left the original 90-file folder empty. Register a copy, or keep a backup, if those files are your only record.

What is the difference between snapshot, migrate and add_files in Iceberg?

All three are Spark procedures that build Iceberg metadata over existing files. snapshot creates a new Iceberg table from a Hive or Spark table and leaves the source untouched, which suits a trial. migrate replaces the source table with an Iceberg table of the same name and keeps the original as a backup. add_files adds files from a table or a path into an Iceberg table that already exists, and can pick up files that arrived after a snapshot or migrate.

How do I migrate Parquet on S3 to Iceberg?

The same calls work with s3:// paths: add_files reads each file's footer once and writes metadata next to your table, so the data never leaves the bucket. Iceberg's docs warn that this read pulls files out of archive tiers such as S3 Intelligent-Tiering. Check that every file has the table's columns, that a partitioned table's partition column is inside the files, not only in the folder names, and that no bucket lifecycle rule will delete files the table now owns.

Can I add existing Parquet files to DuckLake?

Yes, with CALL ducklake_add_data_files('lake', 'table', 'file.parquet'). DuckLake records the file without copying it and takes ownership, so its compaction and cleanup can later delete it. Every table column must be in the file unless you pass allow_missing => true. Wrap many calls in one transaction: 90 calls made 92 snapshots on their own and 3 inside BEGIN and COMMIT, three times faster.

Convert Parquet to Iceberg (or DuckLake) without rewriting the files

To convert Parquet to Iceberg, you do not have to copy a byte. Both Iceberg and DuckLake can register files you already have: they write metadata that points at your Parquet, and the folder becomes a table. On 90 files it took a quarter of a second.

That speed hides two catches. The table may not see what you think is in the files. And once registered, the files belong to the table, which will delete them during routine cleanup. I ran the conversion four ways and then ran the cleanup, so here is what each step did.

The test

The source: one Parquet file per day, 1 July to 28 September 2026. That is 90 files, 20.1 MB and 1,080,000 orders with order_id, ts, user_id, country and amount. I also wrote the same data as Hive folders (dt=2026-07-01/), the layout most lakes start with.

PyIceberg 0.12.0 with a SQL catalog on SQLite, and pyiceberg-core 0.10.1 for partitioned writes.
Spark 4.1.3 with the Iceberg 1.12.0 runtime.
DuckDB 1.5.6 with the ducklake and iceberg extensions, also used to check every result.

All on local SSD on my workstation, and every run on a copy of the source folder. I have not migrated a production lake to Iceberg or DuckLake. These are lab results.

Parquet to Iceberg migration: register or rewrite

Method	Time	Data written	Source files afterwards
PyIceberg `add_files`	0.24 s	none (4 metadata files, 38 KB)	owned by the table
Spark `add_files` procedure	2.7 s	none	owned by the table
PyIceberg read + `append` (rewrite)	0.83 s	90 files, 17.1 MB	untouched
DuckLake `ducklake_add_data_files`	0.26 s	none	owned by the table
DuckLake `INSERT … SELECT` (rewrite)	0.06 s	1 file, 20.0 MB	untouched

At 20 MB the rewrite is cheap, so time is not the reason to register. At 20 TB it would be. The real difference is the last column.

Register with PyIceberg add_files

from pyiceberg.partitioning import PartitionSpec, PartitionField
from pyiceberg.transforms import DayTransform

spec = PartitionSpec(PartitionField(source_id=2, field_id=1000,
                                    transform=DayTransform(), name="ts_day"))
t = catalog.create_table("shop.orders", schema=schema, partition_spec=spec)
t.add_files(file_paths=files)   # 90 paths

That took 0.24 seconds. The warehouse folder gained four metadata files and no data. DuckDB's iceberg_scan returned 1,080,000 rows and the same sum as read_parquet on the raw folder. A one-week filter on ts read 7 of the 90 files, so partition pruning worked on files Iceberg never wrote.

Two details from the PyIceberg docs explain how:

Field IDs. Iceberg tracks columns by ID, and plain Parquet files have none. add_files creates a name mapping from column names to IDs when the table lacks one. Mine got one automatically.
Partition values. It reads only each file's footer and takes the partition value from the column's min and max. That works for transforms that keep order, such as day or month, as long as each file holds one partition.

What add_files checks, and what it doesn't

I fed it four bad inputs.

Input	Result
A file already in the table	Rejected: "Cannot add files that are already referenced by table"
A file with an extra `coupon` column	Rejected: "PyArrow table contains more columns: coupon. Update the schema first"
A file whose rows span two days	Rejected: "Cannot infer partition value from parquet metadata as there are more than one partition values"
A file with `amount` as float32, not double	Accepted, read back as double

The float32 case is fine: float to double is a type promotion Iceberg allows. So PyIceberg 0.12 is stricter than I expected. Spark's procedure is not: its docs say add_files "will not analyze the schema of the files", and warn that a mismatched schema "will cause issues".

The two-day rejection matters for real lakes. A file written by a job that ran over midnight cannot go into a day-partitioned table as it is. Either rewrite it or register it into an unpartitioned table.

The Hive folder trap

This one was accepted, and it was wrong.

DuckDB's COPY … PARTITION_BY (dt) writes dt into the folder name and leaves it out of the file. The Parquet holds five columns; dt exists only in the path. So I made a table with a dt column, partitioned by identity(dt), and registered two of the files.

add_files accepted them. Every dt came back NULL, the partition value was NULL, and WHERE dt = '2026-07-01' returned 0 rows instead of 12,000. No error at any step. add_files reads the file footer, not the folder name.

The fix is to partition on a real column. The same Hive files went into a day(ts) table cleanly: 1,080,000 rows, with dt derived from ts by the transform. Iceberg partitioning shows why that is the better layout anyway. Before you register a Hive-style lake, check whether its partition columns are inside the files:

pyarrow.parquet.read_schema(one_file).names
# ['order_id', 'ts', 'user_id', 'country', 'amount']  -- no dt

Spark: snapshot, migrate and add_files

If your Parquet is already a Hive or Spark table, Iceberg's table migration docs describe three Spark procedures:

snapshot makes a new Iceberg table over the source's files and leaves the source table unchanged. Good for a trial.
migrate replaces the source table with an Iceberg table of the same name, keeping the original as a backup. Writers must stop first.
add_files adds files from a table or a path to an Iceberg table that already exists. It also picks up files written after a snapshot or migrate.

I ran add_files against a plain folder:

CALL lab.system.add_files(
  table => 'shop.orders',
  source_table => '`parquet`.`/data/orders_copy`'
);

It added 90 files in 2.7 seconds, and the count matched. On S3, the docs warn that add_files reads each file's footer once, which pulls files out of archive tiers such as S3 Intelligent-Tiering.

The same in DuckLake

DuckLake's version is a table function, one file per call:

CALL ducklake_add_data_files('lake', 'orders', '/data/orders/orders-2026-07-01.parquet');

Run on its own, each call is a transaction. Ninety calls took 0.85 seconds and left 92 snapshots. Inside one BEGIN … COMMIT they took 0.26 seconds and left 3. Wrap them.

The week query read 7 of 90 files, using the min and max values DuckLake stored per file. A file missing the country column was rejected with a clear error. The DuckLake docs list allow_missing and ignore_extra_columns to relax that.

Who owns the files now

This is the part to read twice. Iceberg's docs say files added this way are treated "as if they are part of the set of files owned by Iceberg", and that later expire_snapshots calls "will be able to physically delete the added files". DuckLake's docs say "the ownership of the Parquet file is transferred to DuckLake".

I tested both with the normal cleanup routine:

Iceberg (Spark): add_files, then rewrite_data_files (90 files into 1), then expire_snapshots. Expiry deleted 90 data files. The source folder went from 90 files to 0. The table still returned 1,080,000 rows, from its one new file.
DuckLake: ducklake_add_data_files, then merge_adjacent_files, expire_snapshots and cleanup_old_files. The source folder went from 90 files to 0. The table was still correct.

Nothing went wrong here. Both did exactly what their docs say. But if that folder was your raw archive, it is gone, and the only copy of the data is now inside the table. Iceberg table maintenance covers the cleanup jobs that do this.

A checklist before you convert

Decide who owns the files. If they are your only raw copy, register a copy, or rewrite instead.
Check that every partition column is inside the files, not only in folder names.
Check one file per day (or per partition). Files that span partitions must be rewritten or go into an unpartitioned table.
Check schemas across all files before Spark's add_files, which does not.
In DuckLake, add files inside one transaction.
Count rows and sum one column on the raw folder and the new table. They should match exactly.
Turn off bucket lifecycle rules on the table's paths. The table decides what to delete now.

So: register or rewrite?

Register when the files are large, already well sized, and you are happy for the table to own them from now on. It is fast and writes almost nothing.

Rewrite when the files are small or badly sorted, when partition values live only in folder names, or when the source must stay untouched. A rewrite is also the moment to fix file sizes, which registration carries over as they are.

For most small teams I would rewrite into the new table and keep the raw folder read-only. The cost is one copy of the data. In exchange, the table can compact and expire freely without touching the only record you have.

Local-First Analytics argues for that split in Chapter 4: a raw/ zone that is append-only and never modified, and curated/ tables built from it. Chapter 3 covers how far plain Parquet's schema evolution goes before a table format is worth it. Chapter 1 is free to read; the rest is on Amazon.

If you are moving a Parquet lake to Iceberg or DuckLake and want a second opinion on the plan, a Data Platform Audit is a week and a written roadmap you keep.

More in this series: Iceberg partitioning, Iceberg table maintenance, DuckLake vs Iceberg and DuckLake with Postgres and S3.