Iceberg table maintenance: compaction, expire snapshots and orphan files

Iceberg table maintenance is the part of a lakehouse nobody puts in the plan. A table format gives you snapshots, deletes and time travel. The price is that nothing is ever deleted until you ask, and the files pile up in three different ways.

I wanted to see the pile, and what each cleanup job removes. So I made one table with 100 small commits and a delete, then ran each maintenance procedure in turn and counted files after every step. Two of the three procedures did nothing on their first run, because of their defaults. That is the most useful thing in this post.

Why an Iceberg table needs maintenance

An Iceberg table leaves behind four kinds of leftovers:

• Small data files. Every commit writes at least one new file. Frequent small writes mean many small files, and each one costs a request to open.
• Old snapshots. Every commit is a snapshot. Each keeps its data files alive, so you can time travel to it. Replaced files stay on disk.
• Orphan files. A write that crashes before it commits leaves files that no snapshot knows about.
• Old metadata files. Every commit writes a new metadata.json. By default the old ones stay.

Iceberg's maintenance docs name three jobs for these: expire snapshots, delete orphan files and compact data files. Below is each one on a real table.

A note on where this comes from. I run plain Parquet with DuckDB in production. I have not run Iceberg in production. These numbers come from a lab on my workstation (Intel i7-12700K, local SSD).

The test

• Spark 4.1.3 with the Iceberg 1.12.0 runtime, a Hadoop catalog on local disk, merge-on-read deletes.
• DuckDB 1.5.6 reading the same table through iceberg_scan.
• PyIceberg 0.12.0 for a second table, to see what a Python-only team can do.

The table: id, ts, kind, amount. One hundred INSERTs of 1,000 rows each, then a delete of every row where kind = 'view'.

Spark wrote four files per insert, one per task, so 100 commits left 400 data files.

After 100 inserts	Count	Size
Data files	400	1.05 MB
metadata.json files	101
Manifests and manifest lists	200
All metadata	301	7.5 MB

The metadata was seven times the size of the data. That gap comes from tiny commits, and it is normal for a table fed by a scheduler or a stream.

The delete wrote 400 position-delete files, one beside each data file. A Spark count(*) and sum went from 162 ms to 1,353 ms (best of five). DuckDB went from 8.7 ms to 27.2 ms. Merge-on-read makes deletes cheap to write and costly to read until something merges them.

Iceberg compaction with rewrite_data_files

Compaction reads small files and writes them back as large ones. In Spark it is a procedure:

CALL lab.system.rewrite_data_files(
  table => 'shop.events',
  options => map('delete-file-threshold', '1')
);

It rewrote 400 data files into 1 in 5.1 seconds. The rewritten file has the deletes applied. The Spark query fell from 1,353 ms to 31 ms.

Two things surprised me.

The defaults decide which files qualify. The procedure docs list a 512 MB target file size and a min-input-files of 5. A file group with five or more files is rewritten. Files with deletes count only past delete-file-threshold (default 2,147,483,647 deletes) or delete-ratio-threshold (30% of rows). I set the threshold to 1 so every file with a delete qualified.

The delete files stayed. After compaction, the 400 delete files were still live in the current snapshot, though no live data file needed them. rewrite_position_delete_files with rewrite-all dropped all 400. The docs call these "dangling" deletes. You can also pass remove-dangling-deletes => true to rewrite_data_files, which costs one extra commit.

rewrite_manifests had nothing to do here: 0 manifests rewritten. It matters for tables with thousands of manifests, not this one.

Expire snapshots

Compaction does not free any disk space. After it, the folder held 801 data files, more than before, because the old files belong to older snapshots. Snapshot expiry is what deletes them.

CALL lab.system.expire_snapshots(table => 'shop.events');

That call deleted nothing. The default older_than is five days ago, and every snapshot was minutes old. All 103 snapshots survived. If you try maintenance on a test table and see no change, this is why.

With an explicit cut-off:

CALL lab.system.expire_snapshots(
  table => 'shop.events',
  older_than => TIMESTAMP '2026-10-02 16:30:00',
  retain_last => 1
);

It ran in 1.4 seconds and deleted 400 data files, 400 delete files, 102 manifests and 102 manifest lists. One snapshot remained, with one 278 KB data file. DuckDB's sum fell to 2.7 ms.

Expired snapshots are gone for good. Time travel to them no longer works. So pick the retention window from how far back people actually query, not from disk cost.

Remove orphan files

Orphans are files under the table folder that no snapshot references. I made one the honest way: I wrote a Parquet file into data/ by hand, as a crashed writer would.

CALL lab.system.remove_orphan_files(table => 'shop.events', dry_run => true);

The dry run returned nothing. The default only considers files older than three days. When I passed older_than set to a minute ago, Spark refused:

Cannot remove orphan files with an interval less than 24 hours. Executing this procedure with a short interval may corrupt the table if other operations are happening at the same time.

That guard is right. A file a writer is producing at this moment has no snapshot yet, so it looks exactly like an orphan. I set the planted file's timestamp back four days and ran it again. The dry run listed that one file, and the real run removed it.

The docs add a second warning. Orphan detection compares path strings. If the same file shows up as s3a:// in metadata and s3:// in a listing, it looks like an orphan. The procedure maps s3a and s3n to s3 by default and errors on other mismatches. Read the dry run before you trust it.

The metadata.json pile

After all of that, the folder still held 105 metadata.json files, 6.7 MB. Snapshot expiry does not delete them. They are kept as history by default.

One table property fixes it going forward:

ALTER TABLE lab.shop.events SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '10'
);

Three commits later there were 15 metadata.json files and 866 KB of metadata in total. The defaults are false and 100, per the configuration docs. The maintenance page adds a catch: files that have already dropped out of the metadata log are untracked, and only orphan removal deletes those. Set this property when you create the table.

Iceberg maintenance with PyIceberg

Many small teams use Iceberg from Python only, with no Spark. PyIceberg 0.12.0 is the latest release on PyPI as I write. Its API docs list one maintenance call: table.maintenance.expire_snapshots(). There is no rewrite_data_files and no orphan removal.

On a second table, 50 appends of 1,000 rows, then the same delete:

• Compaction stand-in. t.overwrite(t.scan().to_arrow()) rewrote the table as one live file in 0.34 seconds. That reads everything into memory, so it only works for tables that fit.
• Snapshot expiry. It cut 53 snapshots to 1 in 0.03 seconds. The folder still held 101 data files, 1.1 MB. The snapshots were gone from the metadata, but their files were not deleted.

So I swept the files by hand. PyIceberg can list every file that any remaining snapshot references:

live = {p.removeprefix("file://")
        for p in t.inspect.all_files().column("file_path").to_pylist()}
on_disk = set(glob.glob(f"{table_dir}/data/**/*.parquet", recursive=True))
orphans = on_disk - live   # 100 files, 789 KB

It found 100 orphans in 2 ms. I deleted them, and DuckDB and PyIceberg both still returned 33,333 rows. Only run a sweep like this when no writer is running. It has none of Spark's 24-hour safety window, and a file mid-write would be in the orphan set.

DuckDB cannot fill the gap. Its Iceberg extension can write to tables in a REST catalog. The writing docs say updates and deletes are merge-on-read only, so they add delete files. The function list has no compaction or expiry call. Something else has to clean up after it.

A maintenance routine for a small team

The order matters: compaction creates the files expiry deletes, and expiry creates the orphans the sweep finds.

1. On create: set write.metadata.delete-after-commit.enabled = true and a previous-versions-max you can live with.
2. Daily or after heavy writes: rewrite_data_files, with remove-dangling-deletes if you use merge-on-read.
3. Daily, after compaction: expire_snapshots with an explicit older_than that matches your time-travel promise.
4. Weekly: remove_orphan_files, dry run first, with the default three-day window or longer.
5. Every run: log the counts each procedure returns. A run that deletes zero files for weeks is a run with the wrong arguments.

So: who runs your Iceberg maintenance?

That is the real question. The procedures are short and fast. On this table every one took under six seconds. The hard part is that someone has to schedule them, pick the windows and notice when they silently do nothing.

If you have Spark, the procedures above are the reference version. If you only have Python, PyIceberg 0.12 leaves you with an overwrite and a hand-rolled sweep. If you want neither, pick something that runs it for you: pg_lake vacuums its Iceberg tables, and some managed catalogs compact for you, as Iceberg catalogs explained lists. DuckLake does the same jobs in four SQL calls, which DuckLake vs Iceberg measures.

And if one job writes and everything else reads, you may not need any of this. Plain Parquet has no snapshots to expire.

Local-First Analytics does not cover Iceberg maintenance. It covers the plain-Parquet side of the same problem: Chapter 4's rule to keep partition files at 50 MB or more, and to drop a partition level when files fall below that. Chapter 1 is free to read; the rest is on Amazon.

If you run Iceberg tables and want a second opinion on how they are kept healthy, a Data Platform Audit is a week and a written roadmap you keep.

More in this series: Iceberg vs Parquet, DuckLake vs Iceberg, Iceberg catalogs explained and Postgres and Iceberg.