What partition transforms does Iceberg support?

Identity (the raw value), year, month, day and hour for dates and timestamps, bucket(N, col), which hashes a value into N buckets, and truncate(W, col), which cuts strings to W characters or rounds numbers down to a multiple of W. A table can combine several, for example bucket(16, id) and day(ts).

What is Iceberg partition evolution?

Changing a table's partition spec without rewriting it. Old files keep the layout they were written with, new files use the new spec, and queries plan across both. In a test, replacing month(ts) with day(ts) took 35 ms and touched no data files. The catch: a query over the old data still reads whole month files, so a one-week filter on old data read one file holding a whole month.

What are Iceberg partitioning best practices?

Partition by the column most queries filter on, usually a timestamp, at the coarsest grain that still skips most files. Avoid partitions so fine that each file is small: in a test, partitioning 1.08 million rows by hour made 2,160 files, 34% more bytes and a full scan ten times slower than by day. Use bucket for high-cardinality keys. For small tables, sorting by the filter column may be enough. Change the spec with partition evolution rather than a rewrite.

Hive partitioning vs Iceberg partitioning: what is the difference?

Hive partitioning is a folder convention: dt=2026-09-01/ in the path, and a dt column the writer must fill and the reader must filter on. Iceberg partitioning lives in table metadata: the writer applies a transform to a real column, and readers filter on that column. Iceberg can also change the scheme later without moving files. Hive folders need no catalog and any engine can read them.

Does DuckLake support partitioning?

Yes. ALTER TABLE ... SET PARTITIONED BY sets partition keys for new data, using identity, year, month, day, hour or bucket. Earlier data keeps its old layout, as in Iceberg. Files go in Hive-style folders by default. In a test, a table partitioned by year, month and day wrote 90 files and a one-week query read 7.

Iceberg partitioning: hidden partitions vs Hive folders, measured

Q: What is Iceberg hidden partitioning?

Iceberg derives partition values from a column you already have, using a transform such as day(ts), and records the link in the table metadata. Queries filter on the real column (WHERE ts >= ...) and Iceberg works out which partitions, and so which files, can match. With Hive-style folders, the query must filter on a separate partition column such as dt, or every file is read. In a test on 90 days of data, a one-week filter on ts read 7 of 90 files from the Iceberg table and all 90 from the Hive folders.

Iceberg partitioning is sold on one feature: hidden partitioning. You filter on the timestamp you already have, and Iceberg works out which day folders to read. With Hive-style folders, you filter on a separate date column, or you read everything.

That sounded like a small convenience until I measured it. I loaded the same 1.08 million rows of orders, 90 days of them, as Hive folders, as Iceberg tables partitioned three ways, and as a DuckLake table. Then I counted how many files each one-week query read. The answer ranged from 1 to 168, and it depended more on the partition grain than on the format.

The test

Data: 1,080,000 orders from 1 July to 28 September 2026, 12,000 a day: order_id, ts, user_id, country, amount. About 20 MB of Parquet.
Writers: DuckDB 1.5.6 for the Hive folders and DuckLake. Spark 4.1.3 with the Iceberg 1.12.0 runtime for the Iceberg tables.
Readers: DuckDB 1.5.6 (EXPLAIN ANALYZE reports how many files a scan read), PyIceberg 0.12.0 for scan planning, and Spark.
Query: count and sum of amount for 1 to 7 September, 84,000 rows.

Everything ran on local SSD on my workstation, so every query took milliseconds. The file counts are the number to watch. On object storage, each file read is at least one request. I have not run Iceberg in production; these are lab numbers.

Hive partitioning: the column you must remember

DuckDB wrote the data as dt=2026-07-01/, dt=2026-07-02/ and so on, 90 folders with one file each. Then I ran the week query two ways.

-- Filter on the partition column
SELECT count(*), sum(amount)
FROM read_parquet('orders/**/*.parquet', hive_partitioning = true)
WHERE dt BETWEEN '2026-09-01' AND '2026-09-07';

-- Filter on the timestamp, which is what people actually write
... WHERE ts >= '2026-09-01' AND ts < '2026-09-08';

Filter	Files read	Rows
`dt BETWEEN …`	7 of 90	84,000
`ts >= … AND ts < …`	90	84,000

Both return the right answer. Only the first one skips files. DuckDB does not know that dt is the date of ts. Nothing records that link except the person who wrote the folders.

Iceberg's partitioning docs list more problems with this. The writer must produce the right value in the right format. A dt written as 2018-12-01 in one job and 20181201 in another gives silently wrong results, not an error.

What is Iceberg hidden partitioning?

In Iceberg the partition is a transform of a real column, declared once:

CREATE TABLE lab.shop.by_day (
  order_id BIGINT, ts TIMESTAMP_NTZ, user_id BIGINT, country STRING, amount DOUBLE
) USING iceberg
PARTITIONED BY (days(ts));

There is no dt column. The writer computes the day from ts, and the metadata records the day for each file. A query filters on ts, and the planner turns that into a day range.

On the same week query against this table, DuckDB read 7 of 90 files. PyIceberg's scan planner picked the same 7. Spark answered in 60 ms. Nobody had to know the table was partitioned.

Iceberg partition transforms

The Spark DDL docs list the transforms:

Transform	Example	Use it for
identity	`country`	Low-cardinality columns that filters name directly
year, month, day, hour	`day(ts)`	Time ranges, the common case
bucket	`bucket(16, user_id)`	High-cardinality keys, spread into a fixed number of groups
truncate	`truncate(4, sku)`	String prefixes, or numbers rounded down to a multiple of W

You can combine them, for example bucket(16, id), day(ts). The older names days(ts) and hours(ts) still work, which is what I used.

How fine to partition: day vs hour

Finer partitions skip more data per query. They also make more, smaller files. I wrote the same rows three ways to see where that breaks.

Iceberg table	Data files	Size	Full scan (DuckDB)	Week query: files, time
Unpartitioned, sorted by `ts`	4	10.4 MB	3.4 ms	2 files, 5.2 ms
`days(ts)`	90	10.5 MB	3.7 ms	7 files, 2.4 ms
`hours(ts)`	2,160	14.1 MB	38.3 ms	168 files, 23.4 ms

Partitioning by hour gave 2,160 files of about 500 rows each. The table grew 34%: each small file carries its own footer and statistics, and compresses only 500 rows at a time. The full scan was ten times slower than by day. The week query read 168 files instead of 7. Spark also needed more memory to write it: at the default 1 GB driver heap the hourly write ran out of memory, and at 6 GB it took 4.6 seconds against 1.2 seconds by day.

Hourly partitions make sense when each hour holds hundreds of megabytes. At 500 rows an hour, they are only small files.

The first row is the quiet surprise. The unpartitioned table, written in ts order, let DuckDB skip 2 of its 4 files using each file's min and max values. Sorting got some of the benefit with no partition spec at all. For a small table, sort before you partition.

Iceberg partition evolution

Say a table started with monthly partitions and now gets enough data for daily ones. In Hive folders, that is a rewrite and every query changes. In Iceberg it is one statement:

ALTER TABLE lab.shop.evolve REPLACE PARTITION FIELD months(ts) WITH days(ts);

I loaded July and August under months(ts), which made 2 files. The ALTER took 35 ms and changed no data files. Then I loaded September.

Spec	Files	Rows
0: `months(ts)`	2	744,000
1: `days(ts)`	28	336,000

On disk, ts_month=2026-07/ and ts_day=2026-09-01/ folders sat side by side. A count over the whole table returned all 1,080,000 rows. The evolution docs call this split planning: each spec's files are planned with that spec's rules.

The catch shows up in the file counts:

A week in September read 7 daily files.
A week in August read 1 file: the whole of August, 372,000 rows, to return 84,000.

Evolution changes the future, not the past. If the old data needs the new layout, compact it into the new spec. Spark's rewrite_data_files has an output-spec-id option for that. Compaction has its own costs, covered in Iceberg table maintenance.

Partitioning in DuckLake

DuckLake has the same idea, with SQL you already know:

ALTER TABLE lake.orders SET PARTITIONED BY (year(ts), month(ts), day(ts));
INSERT INTO lake.orders SELECT * FROM read_parquet('orders/*.parquet');

The insert took 0.24 seconds and wrote 90 files under paths like year=2026/month=7/day=1/. The week query on ts read 7 files.

The DuckLake partitioning docs say three things worth knowing:

Partition keys apply only to new data. Old files keep the layout they were written with, as in Iceberg.
Pruning uses per-file min and max values stored in the catalog. Partitioned writes keep those ranges tight.
Folders are Hive-style by default. You can turn that off with the hive_file_pattern option, because the catalog, not the path, holds the partition values.

Transforms are identity, year, month, day, hour and bucket. There is no truncate. Bucket uses the same Murmur3 hash as Iceberg.

Iceberg partitioning best practices for a small team

Partition by what queries filter on. For event and order data that is almost always a timestamp.
Pick the coarsest grain that still skips most files. Day for this data, month for slower tables. Look at the average file size per partition, not the row count.
Do not partition small tables. Write them sorted by the filter column and let min/max statistics do the skipping.
Use bucket for high-cardinality keys. Partitioning by user_id directly would make one folder per user.
Evolve instead of rewriting. Change the spec when the data outgrows it, and compact old partitions only if queries on them are slow.
Check the plan. EXPLAIN ANALYZE in DuckDB, or the files table in Spark, tells you how many files a query read. Do not guess.

So: does hidden partitioning matter?

Yes, mostly for people who did not write the table. With Hive folders, a dashboard query that filters on ts instead of dt reads every file and nobody notices until the bill or the latency does. With Iceberg or DuckLake, the right filter is the natural one.

But the bigger lever was the grain. Day partitions read 7 files for a week. Hour partitions read 168 and made the whole table slower. Sorting alone read 2 of 4. Get the grain right first and the format second.

If you do not need snapshots or several writers, Hive folders plus a rule that every query filters on dt still work. Iceberg vs Parquet covers when that is enough.

Local-First Analytics covers the Hive-folder side of this in Chapter 4: partition_by on year and month, bucketing for high-cardinality columns, the rule to keep partition files at 50 MB or more, and why date folders without key=value names lose DuckDB's automatic pruning. Chapter 1 is free to read; the rest is on Amazon.

If you are laying out a lake for a real workload and want a second opinion on the partition scheme before it is hard to change, a Data Platform Audit is a week and a written roadmap you keep.

More in this series: Iceberg table maintenance, DuckLake vs Iceberg, Iceberg vs Parquet and lakehouse vs data warehouse.