parquet vs csv file size comparison

On the same 10 million rows, a CSV around 635 MB writes to roughly 29 MB of Parquet, about 22 times smaller. Do not treat that as a rate you will get. The ratio is decided almost entirely by column types: booleans compress 50 to 100 times, timestamps 10 to 50 times through delta encoding, low-cardinality strings 10 to 30 times through dictionary encoding, and float measurements only 2 to 4 times. A table of mostly high-variance floats lands nearer 4x. A table of IDs, flags and timestamps lands nearer 20x. Real NYC taxi data goes from 2.1 GB of CSV to 180 MB of Parquet, about 12 times.

parquet vs csv performance

The honest answer depends on what you compare against and whether the file is already in the page cache. Summing one column of a 10 million row file, CSV takes 1 to 2 seconds cold and Parquet 0.1 to 0.4, so roughly 4 to 15 times faster end to end while reading about 20 times less data. Against a modern multi-threaded CSV reader on a warm cache that gap collapses to about 1.5 to 2 times. Against pandas.read_csv it widens to 20 to 50 times. The I/O reduction is the part that holds everywhere, and it is the part that matters when the file sits in object storage and you pay per byte scanned.

parquet compression ratio

Expect 4 to 25 times depending on your columns, not a single headline number. Event logs heavy with timestamps land around 15 to 25 times, transactional data full of IDs around 10 to 15, sensor data of high-variance floats around 4 to 8. Parquet picks the encoding per column by itself: run-length encoding for long runs of the same value, dictionary encoding for repeated strings, delta encoding for timestamps, then a general compressor on top. You do not configure any of it, which is why guessing your ratio from someone else's blog post is a waste of time. Write a sample and measure.

apache arrow vs parquet

They are not alternatives. Parquet is a file format for disk, optimised for size and for reading a subset of columns. Arrow is a memory format, optimised for a CPU to compute over and for two tools to share without copying. A normal pipeline uses both: you store Parquet, and when you read it the bytes are decoded into Arrow buffers in RAM. The reason to care is that when every tool in a chain speaks Arrow, handing data between them passes pointers instead of copying gigabytes.

Parquet vs CSV: what you actually save, and why

"Just use Parquet" is the most repeated advice in data engineering and the least explained. It is good advice. But if you do not know why it wins, you cannot tell when it will not, and you end up either converting files that should have stayed CSV or being surprised when your ratio is 3x instead of the 20x you read about.

There are three mechanical reasons, and none of them need a paper.

1. It reads only the columns you asked for

A CSV stores a row at a time:

id,name,email,signup_date,revenue
1,Alice,[email protected],2026-01-15,127.50
2,Bob,[email protected],2026-01-16,340.00

To total the revenue, the reader walks the whole file. Every name, every email, every date, all read off disk and parsed to reach one column.

Parquet stores a column at a time. SELECT SUM(revenue) reads the revenue column and nothing else. Most analytical queries touch 10 to 20% of the columns, so they read 10 to 20% of the file. CSV always reads 100%.

That is the entire trick behind the headline benchmarks, and it is why the win grows as your tables get wider.

2. Similar things sit next to each other, so they compress

Compression works on patterns. A CSV row is a string, a date, a float and two integers jammed together, which is close to noise. A Parquet column is ten million floats, or ten million dates, or the word "USA" repeated ten thousand times.

Parquet picks an encoding per column with no input from you:

Run-length encoding turns ten thousand repeats of "USA" into the pair ["USA", 10000].
Dictionary encoding maps "San Francisco" to 0 and "New York" to 1, then stores the small integers.
Delta encoding turns hourly timestamps [1704067200, 1704070800, 1704074400] into [1704067200, +3600, +3600].

Then a general compressor runs over the result.

This is why asking "what compression ratio will I get?" has no answer without your schema. What decides it:

Column type	Typical ratio	Why
Booleans	50-100x	Bit-packing
Timestamps	10-50x	Delta encoding
Low-cardinality strings	10-30x	Dictionary encoding
Integers, IDs, counts	5-20x	Dictionary and bit-packing
High-cardinality strings	2-5x	Little structure to exploit
Floats, measurements	2-4x	Little structure to exploit

A table of flags, IDs and timestamps compresses enormously. A table of sensor readings barely compresses at all. Both are normal.

Real numbers, so you have an anchor: ten million rows of synthetic user data go from about 635 MB of CSV to about 29 MB of Parquet, roughly 22x. Ten million NYC taxi rides go from 2.1 GB to 180 MB, roughly 12x. Event logs usually land at 15 to 25x, transactional data at 10 to 15x, float-heavy sensor data at 4 to 8x.

Measure your own. It takes one write.

3. The layout suits how CPUs actually work

A modern CPU processes several values per instruction. Give it a contiguous run of floats and it sums eight at a time. Give it rows, and it parses one value, then the next, one at a time.

DuckDB combines that with vectorised execution: it works through a column in chunks of about 2,048 values so the hot loop stays in cache, then adds multiple cores. An average over ten million rows finishes in around 30 milliseconds. Not because DuckDB is exotic, but because the data was already in the shape the CPU wanted.

The honest part: where the speed number comes from

Most Parquet benchmarks are quietly unfair, so here is the shape of the real result.

Summing one column of a ten million row file, cold cache: CSV takes 1 to 2 seconds, Parquet 0.1 to 0.4. Call it 4 to 15 times faster, reading roughly 20 times less data.

Two things move that number a lot:

The page cache. Run the benchmark twice and the operating system has the file in RAM, which hides most of the I/O cost. The second run flatters CSV enormously.

What you compare against. A modern multi-threaded, SIMD-accelerated CSV reader on a warm cache narrows the gap to about 1.5 to 2 times. Compare against pandas.read_csv and it widens to 20 to 50 times. Same files, same machine, a twenty-fold difference in the claim depending on which reader you picked.

The number that survives all of it is the I/O reduction. That is also the number that turns into money when the file is in object storage and you are billed per byte scanned.

Arrow is the other half, and it is not a competitor

Parquet is a disk format. Arrow is a memory format. You use both, usually without noticing.

The reason to notice is copying. A pipeline that reads Parquet into pandas, transforms, then hands the result to DuckDB pays for a format conversion at each boundary: fresh memory allocated, CPU spent translating layouts, garbage collector pressure. On a gigabyte of Parquet that is roughly 2.3 seconds through the copying path against 0.4 through an Arrow-native one, at about half the peak memory.

When every hop speaks Arrow, you hand over a pointer instead of the bytes.

It stops mattering for single-tool pipelines and for small data, where the copy costs nothing worth measuring.

When CSV is still right

Parquet is not free. It is a binary format, so you cannot inspect it with less, diff it usefully, or append a line to it.

CSV stays the better answer for files of a few megabytes, for handing data to someone non-technical once, and for append-heavy logs where a writer adds a line at a time. In those cases the conversion costs more than it returns.

The moment the data is read more than once, filtered, or has columns you usually skip, the balance tips and does not tip back.

The short version

Parquet wins because it skips columns, compresses runs of similar values, and lays data out the way CPUs like.
Your compression ratio is a property of your column types. Expect 4 to 25x, and measure rather than assume.
Speed comparisons swing by an order of magnitude depending on the cache and the CSV reader. The I/O reduction is the honest number.
Arrow complements Parquet rather than competing with it: one for disk, one for memory.
For small, one-shot or append-heavy files, CSV is still fine.

The full treatment, with runnable benchmarks for each of these and the file-layout decisions that come next, is in my book Local-First Analytics. If you want the practical follow-on, running DuckDB on your own infrastructure covers what to do with these files once you have them, and do you actually need a data warehouse covers whether you needed the warehouse at all.