is polars faster than pandas

Usually yes, and the reason matters more than the number. On the same machine, a filter plus five transforms over tens of millions of rows runs in seconds with Polars against tens of seconds with pandas, and the memory gap is wider than the time gap because pandas materialises every intermediate frame while Polars does not. But pandas 2.x with the PyArrow dtype backend closes a good part of the wall-clock difference, so "ten times faster" is marketing rather than a measurement. The dependable wins are memory and predictability on long chains of transforms.

polars vs pandas performance

Three mechanical differences drive it. Lazy execution lets Polars see the whole pipeline before running any of it, so it can push filters and column selection down into the Parquet reader and never load what it will discard. Being Arrow-native means it hands data to DuckDB or PyArrow without copying. And it uses every core by default rather than through an apply loop. Pandas is eager, so each step reads or walks the frame again and allocates a new one.

polars lazy evaluation

Use `scan_parquet` rather than `read_parquet` and nothing runs until you call `collect`. In between, Polars builds a plan it can optimise: projection pushdown reads only the columns your query mentions, and predicate pushdown applies filters at the file level so rows are never loaded. One trap is worth knowing. `lazy_df.collect().head(100)` runs the entire query and then throws almost all of it away; `lazy_df.head(100).collect()` pushes the limit into the plan and stops scanning early. Storing intermediate LazyFrames in variables is fine. Calling collect in the middle is what breaks the optimisation.

polars vs pandas vs duckdb

Polars and DuckDB are in the same performance league and both are far ahead of pandas; which of the two wins a given query flips with cardinality, join shape and how much work can be pushed into the Parquet reader. Choose on the task, not the benchmark. DuckDB is better for aggregations, window functions and ad-hoc questions, because SQL says those things in one line. Polars is better for row-level transformation logic, deduplication with real business rules, and long chains that would become an unreadable stack of CTEs. In a real pipeline you use both and pass data between them through Arrow, which costs nothing.

Polars vs Pandas: the three differences that matter

Every Polars post opens with a benchmark where it beats pandas by a factor of ten. That number is real on some workload somewhere, and it is the least useful thing about the library.

Three differences actually cause the gap. Once you know them, you can predict when Polars will help you and when it will not, which is worth more than a chart.

1. It builds the plan before it does the work

Pandas is eager. Each line runs when it is read:

df = pd.read_parquet("transactions.parquet")   # whole file, all columns
df = df[df["amount"] > 0]                      # walk it, allocate a new frame
df["fee"] = df["amount"] * 0.029               # walk it again
df = df.drop_duplicates(["transaction_id"])    # and again

The file is fully loaded before the first filter, including columns you never mention again. Every step allocates.

Polars, given scan_ instead of read_, does nothing at all until you ask:

result = (
    pl.scan_parquet("transactions.parquet")
    .filter(pl.col("amount") > 0)
    .with_columns((pl.col("amount") * 0.029).alias("fee"))
    .unique(subset=["transaction_id"])
    .collect()          # only now does anything run
)

By the time collect fires, Polars knows the whole pipeline, so it can do two things pandas structurally cannot:

Projection pushdown. Your query mentions three columns, so it reads three columns off disk. The other forty are never touched.
Predicate pushdown. The filter goes down to the file reader, so non-matching rows are never materialised at all.

This is also where Parquet and Polars compound. Columnar storage makes reading three of forty-three columns cheap, and lazy execution is what tells the reader it only needs three.

One trap worth knowing, because it silently undoes all of it:

lazy_df.collect().head(100)   # runs the whole query, throws away almost everything
lazy_df.head(100).collect()   # pushes the limit into the plan, stops scanning early

If you learned Polars from older material you may reach for .fetch(n). That is deprecated in Polars 1.x; .head(n).collect() is the current form.

2. No hidden copies, because everything is Arrow

Polars stores data in Arrow buffers, which is the same layout DuckDB and PyArrow use. When you pass a frame between them, they exchange pointers rather than bytes.

That sounds like an implementation detail until you count the boundaries in a real pipeline. Read Parquet into pandas with the NumPy backend, transform, hand the result to DuckDB, and each crossing allocates fresh memory the size of your data, spends CPU translating layouts, and adds garbage collector pressure. On a gigabyte of decoded data that is roughly 2.3 seconds through the copying path against 0.4 through the Arrow one, and about half the peak memory.

Memory is the more reliable win, and it is the one people notice on real jobs. Pandas materialises every intermediate frame, so a chain of eight transforms over a large table holds several copies at once. That is the difference between a job that fits in your container and one that gets killed.

3. It uses every core without being asked

Pandas is largely single-threaded, and the usual escape hatch is .apply(), which is a Python-level loop wearing a costume. Polars parallelises across cores by default, including the Parquet read, the filter and the aggregation.

You do not configure it, which is the point.

The honest part about the speed claims

Here are anecdotal timings from a laptop, which is what most comparison posts are, stated plainly:

Task	Pandas	Polars	DuckDB
Load 50M rows from CSV	23s	8s	6s
Filter plus five transforms	18s	3s	not its job
Group by and aggregate	12s	2s	1s
Complex join	45s	9s	5s

Treat that as "the first column is a different league from the other two", not as a ranking. Two things soften it. Pandas 2.x with dtype_backend="pyarrow" is much closer than pandas 1.x was, so a comparison against an old pandas overstates the gap. And on published benchmarks Polars and DuckDB trade places constantly depending on cardinality, join shape and how much of the plan can be pushed into the reader.

The durable claim is not "Polars is ten times faster". It is that Polars holds far less memory on chained transforms and stays predictable as the chain grows.

Where DuckDB is the better answer

Reaching for Polars for everything is a mistake, and so is reaching for SQL for everything.

DuckDB suits aggregations, window functions like LAG and ROW_NUMBER, querying Parquet as it sits, and anything ad-hoc. SQL expresses those in one line. Polars expresses them in five.

Polars suits row-level transformation logic across many columns, deduplication with real business rules, and pipelines with ten or more steps. This is where SQL turns into a stack of nested CTEs that nobody can debug three months later.

In practice a pipeline uses both, and the handoff is free because they share Arrow. Complex SQL in DuckDB, transformation logic in Polars, back to DuckDB for serving. No conversion, no copy.

When to stay on pandas

Pandas is not the villain in this story. If your data fits in memory with room to spare, if your team knows the API, and if the job takes two seconds either way, switching buys you nothing and costs you a rewrite. Pandas also still has the deeper ecosystem for plotting, statistics and machine learning glue.

Change when a job is being killed for memory, when a transformation chain has grown past readable, or when you are handing data between tools often enough that the copying shows up in the profile.

The short version

Lazy execution is the big one: it lets the engine skip columns and rows before they are ever read.
Arrow-native means no copies at tool boundaries, and memory is a more reliable win than wall-clock time.
Parallelism comes free instead of through .apply().
Polars and DuckDB are in the same league. Pick by task: transforms in Polars, aggregations and windows in DuckDB.
Pandas is fine until memory or readability breaks. Switch on a symptom, not on a benchmark.

The longer version, including the chaining patterns that stay readable at twenty steps and a worked reconciliation pipeline with validation, is in my book Local-First Analytics. If you want the storage half of this story, Parquet vs CSV covers why the files these read are shaped the way they are.