Polars LazyFrames: The Speedup Is the Plan, Not the Syntax

I ported a feature job to Polars in an afternoon and it still died on a 12 GB container. Peak RSS was 14.1 GB. The input Parquet was 3.8 GB. The rewrite had been honest eager Polars: pl.read_parquet, then filter, then group_by, then another frame for the join.

Nothing ran until the first line had already pulled every column into memory. I had changed the spelling, not the plan. The OOM was the same OOM I already knew from pandas, just with a nicer API.

The lazy version of that job — scan_parquet, filters before any materialize, collect(engine="streaming") — finished in 9 minutes at 3.2 GB peak. Same outputs, after I stopped calling map_elements on a status string.

That is the production lesson. Polars is not a faster pandas import. It is a query engine you have to feed a query.

Eager runs now. Lazy builds a plan

pl.read_parquet is a load. pl.scan_parquet is a promise. Nothing hits disk until collect(), and the optimizer can push predicates and projections into the scan. If you want the mechanical pandas translations, they are in the pandas to Polars migration post. This one is about not wasting that migration.

import polars as pl
from datetime import date

# Eager: whole file, then you throw columns away.
# df = pl.read_parquet("s3://raw/orders/*.parquet")

out = (
    pl.scan_parquet("s3://raw/orders/*.parquet")
      .filter(pl.col("order_date") >= date(2026, 1, 1))
      .filter(pl.col("status") == "shipped")
      .select("order_id", "region", "revenue", "cost")
      .with_columns(
          (pl.col("revenue") - pl.col("cost")).alias("margin")
      )
      .group_by("region")
      .agg(
          pl.col("margin").sum().alias("margin"),
          pl.col("order_id").n_unique().alias("orders"),
      )
      .sort("margin", descending=True)
      .collect(engine="streaming")
)

Read explain() before you trust it. If PROJECT still lists forty columns, something in the chain blocked pushdown — usually a select(*), a Python UDF, or an eager break in the middle.

flowchart TD
  eager["pl.read_parquet"] --> ram["Full file in RAM"]
  ram --> oom["OOM on a 12 GB box"]
  lazy["pl.scan_parquet"] --> plan["Query plan"]
  plan --> opt["Pushdown and prune"]
  opt --> stream["Streaming collect or sink"]
  stream --> out["Result or Parquet"]

The filter has to reach the reader. An eager load plus a later filter is still a full load.

Apply loops lose on purpose

map_elements is a Python loop with extra ceremony. It serializes, it blocks the optimizer, and it is why a “Polars rewrite” can be slower than the pandas it replaced. Status maps, tier buckets, and simple string fixes are when/then or a replace expression.

If the logic cannot be an expression, Polars will not save you. Leave that job on pandas, or isolate the UDF at the edge after a lazy reduce. The memory post is still the right first move when the real problem is object-dtype strings and a merge copy, not the library name.

Polars vs DuckDB vs Spark

I get asked to pick a winner. The honest split is the working set, not the logo.

Polars when the job is a dataframe pipeline you already think in columns, one machine, and you want expressions rather than SQL. Streaming collect and sink_parquet cover a lot of “bigger than RAM” cases if the plan stays in the supported set.

DuckDB when the job is SQL over Parquet you do not want to cluster. Joins and aggregations spill. One writer per file. Fine.

Spark when shuffle is large, you need mid-job fault tolerance, or several hundred GB with a nasty join shape. The PySpark tutorial is the cluster end of this spectrum. Do not stand up a cluster for 40 GB of Parquet and a group-by.

Rewriting every pandas script is a waste. The 20 percent of jobs that dominate runtime and memory pay for the work. A 40,000-row QA notebook can stay on pandas until someone has a reason.

Pitfalls

Declaring victory after an eager port. You got maybe 30 percent. The plan is the rest.

collect() inside a loop. Each call is a full optimize-and-run. Build one plan, or pl.collect_all.

Streaming as a slogan. Some operations fall back to in-memory. Read the plan. If it says it will not stream, believe it.

NaN vs null after a pandas Parquet write. Aggregations skip null and propagate NaN. Normalize once at the edge.

Forcing Polars on a job that is a SQL scan plus one aggregate. DuckDB will do that in five lines and you will stop arguing about LazyFrames.

What this means for your pipelines

I use lazy Polars on the jobs that used to need a 64 GB box for a 4 GB file. I do not use it as a religion. Scan, filter, project, collect or sink, and keep a differential check against the pandas output until the numbers match.

If the working set and the shuffle outgrow one machine, that is Spark. If the job is already SQL, that is DuckDB. If the script is small and the ecosystem wants a pandas DataFrame, leave it. The OOM I opened with was not a missing library. It was a plan I never built.