Polars LazyFrames: The Speedup Is the Plan, Not the Syntax

I ported a feature job to Polars in an afternoon and it still died on a 12 GB container. Peak RSS was 14.1 GB. The input Parquet was 3.8 GB. The rewrite had been honest eager Polars: pl.read_parquet, then filter, then group_by, then another frame for the join.

Nothing ran until the first line had already pulled every column into memory. I had changed the spelling, not the plan. The OOM was the same OOM I already knew from pandas, just with a nicer API.

The lazy version of that job — scan_parquet, filters before any materialize, collect(engine="streaming") — finished in 9 minutes at 3.2 GB peak. Same outputs, after I stopped calling map_elements on a status string.

That is the production lesson. Polars is not a faster pandas import. It is a query engine you have to feed a query.

Eager runs now. Lazy builds a plan

pl.read_parquet is a load. pl.scan_parquet is a promise. Nothing hits disk until collect(), and the optimizer can push predicates and projections into the scan. If you want the mechanical pandas translations, they are in the pandas to Polars migration post. This one is about not wasting that migration.

import polars as pl
from datetime import date

# Eager: whole file, then you throw columns away.
# df = pl.read_parquet("s3://raw/orders/*.parquet")

out = (
    pl.scan_parquet("s3://raw/orders/*.parquet")
      .filter(pl.col("order_date") >= date(2026, 1, 1))
      .filter(pl.col("status") == "shipped")
      .select("order_id", "region", "revenue", "cost")
      .with_columns(
          (pl.col("revenue") - pl.col("cost")).alias("margin")
      )
      .group_by("region")
      .agg(
          pl.col("margin").sum().alias("margin"),
          pl.col("order_id").n_unique().alias("orders"),
      )
      .sort("margin", descending=True)
      .collect(engine="streaming")
)

Read explain() before you trust it. If PROJECT still lists forty columns, something in the chain blocked pushdown — usually a select(*), a Python UDF, or an eager break in the middle.

flowchart TD
  eager["pl.read_parquet"] --> ram["Full file in RAM"]
  ram --> oom["OOM on a 12 GB box"]
  lazy["pl.scan_parquet"] --> plan["Query plan"]
  plan --> opt["Pushdown and prune"]
  opt --> stream["Streaming collect or sink"]
  stream --> out["Result or Parquet"]

The filter has to reach the reader. An eager load plus a later filter is still a full load.

Apply loops lose on purpose

map_elements is a Python loop with extra ceremony. It serializes, it blocks the optimizer, and it is why a “Polars rewrite” can be slower than the pandas it replaced. Status maps, tier buckets, and simple string fixes are when/then or a replace expression.

If the logic cannot be an expression, Polars will not save you. Leave that job on pandas, or isolate the UDF at the edge after a lazy reduce. The memory post is still the right first move when the real problem is object-dtype strings and a merge copy, not the library name.

Polars vs DuckDB vs Spark

I get asked to pick a winner. The honest split is the working set, not the logo.

Polars when the job is a dataframe pipeline you already think in columns, one machine, and you want expressions rather than SQL. Streaming collect and sink_parquet cover a lot of “bigger than RAM” cases if the plan stays in the supported set.

DuckDB when the job is SQL over Parquet you do not want to cluster. Joins and aggregations spill. One writer per file. Fine.

Spark when shuffle is large, you need mid-job fault tolerance, or several hundred GB with a nasty join shape. The PySpark tutorial is the cluster end of this spectrum. Do not stand up a cluster for 40 GB of Parquet and a group-by.

Rewriting every pandas script is a waste. The 20 percent of jobs that dominate runtime and memory pay for the work. A 40,000-row QA notebook can stay on pandas until someone has a reason.

Sink when the result is the file

If the output is Parquet, do not collect into a Python object just to write it.

(
    pl.scan_parquet("s3://raw/orders/*.parquet")
      .filter(pl.col("order_date") >= date(2026, 1, 1))
      .select("order_id", "region", "margin")
      .sink_parquet("s3://curated/orders_2026.parquet")
)

sink_* keeps the engine in charge. collect then write_parquet is a second copy in RAM.

I keep a pandas vs Polars checksum on a sample until the first production week is boring. Null vs NaN is the usual mismatch.

More decision context: polars vs pandas. Hub: Polars.

FAQ

Why is lazy slower than eager on a tiny file? Plan overhead. Under a few million rows, pandas or eager Polars is fine. Lazy pays when IO and unused columns dominate.

Can I mix pandas in the middle? .collect().to_pandas() is an eager wall. Do it once at the edge for sklearn, not between two scans.

Streaming collect failed / fell back. Read explain(). Some joins and sorts will not stream. Then you need a bigger box, DuckDB spill, or Spark.

Hive-style partitions on S3? scan_parquet with a hive partition filter is the point of lazy. If you read_parquet the prefix, you already paid.

Pitfalls

Declaring victory after an eager port. You got maybe 30 percent. The plan is the rest.

collect() inside a loop. Each call is a full optimize-and-run. Build one plan, or pl.collect_all.

Streaming as a slogan. Some operations fall back to in-memory. Read the plan. If it says it will not stream, believe it.

NaN vs null after a pandas Parquet write. Aggregations skip null and propagate NaN. Normalize once at the edge.

Forcing Polars on a job that is a SQL scan plus one aggregate. DuckDB will do that in five lines and you will stop arguing about LazyFrames.

What this means for your pipelines

I use lazy Polars on the jobs that used to need a 64 GB box for a 4 GB file. I do not use it as a religion. Scan, filter, project, collect or sink, and keep a differential check against the pandas output until the numbers match.

If the working set and the shuffle outgrow one machine, that is Spark. If the job is already SQL, that is DuckDB. If the script is small and the ecosystem wants a pandas DataFrame, leave it. The OOM I opened with was not a missing library. It was a plan I never built.