Pandas to Polars: What Translates, What Does Not, and What Breaks Quietly

I migrated a nightly feature pipeline from pandas to polars last year. Runtime went from 47 minutes to 6, and peak memory from 38 GB to 9. Those numbers are real, and they are also the reason people migrate for the wrong reasons and get disappointed.

The first version of that migration was a mechanical port — pd.read_parquet became pl.read_parquet, df.groupby became df.group_by, and so on. It ran in 31 minutes. Useful, not transformative. The 6-minute number came two weeks later, after rewriting the pipeline as one lazy query so the optimizer could push filters into the Parquet scan and skip columns entirely.

That is the honest framing. Polars is faster than pandas at the same operation, mostly from multithreading and better memory layout. But the order-of-magnitude wins come from the lazy engine, and the lazy engine only helps if you restructure your code to give it a whole query to optimize.

This post is what I wish I had known on day one: the mental model shift, the mechanical translations, the semantics that differ silently, and the four things that genuinely do not translate.

The three mental model shifts

No index. Polars has no row index. A pandas codebase carries set_index, reset_index, reindex, loc on labels, and index-aligned arithmetic throughout, and all of it disappears. Joins become explicit on= arguments. Alignment becomes an explicit join. This deletes code, and it removes an entire class of bug where two series silently align on an index nobody checked.

Expressions, not values. pl.col("amount") * 2 is not a computation, it is a description of one. You build expressions and hand them to a context — select, with_columns, filter, group_by().agg() — which executes them, in parallel, against the frame. This is the single biggest syntactic difference and the source of most early frustration.

Lazy by default in production. scan_parquet returns a LazyFrame; nothing runs until .collect(). In between, the optimizer does projection pushdown, predicate pushdown, common subexpression elimination, and slice pushdown. This is where the big wins live.

flowchart TD
  eager["Eager: pl.read_parquet"] --> mem["Whole file into memory"]
  mem --> ops["Each operation materializes a result"]
  lazy["Lazy: pl.scan_parquet"] --> plan["Build a query plan"]
  plan --> opt["Optimizer: pushdown and pruning"]
  opt --> exec["Read only needed columns and row groups"]
  exec --> out["Collect final result"]

Eager polars beats pandas. Lazy polars is a different category, because the filter reaches the file reader.

Translating the common operations

Most day-to-day pandas has a direct equivalent. The pattern to internalize is that assign becomes with_columns and everything inside it is an expression.

import pandas as pd
import polars as pl

# --- pandas ---
df = pd.read_parquet("orders.parquet")
df = df[df["status"] == "shipped"]
df["margin"] = df["revenue"] - df["cost"]
df["margin_pct"] = df["margin"] / df["revenue"]
out = (
    df.groupby(["region", "product_id"])
      .agg(total=("margin", "sum"), orders=("order_id", "nunique"))
      .reset_index()
      .sort_values("total", ascending=False)
)

# --- polars, eager ---
df = pl.read_parquet("orders.parquet")
out = (
    df.filter(pl.col("status") == "shipped")
      .with_columns(
          (pl.col("revenue") - pl.col("cost")).alias("margin"),
      )
      .with_columns(
          # Second with_columns: margin must exist before we divide by it.
          (pl.col("margin") / pl.col("revenue")).alias("margin_pct"),
      )
      .group_by(["region", "product_id"])
      .agg(
          pl.col("margin").sum().alias("total"),
          pl.col("order_id").n_unique().alias("orders"),
      )
      .sort("total", descending=True)
)

The two separate with_columns calls matter. Expressions inside one with_columns all evaluate against the input frame in parallel, so a column created in the same call is not visible to its siblings. In pandas, chained assign sees prior keys; in polars you need a new context. This is the mistake I made most often in the first week, and it fails loudly with a ColumnNotFound, which is the good kind of failure.

The lazy version of the same pipeline is where the real work happens:

out = (
    pl.scan_parquet("s3://bucket/orders/*.parquet")   # nothing read yet
      .filter(pl.col("status") == "shipped")
      .filter(pl.col("order_date") >= date(2026, 1, 1))
      .with_columns((pl.col("revenue") - pl.col("cost")).alias("margin"))
      .group_by(["region", "product_id"])
      .agg(
          pl.col("margin").sum().alias("total"),
          pl.col("order_id").n_unique().alias("orders"),
      )
      .sort("total", descending=True)
      .collect()          # optimizer runs here, then execution
)

# Read the plan before you trust it.
print(plan.explain())     # shows PROJECT and SELECTION pushdown

explain() is the polars equivalent of reading a query plan, and you should read it for every pipeline you migrate. If the projection line shows every column instead of the six you use, a select("*") or an apply somewhere is blocking pushdown, and you are paying full-width reads for nothing.

Window functions and the over() pattern

Pandas expresses group-relative computation with groupby().transform(), which is one of the slowest paths in the library. Polars uses .over(), and it is fast.

# pandas: transform broadcasts the group aggregate back to rows
df["region_total"] = df.groupby("region")["revenue"].transform("sum")
df["pct_of_region"] = df["revenue"] / df["region_total"]
df["rank_in_region"] = (
    df.groupby("region")["revenue"].rank(method="first", ascending=False)
)

# polars: over() does the same in one pass, multithreaded
df = df.with_columns(
    (pl.col("revenue").sum().over("region")).alias("region_total"),
    (pl.col("revenue") / pl.col("revenue").sum().over("region"))
        .alias("pct_of_region"),
    # rank within group; descending revenue, ties broken by order
    (pl.col("revenue").rank("ordinal", descending=True).over("region"))
        .alias("rank_in_region"),
)

# Top 2 per region without a groupby-apply-concat dance
top2 = df.filter(
    pl.col("revenue").rank("ordinal", descending=True).over("region") <= 2
)

That last filter is the one that converts people. The pandas equivalent — groupby().apply() returning a slice of each group — is slow, memory-hungry, and returns a frame with an index shape that depends on the pandas version. The polars form is one expression and it parallelizes across groups.

The semantics here mirror SQL window functions closely enough that the reasoning transfers directly; if the frame and tiebreaker concepts are fuzzy, the window frames deep dive covers the same ground in SQL, and the determinism warning about ties applies identically in polars.

Null versus NaN, and other silent differences

This is the category that produces wrong numbers rather than exceptions, so it deserves care.

Null is not NaN. Polars distinguishes missing (null) from the floating-point value NaN. Pandas historically used NaN for both, and np.nan propagated through arithmetic while pandas aggregations skipped it. In polars, null is skipped by aggregations and NaN propagates. A column that arrived from pandas with NaN markers for missing data will produce NaN totals in polars until you convert:

df = df.with_columns(
    # Treat NaN as missing, the way pandas did.
    pl.when(pl.col("amount").is_nan())
      .then(None)
      .otherwise(pl.col("amount"))
      .alias("amount")
)

Integer columns stay integers. Pandas silently upcasts an int column to float when a null appears. Polars keeps Int64 with a null mask. This is better, and it means comparisons and joins that used to work by float coincidence now need real type alignment.

Joins on mismatched types fail loudly. Pandas will happily join int64 to object and produce zero matches. Polars raises. Every migration I have done surfaced at least one join that had been silently returning nothing.

Sort stability and null placement. Polars puts nulls last by default in sort; pandas puts them last too, but nulls_last is explicit in polars and worth setting. Sort is not stable by default — pass maintain_order=True if you depend on it, and if you do depend on it, add a real tiebreaker column instead.

group_by does not sort. Pandas sorts group keys by default; polars does not, for speed. If downstream code or a test assumes ordered output, add maintain_order=True or an explicit .sort(). This bites in tests before it bites in production.

What does not translate

Four things, and being honest about them saves weeks.

Row-wise apply. df.apply(fn, axis=1) has no fast polars equivalent, because there is no fast version of calling Python per row anywhere. map_elements exists and is roughly as slow as pandas, plus a warning. If the logic cannot be expressed with when/then/otherwise, string expressions, or struct field access, polars will not save you. This is the same principle as per-row UDFs in SQL — the engine can only vectorize what you express as a set operation.

The pandas ecosystem. Anything expecting a DataFrame — scikit-learn pipelines built on column names, statsmodels, seaborn, most plotting libraries, many warehouse connectors. The escape hatch is cheap: df.to_pandas() at the boundary, ideally with use_pyarrow_extension_array=True to avoid a copy. Do the heavy transforms in polars and convert at the last step.

MultiIndex. There is no equivalent, by design. Hierarchical results become regular columns, and pivoted output becomes either long form or explicitly named columns. In my experience this deletes more code than it adds, but it is not a mechanical port.

inplace=True and mutation. Polars frames are immutable in practice; every operation returns a new frame. Code written around in-place mutation and side effects needs restructuring into a chain. Again, usually an improvement, and definitely not a find-and-replace.

Migrating without a big-bang rewrite

The strategy that worked for me:

1. Profile first. Find the jobs that dominate runtime and memory. In every codebase I have looked at, a handful of jobs account for most of the cost. Migrate those.
2. Convert at the edges. Rewrite the internals of one job in lazy polars and keep to_pandas() at the boundary so callers and tests do not change.
3. Diff the output. Run both versions on the same input and compare with a tolerance-aware assertion, column by column.

def assert_frames_equivalent(pandas_out, polars_out, keys, tol=1e-9):
    """Compare a legacy pandas result against the polars rewrite."""
    left = pl.from_pandas(pandas_out).sort(keys)
    right = polars_out.sort(keys)

    assert left.height == right.height, (
        f"row count {left.height} != {right.height}"
    )
    assert set(left.columns) == set(right.columns), (
        f"column mismatch: {set(left.columns) ^ set(right.columns)}"
    )

    for col in left.columns:
        if left[col].dtype.is_numeric():
            # Nulls compare equal; NaN does not, so normalize first.
            diff = (left[col].fill_null(0) - right[col].fill_null(0)).abs().max()
            assert diff is None or diff < tol, f"{col} differs by {diff}"
        else:
            assert left[col].to_list() == right[col].to_list(), f"{col} differs"

Run that in CI against a frozen sample for a couple of weeks before you delete the pandas path. The differences it catches are almost always null handling, integer-versus-float, or an ordering assumption — the exact list from the previous section. If you do not already have a place to put a test like this, the pipeline testing guide covers where differential checks fit alongside the rest of the suite.

4. Then restructure for lazy. Only after correctness is established, collapse the eager steps into one scan to collect chain and read explain() to confirm pushdown.

Step 4 is the one that produces the headline numbers, and it is also the one that changes behavior most, which is why it goes last.

Pitfalls

Porting eagerly and declaring victory. read_parquet plus eager operations gets you maybe 30 to 40 percent. scan_parquet with filters before collect() is where the multiples are.

Expecting sibling columns inside one with_columns. All expressions in a single call see the input frame only. Chain a second with_columns when one column depends on another.

Leaving map_elements in a hot path. It is a Python loop with extra ceremony, and it blocks the optimizer. Every one I have removed was replaceable with when/then/otherwise or a string expression.

Assuming group_by output order. Unordered by default. Tests that pass locally on small data will fail nondeterministically on larger inputs with more threads.

Ignoring the NaN/null split. Aggregations skip nulls and propagate NaN. A pandas-produced Parquet file with NaN for missing values yields NaN totals, which at least is visible; the worse case is a mean computed over a different denominator.

collect() inside a loop. Each call runs a full optimization and execution. Build one plan, or use pl.collect_all() for several independent plans so they share threads.

Assuming lower memory automatically. collect() materializes the full result. For outputs bigger than memory you need collect(streaming=True) or sinks, not just laziness.

FAQ

Is polars always faster than pandas?

No. For small frames — under roughly 100,000 rows — the difference is negligible and startup overhead can favor pandas. Polars wins on medium-to-large single-machine data, especially with many columns, string-heavy operations, or group-bys with many groups.

Do I need to migrate everything?

No, and you should not. Migrate the jobs that hurt: the ones that run long, spill to disk, or get scaled up to expensive instance types. Keep pandas where the ecosystem matters and the data is small.

Can polars replace Spark?

For anything that fits on one large machine — and that now includes hundreds of gigabytes with the streaming engine — often yes, with far less operational overhead. Beyond that, or where you need cluster-level fault tolerance mid-job, Spark still wins. The same reasoning applies to DuckDB for local pipelines: single-node tools have eaten a lot of what used to require a cluster.

How does polars handle out-of-core data?

collect(streaming=True) processes in batches for supported operations, and sink_parquet writes results without materializing them. Coverage is not complete — some operations fall back to in-memory — so check the plan rather than assuming.

Should I use the lazy or eager API?

Lazy for anything in production. Eager for interactive exploration, where you want to see intermediate results. The APIs are close enough that moving between them is mostly adding a scan_ and a .collect().

What about data quality checks in polars?

They translate well — expressions make row-level and aggregate assertions concise, and lazy evaluation means a check can run in the same scan as the transform. The framework-level thinking in the data quality checks post applies unchanged.

What this means for your pipelines

The real benefit of polars is not the benchmark. It is that a job which needed a 64 GB instance now runs on 16 GB, which changes what you can run where — a nightly Spark job becomes a container, an hourly batch becomes a five-minute one, and the orchestration around it gets simpler because there is no cluster to wait for.

Get there by migrating the expensive tail first and leaving the rest alone. Write the lazy version, read explain() to confirm the filters reached the scan, and keep a differential test against the pandas output until you trust it. The bugs you will find are predictable: null versus NaN, integer versus float, and an ordering assumption nobody wrote down.

The part that surprised me most was how much code disappeared. No index manipulation, no reset_index after every group-by, no inplace bookkeeping, no SettingWithCopyWarning suppression. The polars version of that feature pipeline is about 40 percent fewer lines than the pandas one, and the remaining lines say what the transformation is rather than how to keep the frame’s bookkeeping consistent. That has been worth as much as the runtime.