Topic
15 posts ·view all posts
PythonWhere the memory actually goes in a pandas job, how much dtype selection and Arrow-backed strings really save, and when to stop loading and start streaming.
PythonA production migration guide: the expression API, lazy frames and query optimization, null versus NaN semantics, and the pandas patterns with no polars equivalent.
SnowflakeA production comparison of Snowpark Python and plain SQL in Snowflake, covering lazy DataFrames, UDF serialization costs, and the four workloads where Python genuinely earns its place.
SQLDISTINCT, GROUP BY, ROW_NUMBER, QUALIFY, MERGE, and stream-side dedup compared on cost and semantics, plus how to pick the survivor row deterministically.
Apache SparkHow Spark executor memory is actually divided, why most OOMs are one oversized partition rather than a small cluster, and the six failures I can now diagnose from the error text alone.
BigQueryWhat BigQuery reads before it reads your data: partition metadata, Capacitor block statistics, and the specific query shapes that silently turn a pruned scan into a full one.
SnowflakeHow to pick a Snowflake warehouse size from evidence instead of intuition, using spilling, queueing, and scaling tests to decide between a bigger warehouse and more clusters.
Apache SparkOne task at 40 minutes while 399 finished in 20 seconds is skew. Here is how I find the offending key, and the four fixes ranked by how much they cost me.
SQLWhat actually happens to a CTE in Postgres, Snowflake, and BigQuery, when it blocks predicate pushdown, and when referencing it twice costs you two scans.
Apache SparkA shuffle is a disk write, a network fetch, and a sort you did not ask for. Here is what it costs, how to spot it in the Spark UI, and the query rewrites that delete it.
SQLHow window frames actually evaluate, why RANGE and ROWS give different answers on ties, and six window patterns that delete self-joins from your models.
SnowflakeHow the search access path works, what it costs to build and maintain, which predicates it serves, and when clustering keys are the better buy.
SnowflakeClustering depth, what automatic reclustering actually costs, and a decision framework for clustering keys vs Search Optimization vs doing nothing.
SQLFive SQL habits that scan more than they should: SELECT *, functions on filter columns, DISTINCT as a bug fix, OR-joins, and per-row UDF calls.
SnowflakeHow I read the operator tree: TableScan bytes, partition pruning, local and remote spilling, and the exchange steps that quietly dominate runtime.