All stacks · Compute & processing
Distributed processing with PySpark and Spark SQL.
Apache Spark🟡IntermediateLearn the PySpark DataFrame API, understand lazy evaluation and partitions, and avoid the classic mistakes that make Spark jobs slow.
Apache Spark🟡IntermediateThe four things that decide whether a Spark stream survives a year in production: trigger choice, checkpoint discipline, watermark math, and what exactly-once really guarantees.
Apache Spark🟡IntermediateHow Spark executor memory is actually divided, why most OOMs are one oversized partition rather than a small cluster, and the six failures I can now diagnose from the error text alone.
Apache Spark🟡IntermediateOne task at 40 minutes while 399 finished in 20 seconds is skew. Here is how I find the offending key, and the four fixes ranked by how much they cost me.
Apache Spark🟡IntermediateA shuffle is a disk write, a network fetch, and a sort you did not ask for. Here is what it costs, how to spot it in the Spark UI, and the query rewrites that delete it.
compare🟢BeginnerA pragmatic comparison: Databricks for Lakehouse/Spark/ML vs BigQuery for Serverless/SQL-native analytical workflows on GCP.
Apache Flink🟡IntermediateA production comparison of Flink and Spark Structured Streaming: how each one holds state, what their latency floors really are, and why the honest deciding factor is usually your existing batch stack.
Iceberg🟡IntermediateWhat each file format is actually built for — columnar analytics, Hive-native storage, and row-oriented streaming records — plus the compression, schema evolution, and small-file realities that decide your bill.
Microsoft Fabric🟢BeginnerOneLake shortcuts are not a copy job, capacity units are shared, and a Power BI refresh can queue a Spark notebook. Fabric is not rebranded Synapse.
Polars🟢BeginnerEager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.
Scala🟢BeginnerA typed Dataset[T] job caught a column I had been silently dropping in PySpark. You still have to read Scala; you do not have to write every job in it.
Spark is still the engine that processes more data than anything else in the industry — and the framework where the gap between "it runs" and "it runs well" is widest. Shuffles, skew, and memory spills separate ten-minute jobs from three-hour ones on identical clusters.
These tutorials treat Spark as a distributed SQL engine first: how the optimizer plans your query, where shuffles come from, how to read the Spark UI without dread, and the handful of configuration knobs that actually matter in PySpark jobs.
For pure SQL transformation on warehouse tables, often not. Spark earns its place for ML pipelines, semi-structured and unstructured data, streaming, and cost-sensitive heavy lifting on lake storage where warehouse compute would be expensive.
In order of likelihood: too many or too few shuffle partitions, data skew concentrating work in one task, spilling to disk from undersized memory, small-file storms on read, or a join that should have been broadcast. The Spark UI stage timeline tells you which one in about two minutes.
A shuffle redistributes rows across the cluster by key — required by wide operations like groupBy and joins. It serializes data to disk and moves it over the network, making it the most expensive thing Spark does. Good Spark tuning is mostly shuffle management.
PySpark. The DataFrame API compiles to the same plans as Scala, so performance is equivalent for untyped transformations, and the ecosystem, hiring pool, and notebook experience are all Python-first now. Scala matters mainly for low-level RDD work and library development.
Aim for 128–256 MB of data per partition as a starting point. Thousands of tiny tasks waste scheduler overhead; a handful of giant ones create stragglers. spark.sql.shuffle.partitions and adaptive query execution are the levers.
