Spark

Solution

The driver builds the plan. Executors run tasks on partitions. A Python UDF runs in a Python worker next to the executor — it is not “free SQL.” Prefer Spark SQL / built-in functions. collect() on a large frame is how you OOM the driver.

Solution

Narrow (map, filter) stays on the same partition. Wide (groupBy, join) shuffles over the network. Shuffle is the expensive part. If you can filter and project before a join, you cut shuffle bytes. That is the interview.

Solution

repartition shuffles to N partitions (increase parallelism, or partition by a key). coalesce reduces partitions without a full shuffle (after a filter). coalesce(1) to write one file is fine for a small report and a disaster for a 2 TB table.

Solution

Notebooks hide cluster size, caching, and an interactive driver. Jobs hit real data volumes, a smaller cluster, and no cached dataframes. Also: widgets and dbutils shortcuts that are not in the wheel. Always run the job definition on a sample of prod-scale data.

Solution

Broadcast sends the small side to every executor. If it is not actually small, you OOM workers. AQE can convert automatically. A forced broadcast hint on a growing dimension is a time bomb. Measure the small side after filters.

Solution

When the UI shows time in serialization and shuffle bytes are huge. Register classes. It does not fix a bad join.

Solution

Object stores have weaker locality and more LIST/GET. File sizing and partition pruning matter more on S3.

Solution

Side effects belong on executors, idempotently. A driver for-loop over collect() is the anti-pattern.

Solution

BroadcastHashJoin (or AQE converted) in the physical plan / SQL tab. Do not guess from a hint you leftover last year.

Solution

A missing broadcast because the small table was estimated huge, or a explode that created the 2 TB. AQE helps skew and coalesces — it does not rewrite a bad join key. Look at the SQL plan after AQE, not the one you hoped for.

Solution

Cache is lazy until an action; memory was evicted; or you transformed after cache without using the cached handle. persist(StorageLevel) and check the Storage tab. Caching is not a free lunch on a 20-node cluster.

Solution

expr / built-in functions. Python UDFs kill Catalyst and Photon. If you must UDF, pandas/arrow vectorized, and measure. “It was faster on my laptop” is not a plan.

Solution

The file schema drifted. Rescue is a quarantine, not a feature. Add the new fields or fail the pipeline. Dropping rescued_data is how you lose orders.

Solution

When 200 files are still tiny (KB) or still huge (10 GB). Aim for file size (128–512 MB) from table needs, not a lucky integer. coalesce after a wide shuffle if you only need fewer files.

Solution

The notebook was the application. Package a wheel/job, pass configs, and do not depend on interactive widgets. “Worked in the notebook” is the most common Spark incident report.