DataLane
← All stacks

Spark

Jobs, shuffles, and the PySpark questions that show you have shipped

9 questions with solutions

  1. Q1DatabricksAppleNetflix

    Driver vs executor. Where does your Python UDF actually run?

    Solution

    The driver builds the plan. Executors run tasks on partitions. A Python UDF runs in a Python worker next to the executor — it is not “free SQL.” Prefer Spark SQL / built-in functions. collect() on a large frame is how you OOM the driver.

  2. Q2UberLyftLinkedIn

    Narrow vs wide transformation. Why does that decide runtime?

    Solution

    Narrow (map, filter) stays on the same partition. Wide (groupBy, join) shuffles over the network. Shuffle is the expensive part. If you can filter and project before a join, you cut shuffle bytes. That is the interview.

  3. Q3AmazonWalmartTarget

    repartition vs coalesce. When is each correct?

    Solution

    repartition shuffles to N partitions (increase parallelism, or partition by a key). coalesce reduces partitions without a full shuffle (after a filter). coalesce(1) to write one file is fine for a small report and a disaster for a 2 TB table.

  4. Q4DatabricksShellComcast

    Why did a job that “worked in the notebook” fail as a scheduled job?

    Solution

    Notebooks hide cluster size, caching, and an interactive driver. Jobs hit real data volumes, a smaller cluster, and no cached dataframes. Also: widgets and dbutils shortcuts that are not in the wheel. Always run the job definition on a sample of prod-scale data.

  5. Q5MetaGoogleMicrosoft

    Broadcast join vs shuffle join. What is the failure mode of a hint?

    Solution

    Broadcast sends the small side to every executor. If it is not actually small, you OOM workers. AQE can convert automatically. A forced broadcast hint on a growing dimension is a time bomb. Measure the small side after filters.

  6. Q6DatabricksNetflixUber

    Kryo vs Java serialization. When do you care?

    Solution

    When the UI shows time in serialization and shuffle bytes are huge. Register classes. It does not fix a bad join.

  7. Q7NetflixAmazonDatabricks

    S3 vs HDFS locality. Why was the same job faster on HDFS?

    Solution

    Object stores have weaker locality and more LIST/GET. File sizing and partition pruning matter more on S3.

  8. Q8DatabricksAppleLinkedIn

    foreach on the driver vs foreachPartition.

    Solution

    Side effects belong on executors, idempotently. A driver for-loop over collect() is the anti-pattern.

  9. Q9DatabricksMetaNetflix

    Read an explain plan: how do you confirm a broadcast?

    Solution

    BroadcastHashJoin (or AQE converted) in the physical plan / SQL tab. Do not guess from a hint you leftover last year.

↑↓ navigate↵ openesc close