Driver vs executor. Where does your Python UDF actually run?

The driver builds the plan. Executors run tasks on partitions. A Python UDF runs in a Python worker next to the executor — it is not “free SQL.” Prefer Spark SQL / built-in functions. collect() on a large frame is how you OOM the driver.

foreach on the driver vs foreachPartition.

Side effects belong on executors, idempotently. A driver for-loop over collect() is the anti-pattern.

AQE is on. You still got a 2 TB shuffle. What did AQE not fix?

A missing broadcast because the small table was estimated huge, or a explode that created the 2 TB. AQE helps skew and coalesces — it does not rewrite a bad join key. Look at the SQL plan after AQE, not the one you hoped for.

Hardware / privacy · Interview pack

Apple data interview pack

23 written questions with answers, the loop shape people report for Apple, and the stacks to practise. Built from DataLane's original question bank and publicly discussed interview themes, not leaked exams. Last editorial update 2026-08-08.

What is in this pack

Questions by topic

Coding4
Data modeling1
Streaming1
Warehouse12
System design3
Behavioral2

8 easy · 10 medium · 5 hard

Rounds (2)

Design
SQL

Themes: Privacy by default, On-device vs warehouse, Shared lake

Free with a DataLane account

The first 3 questions are below. Sign in (free, no plan needed) to read all 23 and download the pack as a PDF.

Free preview

1. Coding · medium · Spark
Driver vs executor. Where does your Python UDF actually run?
The driver builds the plan. Executors run tasks on partitions. A Python UDF runs in a Python worker next to the executor — it is not “free SQL.” Prefer Spark SQL / built-in functions. collect() on a large frame is how you OOM the driver.
2. Coding · easy · Spark
foreach on the driver vs foreachPartition.
Side effects belong on executors, idempotently. A driver for-loop over collect() is the anti-pattern.
3. Coding · hard · Spark
AQE is on. You still got a 2 TB shuffle. What did AQE not fix?
A missing broadcast because the small table was estimated huge, or a explode that created the 2 TB. AQE helps skew and coalesces — it does not rewrite a bad join key. Look at the SQL plan after AQE, not the one you hoped for.

20 more questions are in the full pack.

The Apple loop

Design
Two engines, one Iceberg table. What you refuse to dual-write. How identifiers stay out of a vendor LLM.
SQL
Grain and uniqueness under a privacy constraint — no “just dump the fact to a notebook.”

Coding (4)

medium · Spark
Driver vs executor. Where does your Python UDF actually run?
The driver builds the plan. Executors run tasks on partitions. A Python UDF runs in a Python worker next to the executor — it is not “free SQL.” Prefer Spark SQL / built-in functions. collect() on a large frame is how you OOM the driver.
easy · Spark
foreach on the driver vs foreachPartition.
Side effects belong on executors, idempotently. A driver for-loop over collect() is the anti-pattern.
hard · Spark
AQE is on. You still got a 2 TB shuffle. What did AQE not fix?
A missing broadcast because the small table was estimated huge, or a explode that created the 2 TB. AQE helps skew and coalesces — it does not rewrite a bad join key. Look at the SQL plan after AQE, not the one you hoped for.
medium · Spark
repartition(200) before a write. When is that the wrong knob?
When 200 files are still tiny (KB) or still huge (10 GB). Aim for file size (128–512 MB) from table needs, not a lucky integer. coalesce after a wide shuffle if you only need fewer files.

Data modeling (1)

hard · Python & modeling
You hashed emails as a join key. Legal says stop. What was the modeling mistake?
A hash of PII is still PII if it is reversible or unique. Use a vault token from a real identity service, or a surrogate that is not derived from the email. Hashing is not anonymization.

Streaming (1)

medium · Apache Flink
Checkpoint vs savepoint.
Checkpoints are automatic recovery. Savepoints are operator-triggered, versioned points for deploy and resize. You do not “take a checkpoint” for a migration; you take a savepoint.

Warehouse (12)

medium · Recent signal, 2026-09-02
A 09:00 dashboard federates Iceberg facts to prod Postgres dims. What do you refuse?
Scanning the primary for BI. Replicate dims to the lake. Federation is a small lookup, not the exec pack.
easy · Databricks
When do you pick Delta on Databricks versus Iceberg on a shared lake?
Delta is the default if Databricks is the compute and you want UniForm / deletion vectors / Liquid Clustering in that ecosystem. Iceberg if Trino, Snowflake, and Spark must read the same files without a vendor lock. Do not dual-write “just in case.”
easy · Databricks
UniForm vs dual-write Delta and Iceberg.
UniForm (or one Iceberg table) with a single writer. Dual-write is how the two copies lie to each other.
medium · Databricks
You VACUUMed with a 7-day retention and a reader is on an 8-day-old notebook. What happens?
The reader can fail on missing files. Time Travel is not a promise beyond retention. Tell analysts the retention SLA; do not VACUUM 0 HOURS to “save storage.”
easy · Trino
What does Trino store?
Nothing durable of yours. Coordinator + workers query catalogs. Gold tables live in Iceberg, Hive, or a warehouse. Calling Trino a warehouse is a fail.
hard · Trino
Trino or Spark SQL on the same Iceberg table?
Trino for interactive federation. Spark for heavy ETL. Same table, different engines — that is the lakehouse pitch. Two writers is still forbidden.
easy · Trino
Trino does not store the table. So what does it store, and why do people still page you?
It stores the query engine’s metadata/catalog config and the plan. The data lives in Iceberg/Hive/Postgres. People page you because a federated scan of prod Postgres looked like a warehouse.
hard · Trino
A federated join of Iceberg facts to Postgres dims took down OLTP. Guardrail?
Do not federate large facts against primary. Replicate dims to the lake, or restrict the catalog, or cap query memory and scan size. Federation is for small lookups, not the 09:00 dashboard.
easy · Trino
SELECT * on a 400-column Iceberg fact in Trino. What happens to the plan?
It still reads every column from Parquet footers/row groups you need — column pruning helps only if you list columns. Teach BI to select metrics. SELECT * is a cost bug.
medium · Trino
Hive connector vs Iceberg connector on the same files. Why does Iceberg win for concurrent writes?
Iceberg snapshots and optimistic commit; Hive table is files + a metastore that races. Two writers on Hive is corrupt partitions. Iceberg is the contract.
medium · Trino
Cost-based optimizer picked a broadcast of a 200 GB table. What do you check?
Stats missing or stale on Iceberg, so NDV looked tiny. ANALYZE / snapshot stats, or hint, or raise the broadcast threshold only with a story. Hints without stats are superstition.
medium · Trino
When is Trino the wrong engine vs Spark?
Heavy shuffles, iterative ML, and huge writes with custom committers. Trino shines at interactive SQL on the lake. Spark shines at jobs. Do not run nightly 8-hour ETL only in Trino because “SQL is simpler” if the plan is a disaster.

System design (3)

medium · Recent signal, 2026-08-26
Two engines must share one lake table. What do you refuse to do?
Dual-write CSV or two independent Delta copies. Pick Iceberg (or UniForm) with a single writer. Two writers is how the copies lie.
hard · System design
Multi-region: EU data cannot leave EU, US analysts want a global dashboard. Draw the legal design.
Region-local bronze/silver, aggregates that are non-PII replicated to a global mart, or a viewer in-region. Do not COPY PII to us-east-1 “temporarily.” Privacy engineering is part of the sketch.
medium · System design
PII in bronze, marketing wants a campaign table with emails. Controls you actually draw.
Separate schema, masking/row policies, approved join to a consent table, no SELECT * grants, and an access log. A hashed email is still often PII. Marketing gets a view, not bronze.

Behavioral (2)

easy · Behavioral
You disagreed with a senior engineer on lake vs warehouse. How did it end?
A spike: one workload, cost, and a consumer. Data, not identity. If you lost, say what you learned (maybe the warehouse was right for BI). If you won, say the rollback plan. Ego is the junior tell.
easy · Behavioral
You were asked to copy PII into a spreadsheet for a vendor demo. What did you do?
Refuse, offer synthetic or masked rows, and if pressured, escalate to security — in writing. A demo is not a lawful basis. This is a pass/fail behavioral, not a clever workaround.