Databricks

Solution

Delta is the default if Databricks is the compute and you want UniForm / deletion vectors / Liquid Clustering in that ecosystem. Iceberg if Trino, Snowflake, and Spark must read the same files without a vendor lock. Do not dual-write “just in case.”

Solution

Bronze is raw as landed (schema-on-read, lineage). Silver is cleaned, typed, deduped, conformed keys. Gold is the grain a dashboard or feature can trust. Skipping silver is how gold lies. Optional: a serving layer that is just gold plus a rollup.

Solution

OPTIMIZE compacts small files. ZORDER (or Liquid Clustering) colocates filter columns. VACUUM removes stale files after the retention window — and it is irreversible. Do not VACUUM 0 hours in prod. Quote the retention and the “time travel vs vacuum” trade.

Solution

Broadcast the small side if it fits. Repartition both on the join key. AQE / skew join hints. Filter before the join. If it is still huge, rethink the grain — you may be exploding a many-to-many. Scaling the cluster last is the expensive answer.

Solution

UC is the governance plane: identities, row filters, lineage, a 3-level namespace. HMS is the legacy catalog. “We still have two catalogs” is a real migration answer. Do not claim ACLs on a path-based table you never registered.

Solution

Auto Loader (cloudFiles) tracks what was ingested with a checkpoint — good for landing buckets that grow all day. A scheduled COPY is fine for a nightly dump. The fail is “streaming” a file drop that arrives once a day, or batching a firehose with no watermark.

Solution

Job cluster that dies when the run ends. An always-on all-purpose box is an idle bill.

Solution

UniForm (or one Iceberg table) with a single writer. Dual-write is how the two copies lie to each other.

Solution

Fail or quarantine when the SLA cannot tolerate bad keys. Warn-only is a blog post. Drop is a product decision you must document.

Solution

When job start time is in the SLA and cold VM spin is 8 minutes. Size the idle count to the SLA, not to max theoretically.

Solution

Liquid Clustering when the table is Delta and you want ongoing layout without OPTIMIZE heroics. ZORDER is the older weekly job. Do not ZORDER every column “because Tableau.”

Solution

No. You silently deleted a cohort. expect_or_fail for contracts; quarantine for the rest. Drop is for true garbage (malformed JSON), not “status we do not like.”

Solution

A Python UDF that leaves Photon, a skew join, or tiny files. Photon does not fix a CROSS JOIN. Look at the Spark UI SQL tab before buying more workers.

Solution

Hard-coded dbfs:/ in production jobs. Volumes + UC grants are the paved road. DBFS mounts with a personal token are how you leak prod.

Solution

SQL warehouse for BI/adhoc SQL; job cluster (or SQL warehouse with a scheduled dbt job) for ELT. All-purpose clusters for scheduled ETL are an idle-VM tax.

Solution

The reader can fail on missing files. Time Travel is not a promise beyond retention. Tell analysts the retention SLA; do not VACUUM 0 HOURS to “save storage.”