All stacks · Lakehouse & table formats
Open table format, catalogs, and multi-engine lakehouses.
Iceberg🟡IntermediateWhat each file format is actually built for — columnar analytics, Hive-native storage, and row-oriented streaming records — plus the compression, schema evolution, and small-file realities that decide your bill.
Iceberg🟡IntermediateThe feature lists have converged. What still differs is metadata structure, catalog architecture, and engine support, and those are what decide which format fits your stack.
Trino🟢BeginnerTrino stores nothing. A federated SELECT * on prod Postgres is a database incident. Use Iceberg for the lake, and keep interactive SQL off OLTP.
Hadoop🟢BeginnerNameNode small-file pauses, the Hive metastore as the real lock-in, and why distcp-plus-EMR keeps the same tables. Iceberg is the exit, not a new cluster.
Snowflake🟡IntermediateWhen to use Snowflake-managed Iceberg vs native tables: external volumes, catalogs, deletion vectors, and the interoperability jobs Iceberg actually wins.
Tools🟡IntermediateAn honest head-to-head: SQL warehousing vs Spark lakehouse, Iceberg interoperability, Cortex vs Mosaic, and when you actually need both.
Databricks🟡IntermediateTable-format choice in 2026: deletion vectors, catalogs, and which engine you are willing to lock in.
Open table formats are the biggest structural shift in analytics since the cloud warehouse: Iceberg, Delta, and Hudi turn object-store files into transactional tables that any engine can read and write. Storage stops being vendor territory; catalogs become the new strategic ground.
These guides cover the format layer in practice — how Iceberg's metadata tree enables time travel and safe schema evolution, running Snowflake and Spark against the same tables, catalog choices from Glue to Polaris to Unity, and the maintenance (compaction, snapshot expiry) that keeps lakehouse tables fast.
Because both Snowflake and Databricks embraced it, making it the neutral interchange format between warehouse and lakehouse worlds. One copy of data in open files, queried by many engines, with no export pipelines — that economic argument is hard to ignore.
In Databricks-centric platforms, Delta is native and excellent. For multi-engine neutrality — Snowflake, Trino, Flink, DuckDB against the same tables — Iceberg has the broader write support and the momentum in catalogs. Increasingly you can hedge: engines are adding read/write for both.
It maps table names to the current metadata file and provides the atomic swap that makes commits transactional. Every engine must go through the same catalog for consistent reads and writes — which is why catalog choice (Glue, Polaris, Unity, REST) is the real architecture decision.
It replaces the warehouse's proprietary storage, not its compute, optimizer, or governance. A common end-state is warehouse compute (Snowflake, BigQuery) operating over Iceberg tables in your own buckets — performance trade-offs versus native tables are shrinking each release.
Regular compaction of small files, snapshot expiration to bound metadata and storage growth, and orphan-file cleanup after failed writes. Managed services increasingly automate this; if you self-manage, schedule it like any other pipeline or reads degrade slowly.
