Iceberg vs Delta Lake in 2026: Metadata Design, Catalog Options, and How to Actually Choose

I have now migrated tables in both directions, and I no longer think either format is better. The 2022 arguments about which one has merge-on-read or proper schema evolution are dead — both have everything on that list. What I learned instead is that the format is the smaller decision, and the catalog is the larger one.

That reframing matters because most teams debate this backwards. They compare feature matrices for two weeks, choose Iceberg for openness, then run it single-engine behind a Hive catalog and get none of the benefit they chose it for. Or they choose Delta because they are a Databricks shop, which is correct, and then spend a quarter worrying about lock-in that UniForm and Unity Catalog’s Iceberg endpoint already addressed.

The differences that remain are structural: how metadata is organized, how a commit becomes visible, what the catalog is responsible for, and which engines can write without heroics. Those genuinely change how you operate a lake, and they point at different answers depending on what writes your tables.

Metadata topology, which is most of the difference

Both formats solve one problem: object storage has no transactions, so the set of files constituting a table must be recorded somewhere authoritative. They record it differently, and nearly every operational difference follows from that.

Delta keeps a numbered, sequential log in _delta_log. Each commit is a JSON file listing add and remove actions. Every ten commits, a Parquet checkpoint captures full state so readers do not replay history. Version N is the table after replaying commits 0 through N. I walked through the mechanics in the transaction log post.

Iceberg keeps a tree. A metadata JSON file points at the current snapshot; the snapshot points at a manifest list; the manifest list points at manifest files; manifests list data files with per-column statistics. Every commit writes a new metadata file, and the catalog atomically swaps a pointer to it.

flowchart TD
  cat["Iceberg: catalog pointer"] --> md["metadata.json v42"]
  md --> snap["Snapshot 8891"]
  snap --> ml["Manifest list"]
  ml --> m1["Manifest A"]
  ml --> m2["Manifest B"]
  m1 --> d1["Parquet data files"]
  m2 --> d1
  lg["Delta: _delta_log"] --> j1["000...41.json"]
  j1 --> j2["000...42.json"]
  j2 --> ck["Checkpoint parquet"]
  ck --> d2["Parquet data files"]

A pointer swap in a catalog versus an atomic file creation in a log. Same guarantee, different dependency.

Three practical consequences.

Commit atomicity depends on different things. Delta needs atomic “create if not exists” on the storage layer, which is why S3 historically needed a coordination service and why ADLS and GCS were easier. Iceberg pushes atomicity into the catalog, so any catalog implementing compare-and-swap works on any storage. If you run on storage with weak guarantees, Iceberg’s model is less fragile.

Planning cost scales differently. Delta reads one checkpoint plus a handful of JSON files — very fast, and roughly constant. Iceberg reads metadata, a manifest list, and then only the manifests whose partition ranges match your predicate, which prunes better on tables with tens of thousands of partitions but costs more small object reads. On a table with 200,000 partitions, Iceberg’s manifest pruning is genuinely better. On a table with 400, Delta plans faster.

Metadata cleanup is a different chore. Delta has logRetentionDuration and VACUUM. Iceberg has snapshot expiration, orphan file removal, and manifest rewriting, each a separate procedure. Neglected Iceberg tables accumulate manifests until planning slows, and the failure is subtle because the table still works, just slower every week.

-- Iceberg maintenance: three separate jobs, all needed
CALL catalog.system.expire_snapshots(
  table => 'prod.silver.orders',
  older_than => TIMESTAMP '2026-05-12 00:00:00',
  retain_last => 10
);

CALL catalog.system.rewrite_manifests('prod.silver.orders');

CALL catalog.system.rewrite_data_files(
  table => 'prod.silver.orders',
  strategy => 'binpack',
  options => map('target-file-size-bytes', '268435456')
);

CALL catalog.system.remove_orphan_files(
  table => 'prod.silver.orders',
  older_than => TIMESTAMP '2026-05-12 00:00:00'
);

-- Delta equivalent: one command plus one property set
ALTER TABLE prod.silver.orders SET TBLPROPERTIES (
  'delta.targetFileSize' = '256mb',
  'delta.autoOptimize.autoCompact' = 'true'
);

OPTIMIZE prod.silver.orders
WHERE order_date >= current_date() - INTERVAL 3 DAYS;

VACUUM prod.silver.orders RETAIN 168 HOURS;

Delta’s maintenance story is simpler, and on Databricks managed tables it is largely automatic. Iceberg’s is more explicit, which means more scheduling work and more control.

Partitioning: hidden versus declared

This is the difference I feel most often as a query author.

Iceberg has hidden partitioning. You declare a partition transform on a column, and Iceberg records the mapping from column value to partition. A query filtering on the raw column prunes correctly without knowing the physical layout.

-- Iceberg: partition by a transform of the timestamp
CREATE TABLE prod.silver.events (
  event_id   BIGINT,
  user_id    BIGINT,
  event_ts   TIMESTAMP,
  payload    STRING
) USING iceberg
PARTITIONED BY (days(event_ts), bucket(16, user_id));

-- Prunes correctly. No derived partition column in the predicate.
SELECT count(*) FROM prod.silver.events
WHERE event_ts >= TIMESTAMP '2026-05-01 00:00:00';

-- Partition evolution: change the scheme without rewriting history
ALTER TABLE prod.silver.events
  REPLACE PARTITION FIELD days(event_ts) WITH hours(event_ts);

In Hive-style or Delta partitioning, you would materialize event_date as a column and every query would need a predicate on it. Forget it, and you scan the table. I have debugged that exact mistake in production more times than I want to admit, and hidden partitioning eliminates the entire class.

Partition evolution is the other real win. Iceberg can change the partition scheme going forward while old data keeps its old layout, because manifests record which spec each file used. In Delta you would rewrite the table.

Delta’s answer is to stop partitioning. Liquid clustering replaces partition columns with clustering keys that the engine maintains incrementally, and it handles the small-partition and skew-partition problems that hand-chosen partition columns create.

-- Delta: cluster instead of partition, and change keys later
CREATE TABLE prod.silver.events (
  event_id BIGINT, user_id BIGINT, event_ts TIMESTAMP, payload STRING
) USING delta
CLUSTER BY (event_ts, user_id);

ALTER TABLE prod.silver.events CLUSTER BY (event_ts, region);
OPTIMIZE prod.silver.events;   -- incremental reclustering

Having run both, liquid clustering is the better default for tables with unpredictable query patterns, and Iceberg’s hidden partitioning plus explicit transforms is better when the access pattern is well-known and time-based. Neither is a reason to switch formats on its own.

Row-level deletes: the one place mechanics still differ

Both formats avoid rewriting whole files for small changes, using different representations.

Delta uses deletion vectors: a bitmap per data file marking dead rows, referenced from the add action. Readers apply the bitmap at scan time.

Iceberg v2 used positional and equality delete files. Positional deletes name file and row positions; equality deletes name key values, which lets a writer delete rows without knowing where they are — useful for CDC, expensive for readers, because every matching delete file must be joined against the data. Iceberg v3 introduced deletion vectors as well, converging on Delta’s approach, which tells you the industry decided.

The operational rule is identical either way: read cost grows with accumulated deletes, so compaction is mandatory on mutation-heavy tables. A CDC target table that never gets rewrite_data_files or OPTIMIZE degrades in a way that looks like “the warehouse got slow” rather than “we skipped maintenance.”

If you are on Iceberg v2 with equality deletes on a high-churn table, that is the configuration I would prioritize fixing. Equality deletes plus no compaction is the worst read path either format offers.

Catalogs, which is the decision that actually locks you in

A format tells an engine how to read files. A catalog tells it which tables exist and which snapshot is current. Change formats and you rewrite metadata; change catalogs and you re-point every engine, every permission, and every pipeline.

Delta’s options are effectively Unity Catalog, Hive metastore, or a storage path with no catalog at all. Unity Catalog is where all the investment goes, and the governance model is genuinely good — the layout and grant patterns I use are in the Unity Catalog guide.

Iceberg’s options are more numerous, and this is where its interoperability story lives:

• REST catalog — the protocol, not an implementation. Polaris, Unity Catalog, Nessie, Lakekeeper, Gravitino, and the cloud catalogs all speak it. This is the standard, and any new build should target it.
• AWS Glue — convenient on AWS, weaker at cross-engine commit coordination than a REST implementation.
• Nessie — git-like branching and tagging of the whole catalog, which is a real capability if you want to test a pipeline against a branch of production data.
• JDBC or Hive — legacy. Works, no reason to choose it now.
• Hadoop / filesystem — no catalog. Do not use it with multiple writers; the pointer swap has no coordinator.

# Same Spark session, both formats, catalog config is the real difference
spark = (SparkSession.builder
    # Iceberg through a REST catalog: the portable option
    .config("spark.sql.catalog.lake", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.lake.type", "rest")
    .config("spark.sql.catalog.lake.uri", "https://catalog.internal:8181")
    .config("spark.sql.catalog.lake.warehouse", "s3://lake/warehouse")
    .config("spark.sql.catalog.lake.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
    # Delta: extension plus a metastore, no per-catalog plumbing
    .config("spark.sql.extensions",
            "io.delta.sql.DeltaSparkSessionExtension,"
            "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .config("spark.sql.catalog.spark_catalog",
            "org.apache.spark.sql.delta.catalog.DeltaCatalog")
    .getOrCreate())

The important development of the last two years is that Unity Catalog speaks the Iceberg REST protocol, so Snowflake, Trino, and Flink can read Databricks-managed tables through a standard interface, and Delta tables can publish Iceberg metadata through UniForm. The either-or framing is largely obsolete for reads. It is still real for writes, because exactly one system should own writing a given table.

Engine support in practice

What I have actually run, rather than what the docs claim:

• Spark — first-class for both. No meaningful difference.
• Databricks — Delta is the native path, with everything (predictive optimization, liquid clustering, DLT) built around it. Iceberg support is real and improving but a step behind.
• Snowflake — Iceberg is the native external format, and Snowflake can act as the Iceberg catalog or read from an external one. That story is covered in Snowflake Iceberg tables.
• Trino, Athena, Dremio — Iceberg is the better-supported format, particularly for writes.
• Flink — Iceberg has the stronger connector for streaming writes and compaction.
• DuckDB, Polars, and the local tools — both readable now, with Iceberg slightly ahead on catalog integration. Handy for the workflows in local DuckDB pipelines.

The pattern is consistent: Delta wins where Databricks is the center of gravity, Iceberg wins where several engines share the lake, and the gap in both directions is narrower every release.

How I choose

Two questions, in order.

What writes your production tables? If the honest answer is Databricks, use Delta. Fighting the platform’s native format to gain theoretical portability costs you real features today. Enable UniForm so Iceberg readers work, and expose tables through Unity Catalog’s REST endpoint.

If the answer is Snowflake, Trino, Flink, EMR, or “several of those,” use Iceberg with a REST catalog. That is precisely the case it was built for, and the multi-engine story works.

How many engines write, versus read? Many readers is fine in both formats through UniForm or REST federation. Many writers to one table is a different problem: pick one writer of record regardless of format, because concurrent writes from engines with different commit implementations is where I have seen actual corruption scares. The format guarantees atomicity per writer; it does not make two independent compaction jobs agree.

Migration between formats is mechanical either way — a CTAS rewrite, or in-place metadata conversion where the Parquet is already compatible. Budget a week for a medium-sized domain, mostly for repointing consumers rather than moving data. It is not the irreversible decision people treat it as, which argues for the pragmatic choice now over the theoretically pure one.

Pitfalls

Choosing Iceberg for openness, then running one engine and a Hive catalog. You paid the operational complexity and bought nothing. Openness comes from the catalog protocol and multiple engines, not from the file format alone.

Skipping Iceberg maintenance procedures. Snapshot expiration, manifest rewriting, data file compaction, and orphan cleanup are four separate jobs. Missing any one degrades the table slowly enough that nobody connects it to the omission.

Two writers, one table. Both formats detect conflicts, but only within their own commit implementation. An Iceberg table written by Spark and Flink with different catalog configurations, or a Delta table written by Databricks and a delta-rs job, is asking for a bad afternoon.

Assuming UniForm makes a table bidirectional. UniForm generates Iceberg metadata for readers. Iceberg engines still must not write that table. It is a read bridge, not shared ownership.

Partitioning an Iceberg table too finely because it is hidden. Hidden partitioning removes the predicate burden, not the small-file cost. hours(ts) on a low-volume table gives you thousands of tiny partitions and a metadata problem.

Protocol and format version drift. Enabling deletion vectors, column mapping, or Iceberg v3 features bumps the minimum reader version. External consumers that do not support it stop reading entirely. Inventory consumers before flipping features on a shared table.

FAQ

Is one format faster than the other?

Not meaningfully, at the same file sizes and clustering quality. I have benchmarked both on the same data and found differences inside the noise for typical analytics queries. Metadata planning diverges at extremes — Iceberg prunes better with very many partitions, Delta plans faster with very many commits — and file layout dominates everything else.

Does UniForm mean the choice does not matter?

For readers, mostly yes. For writers, no: UniForm generates Iceberg metadata alongside Delta as a read bridge, with a small write overhead and a slight metadata lag. One system still owns writes.

Which is better for CDC and merge-heavy tables?

Roughly equal now that Iceberg v3 has deletion vectors. Delta’s MERGE on Databricks with Photon is the fastest implementation I have measured, but that is an engine advantage rather than a format one. Compaction discipline matters more than format either way.

Should I migrate an existing Delta lake to Iceberg?

Only for a concrete reason: a second engine that must write, a governance requirement, or a platform migration already underway. “Openness” without a specific engine attached is not a reason, and the migration consumes weeks of consumer repointing.

What about Hudi?

Still capable for upsert-heavy ingest, but the ecosystem has consolidated around Iceberg and Delta, and engine support reflects that. I would not start a new lake on it in 2026.

What this means for your pipelines

Decide by write engine, not by feature list. If Databricks writes your tables, Delta with liquid clustering and UniForm is the pragmatic answer and you keep every platform feature. If your lake is genuinely shared between Snowflake, Trino, Flink, and Spark, Iceberg behind a REST catalog is the answer and the interoperability is real rather than aspirational.

Then spend your remaining attention on the two things that actually determine whether the lake works: the catalog and the maintenance schedule. Catalog choice outlives format choice and is far harder to reverse, so target the REST protocol if there is any chance of multiple engines. Maintenance — compaction, snapshot or log expiration, orphan cleanup — is what separates a lake that stays fast from one that gets 15 percent slower every month while everyone blames the query engine.

The format debate is settled enough that either choice is defensible. Choosing neither, and leaving a folder of Parquet with a naming convention and a prayer, is the only genuinely wrong answer, and it is still surprisingly common.