Hadoop Hive to Iceberg: Why Lift-and-Shift to S3 Fails and How Banks Actually Exit

Monday’s risk feed started 11 minutes late because the NameNode spent 11 minutes in GC. The warehouse had 340 million files across 18,000 Hive tables. Average file size in the landing database was 2.4 MB. Someone’s hourly Spark job wrote 1,200 files per partition per run and never compacted. The NameNode heap was 96 GB and it was still a stop-the-world event, not a capacity plan.

Six months later the same bank tried the fashionable exit: distcp the Hive warehouse to S3, stand up EMR, point the same metastore at s3://. Jobs got slower. The S3 LIST bill in month two was $4,100. There was no NameNode, and there were still 340 million objects. We had moved the small-file problem into a system that charges per list.

flowchart TD
  hdfs[HDFS + Hive tables] --> nn{NameNode heap / small files}
  nn -->|pause| fire[On-call]
  hdfs --> lift[distcp to S3 + EMR]
  lift --> list[Same files, S3 LIST tax]
  hdfs --> ice[Rewrite as Iceberg]
  ice --> snap[Snapshots, not directory listings]
  snap --> engine[Spark / Trino / warehouse]
  list --> still[Same metastore, same pain]

Lift-and-shift copies files. An Iceberg exit copies *tables* into a format that does not list a directory to know what exists.

This stack is still in banks for boring reasons

Air-gapped networks. Auditors who signed off on Kerberos and HDFS ACLs in 2017. A Hive Metastore that every SAS job, every Informatica mapping, and every Java batch still calls. You do not replace that with a weekend and a lakehouse slide.

I stop arguing about “Hadoop is dead” in those rooms. I argue about file counts, metastore mutation rate, and which table is allowed to move first. The cluster can stay until the tables leave.

Small files are a NameNode incident

The NameNode holds inode metadata in memory. Hundreds of millions of files is a heap problem, then a GC problem, then an availability problem. Compaction jobs that never ran are how you get there. Hive on MapReduce or Tez writing one file per reducer per partition is the default path.

-- Inventory: tables that will take the NameNode down.
-- Run against the metastore DB (Postgres / MySQL), not Hive.
select
  d.name                          as db_name,
  t.tbl_name,
  count(*)                        as partition_count,
  sum(s.num_files)                as files,
  sum(s.num_files) / nullif(count(*), 0) as files_per_part
from dbs d
join tbls t          on t.db_id = d.db_id
join partitions p    on p.tbl_id = t.tbl_id
join tablestats s    on s.part_id = p.part_id
where d.name in ('landing', 'risk')
group by 1, 2
having sum(s.num_files) > 100000
order by files desc
limit 50;

Column names vary by HMS version. The idea does not: find the tables whose file count, not byte count, dominates. Compact those on HDFS before you copy anything to S3, or you copy the incident.

The metastore is the lock-in

HDFS is a disk. The Hive Metastore is the catalog every engine already trusts: databases, tables, partitions, SerDes, permissions that are actually GRANT in a different product. Move the files and keep HMS pointed at paths, and you have not exited. You have changed the URI scheme.

HMS also does not like millions of partition rows being added and dropped on a Monday morning. That is a second outage mode, independent of the NameNode.

Lift-and-shift to S3 plus EMR fails in a specific way

distcp is fine for cold data. It is not a table format migration. Directory-as-partition Hive tables on S3 mean every query plans through LIST. Small files make that list huge. EMR will run the same Spark you had, against worse IO characteristics, and the metastore is still a single database you now share with the cloud estate.

You also lose HDFS block locality and you gain eventual consistency edge cases you forgot existed. The jobs that were “fine” on HDFS become the ones that time out listing dt=2024-03-15.

Exit through Iceberg

Rewrite the hot tables into Iceberg. Keep HMS as the catalog until you can stand up a REST catalog — the format-versus- catalog split is in Iceberg vs Delta Lake. Spark on the existing cluster can write Iceberg to HDFS first. Then copy Iceberg metadata plus data files to S3. Readers open a snapshot. They do not list 40,000 objects to see Monday.

-- Spark SQL, Hive catalog already configured for Iceberg.
create table risk.positions_ice
using iceberg
partitioned by (days(as_of_date))
tblproperties (
  'write.target-file-size-bytes' = '134217728',
  'format-version' = '2'
)
as select * from risk.positions
where as_of_date >= date '2024-01-01';

-- Dual-read window: old table stays until the last SAS job cuts over.
-- Then drop risk.positions and rename.

Target 128 MB files. That one property does more for the NameNode (and for S3) than a new YARN queue. Do not migrate 18,000 tables. Migrate the fifty that page you and the twenty that feed them. Leave the 2016 archive on HDFS until the lease on the disk ends.

Engines change after the table format. Trino or a warehouse can read Iceberg without speaking Hadoop. That is the exit. A new EMR cluster reading Hive directories is a change of hat.

Failure modes

File count ignored, cluster size debated. The NameNode does not care that you added datanodes.

distcp as the migration. Same files, new bill.

Iceberg on day one with a new catalog and a new engine. Three migrations stacked. Do one: the table format.

Compaction never scheduled. Iceberg will grow small files too if every microbatch commits.

Metastore left as an afterthought. It is the lock-in.

Dual-read is the cutover, not a weekend swap

Keep risk.positions readable until the last SAS job and the last Informatica mapping point at risk.positions_ice. I put a row-count and a checksum job on both tables for the overlapping dates. When they agree for two weeks and the consumer list is empty, rename. A dual-write weekend without a consumer inventory is how you run two wrong numbers for a quarter.

Hub: Hadoop. Format vs catalog: Iceberg vs Delta. If EMR is the compute you already have, it can write Iceberg. It should not keep listing Hive directories.

FAQ

Why is Hadoop still in banks? Closed networks, signed-off Kerberos, and a Hive metastore every SAS job still calls. The cluster stays until the tables leave.

Will distcp to S3 fix small files? No. You move 340 million objects and start paying LIST. Compact on HDFS first, or rewrite as Iceberg so readers open a snapshot.

Is the NameNode or the metastore the lock-in? HDFS is the disk. HMS is the catalog. Move files and keep HMS pointed at paths, and you have changed a URI.

Do I migrate 18,000 tables? No. The fifty that page you and the twenty that feed them. Leave the 2016 archive until the disk lease ends.

New cluster or new format first? Format first. A new EMR cluster reading Hive directories is a change of hat. Engines change after Iceberg exists.

What to do Monday

Query the metastore for file counts. Compact the worst Hive tables on HDFS. Pick ten tables and rewrite them to Iceberg with a 128 MB target, same catalog. Dual-read until the last consumer cuts over. Do not buy a new cluster to store the same 2 MB files under a different logo.