All stacks · Compute & processing

Hadoop

HDFS, YARN, Hive, and the on-prem stack still running in many banks.

Illustrated overview of Hadoop Hive to Iceberg: Why Lift-and-Shift to S3 Fails and How Banks Actually Exit

Hadoop🟢Beginner

Aug 25, 20266 min read

Hadoop Hive to Iceberg: Why Lift-and-Shift to S3 Fails and How Banks Actually Exit

NameNode small-file pauses, the Hive metastore as the real lock-in, and why distcp-plus-EMR keeps the same tables. Iceberg is the exit, not a new cluster.

hadoop
hive
iceberg

Related reading

Illustrated overview of Parquet vs ORC vs Avro: Pick by Access Pattern, Not by Benchmark

Iceberg🟡Intermediate

Sep 1, 20267 min read

Parquet vs ORC vs Avro: Pick by Access Pattern, Not by Benchmark

What each file format is actually built for — columnar analytics, Hive-native storage, and row-oriented streaming records — plus the compression, schema evolution, and small-file realities that decide your bill.

iceberg
spark
lakehouse

Illustrated overview of Unity Catalog in Practice: Metastore Layout, Grants That Scale, and Leaving hive_metastore

Databricks🟡Intermediate

Apr 22, 20268 min read

Unity Catalog in Practice: Metastore Layout, Grants That Scale, and Leaving hive_metastore

How I structure Unity Catalog for real teams: three-level namespace design, group-based grants, lineage that works, and a hive_metastore migration that ships.

databricks
governance

About Hadoop

Hadoop is not dead in the places that matter for interviews: banks, telcos, and on-prem estates still run HDFS, YARN, Hive, and Sqoop. The cloud lakehouse is the migration story, not the starting point for those teams.

Learn Hive tables, partitioning, and why small files kill NameNodes — then learn Iceberg on object storage as the exit. Skipping Hadoop entirely is how you fail a legacy-platform screen.

What you'll learn here

HDFS, YARN, and what a “cluster” meant before object storage
Hive Metastore as the catalog everyone still talks to
Small-file problems and why they follow you to S3
Migration patterns: Hive → Iceberg / Delta without a big-bang cutover

Frequently asked questions

Should a new platform start on Hadoop?

No. New platforms start on object storage plus Iceberg or a warehouse. Hadoop skills still matter because migrations and interviews assume you can read a Hive table and a YARN queue.

Hive or Spark SQL on a lake?

Spark (or Trino) on Iceberg/Delta is the modern path. Hive-on-Tez is maintenance. If the metastore is Hive, you can still query it from engines that are not Hive.

What is the usual Hadoop-to-cloud failure?

Lift-and-shift of every small file and every cron Sqoop job into S3 + EMR with no table format. You moved the NameNode problem into LIST requests and a bigger bill.